# Load test results — autoscaling verification, b2g-UAT

**Date:** 2026-08-06 · **Window:** 08:01:55–08:15:55 UTC (load), observed to 08:34 UTC
**Service under test:** `cengex` (0.25 vCPU / 512 MB, min 1 / max 2, target 80% CPU)

## Verdict

Both directions of autoscaling work. Scale-out and scale-in each fired from the correct alarm,
completed successfully, and are recorded in the scaling activity log. Throughput scaled almost
linearly with the added task and latency halved. Nothing else in the environment was affected.

| # | Criterion | Result |
|---|---|---|
| 1 | CPU sustained > 80% under load | **Pass** — 95.4–97.1% |
| 2 | `desiredCount` 1 → 2 | **Pass** — at 08:07:10 UTC, 5m15s after load start |
| 3 | Second task runs and takes traffic | **Pass** — throughput 100 → 186 req/s |
| 4 | Scaling activity `Successful`, cause `AlarmHigh` | **Pass** |
| 5 | `desiredCount` 2 → 1 after load stops | **Pass** — at 08:33:35 UTC, 17m40s after load stop |
| 6 | No other service degraded | **Pass** — 0 failed requests, 0 5xx, RDS untouched |

---

## 1. Throughput and sessions handled

Load generator: bastion `i-04bed8d526c55e47d` (2 vCPU, Ubuntu 24.04),
`ab -c 4 -t 840 -r -k http://<alb>/cengex/web/loadtest-404`.

### Headline numbers

| Measure | Value |
|---|---|
| **Total requests served** | **120,257** |
| **Failed requests** | **0** |
| Mean throughput over the whole run | 143.2 req/s |
| Total bytes transferred | 61,641,471 (58.8 MB) |
| Duration | 840 s (14 min) |
| HTTP 5xx (ALB) | **0** |

ALB `RequestCount` independently totalled 120,715 over the window — the 458-request difference is
health checks and my own spot-checks, so the two sources agree.

### Throughput by phase — this is the scaling result

| Phase | Tasks | vCPU | Throughput | Latency (ALB avg) | Service CPU |
|---|---|---|---|---|---|
| Baseline | 1 | 0.25 | ~0 | 7 ms | 4.3–4.8% |
| Under load | **1** | 0.25 | **99.4 req/s** (5,964/min) | **39.9 ms** | 95.4–97.1% |
| Transition | 1→2 | — | 107 req/s | 35.7 ms | 56.0% ← see note |
| Under load | **2** | 0.50 | **186.2 req/s** (11,173/min) | **20.0 ms** | 93.1–94.3% |
| After load | 2 | 0.50 | ~0 | 1.4 ms | 4.3–4.8% |

**Scaling efficiency: 1.87× throughput for 2× compute (93%), with latency halved from 40 ms to
20 ms.** That is close to ideal for a stateless service and confirms the second task took a genuine
share of traffic rather than sitting idle.

The Service Connect metric for the Kong→cengex hop agrees: `TargetResponseTime` fell from 45 ms to
25 ms at the same moment.

> **Note on the 56% CPU dip at 08:07–08:08.** This is not a throttle. `CPUUtilization` is averaged
> across tasks in the service, and for those two minutes the newly launched task was still booting
> and contributing ~0%, dragging the average down. Once it began serving, the average returned to
> ~94%. Don't misread this dip as the service losing capacity.

### Latency distribution

| p50 | p66 | p75 | p90 | p95 | p99 | max |
|---|---|---|---|---|---|---|
| 7 ms | 8 ms | 71 ms | 88 ms | 90 ms | 97 ms | 1100 ms |

The distribution is **bimodal**: about two-thirds of requests completed in 7–8 ms, the rest in
71–97 ms. That gap is a real effect worth following up (see finding F2), not measurement noise.

### "Sessions handled" — stated precisely

The test drove **4 concurrent in-flight requests**. Keep-alive was requested but the report shows
`Keep-Alive requests: 0`, so each of the 120,257 requests opened its own TCP connection — the run
handled **120,257 connections**, roughly 143 new connections/sec sustained, with zero refusals.

Translating that to *concurrent users* requires a think-time assumption, so the following is
**arithmetic, not a measured result**:

| Assumed user behaviour | Concurrent users at 186 req/s (2 tasks) |
|---|---|
| 1 request per user per 5 s | ~930 |
| 1 request per user per 10 s | ~1,860 |

**Treat these as an optimistic ceiling, not a capacity commitment.** The endpoint used costs
**2.45 ms of CPU per request** and touches no database — measured two ways that agree:

- 1 task: 0.25 vCPU × 0.963 ÷ 99.4 req/s = 2.41 ms
- 2 tasks: 0.50 vCPU × 0.937 ÷ 186.2 req/s = 2.50 ms

Real application endpoints will cost substantially more, because `config/db-local.php` sets
`enableSchemaCache => false`, so every DB-touching request re-reads table schemas from RDS. A
realistic capacity number needs a second test against representative endpoints — and that number
will be a fraction of the above. See finding F3.

---

## 2. Timings — what actually governed the test

Authoritative, from `describe-scaling-activities`:

| Event | Time (UTC) | Elapsed |
|---|---|---|
| Load started | 08:01:55 | — |
| CPU first observed > 80% | 08:02 | +5 s |
| `AlarmHigh` → ALARM, `desiredCount` = 2 | **08:07:10** | **+5m15s** |
| Second task RUNNING and serving | **08:09:16** | **+7m21s** |
| Load stopped | 08:15:55 | — |
| `AlarmLow` → ALARM, `desiredCount` = 1 | **08:33:35** | **+17m40s** |
| Scale-in complete | 08:34:07 | +18m12s |

Both fell inside the windows predicted from the alarm configuration (`AlarmHigh` 3×60s > 80,
`AlarmLow` 15×60s < 72). The 30-second cooldowns were irrelevant by comparison — the alarm
evaluation periods are what set the pace.

**Operationally: expect ~5 minutes to react to a traffic spike, and ~18 minutes to give capacity
back.** The five-minute lag is the number that matters for a real traffic surge.

---

## 3. Resource utilisation

| Resource | Baseline | Under load | Note |
|---|---|---|---|
| cengex CPU | 4.3% | **93–97%** | The binding constraint |
| cengex memory | 21.0% | **23.2%** | 512 MB is heavily oversized |
| **kong CPU** | 3.6% | **26–28% @ 100 req/s → 43–46% @ 186 req/s** | **See F1** |
| RDS CPU | 3.6% | **3.6–4.5%** | Flat |
| RDS connections | 0 | **0** | Confirms no DB in the path |

Memory never moved. If you ever want to trim cost, `cengex` at 512 MB could run on 256 MB; CPU is
what needs headroom, not RAM.

---

## 4. Findings

### F1 — Kong is the environment-wide ceiling, and it cannot scale

Kong used **45% of its 0.5 vCPU to proxy 186 req/s for a single backend**. Extrapolating linearly,
it saturates somewhere around **400 req/s of total traffic across all services**.

Kong has **no scaling policy** — it is not registered as a scalable target. So while the 13
application services can each double, the single gateway they all sit behind cannot. Under real
load Kong saturates first, and when it does, *every* route degrades simultaneously, including the
ones whose own services are healthy and idle.

This is the most consequential result of the test. The application-tier autoscaling you just
verified is real, but it is capped by an un-scalable component in front of it.

Recommended: register `kong` as a scalable target with the same policy, and raise it to 1 vCPU.

### F2 — Bimodal latency: a ~90 ms tail on one-third of requests

Two-thirds of requests took 7–8 ms; the remainder took 71–97 ms. A clean CPU-saturation curve
would not produce that gap.

**Hypothesis, not yet verified:** php-fpm is configured `pm = ondemand`, which spawns workers on
arrival and reaps them after an idle timeout. Under sustained load that causes repeated
fork-and-initialise cycles, and a request that arrives needing a fresh worker pays the PHP startup
cost. `pm = static` (or `dynamic` with warm `start_servers`) would keep workers resident and should
flatten the tail.

Worth confirming before acting — it needs a single controlled comparison, not a config change on
faith.

### F3 — The measured capacity is a framework-bootstrap ceiling, not an application number

Deliberate, and stated so it isn't misread: the test endpoint returns a Yii 404 without touching
MySQL, chosen so the test measured container CPU rather than shared RDS. Proven by RDS staying at 0
connections throughout.

That makes 186 req/s an upper bound on the routing and framework layer. It says nothing about the
throughput of `/dashboard/dashboard/dashboard` or any endpoint that queries. Because
`enableSchemaCache => false`, DB-backed endpoints carry a per-request schema-read penalty against a
`db.t4g.medium` shared by all five schemas — that, not container CPU, is the likely real bottleneck.

Recommended sequence: enable schema caching first, then load test a representative endpoint mix.
Testing DB endpoints *before* fixing the cache would mostly measure a misconfiguration.

### F4 — `max 2` limits headroom to 2×

Scaling worked, but the ceiling is one extra task. At 186 req/s the service was already at 94% CPU
with both tasks running — it was at its configured limit while still fully saturated. There is no
third step. For anything resembling production traffic, `max` needs to be higher.

*(The task oscillation I predicted in the plan did not occur, because `ab` is a closed-loop
generator — it pushes as fast as the system responds, so throughput rose to keep both tasks
saturated instead of CPU halving. The prediction still holds for fixed-rate load, e.g. real traffic
at a steady arrival rate. It was not exercised by this test.)*

---

## 5. CloudWatch observability — what is and isn't tracked

You asked whether CloudWatch is enabled to track resource usage. Partly. What exists was enough for
this test, but there is one significant gap.

### Enabled and working

| Source | Metrics | Status |
|---|---|---|
| `AWS/ECS` (service level) | `CPUUtilization`, `MemoryUtilization`, `LiveTaskCount` | **On by default.** This is what autoscaling reads. |
| `AWS/ECS` (Service Connect) | `RequestCount`, `NewConnectionCount`, `ActiveConnectionCount`, `ProcessedBytes`, `TargetResponseTime`, `RequestCountPerTarget`, `HTTPCode_Target_2XX_Count` | **On**, because Service Connect is configured. Gives per-service throughput and service-to-service latency. |
| `AWS/ApplicationELB` | `RequestCount`, `TargetResponseTime`, `HTTPCode_Target_*`, `ConsumedLCUs`, connection counts | **Always on.** |
| `AWS/RDS` | `CPUUtilization`, `DatabaseConnections`, etc. | **On.** |
| CloudWatch Logs | `/ecs/b2g-uat/*`, 7-day retention | **On.** |
| Autoscaling alarms | `AlarmHigh` / `AlarmLow` per service | **On**, auto-created by the policies. |

### The gap: Container Insights is DISABLED

```
aws ecs describe-clusters --clusters b2g-UAT --include SETTINGS
  → containerInsights: disabled
```

The `ECS/ContainerInsights` namespace is empty (0 metrics). Consequences:

- **No per-task CPU or memory.** Only the service-wide average. During this test I could not tell
  whether both tasks were loaded evenly or one was doing most of the work — I had to infer it from
  aggregate throughput. That inference was sound here, but it wouldn't diagnose an unbalanced service.
- **No per-container breakdown** within a task (app vs. Service Connect sidecar).
- **No network/storage metrics** per task.
- The 56% dip in §1 required interpretation for exactly this reason — with per-task metrics it would
  have been self-evident.

Enabling it is one command and costs roughly $2–4/month for this cluster:

```powershell
aws ecs update-cluster-settings --cluster b2g-UAT `
  --settings name=containerInsights,value=enabled
```

It applies to newly launched tasks, so services need a redeploy to start reporting. Worth doing
before you rely on autoscaling in production — diagnosing an uneven or thrashing service without
per-task metrics is guesswork.

### Not configured

- **No CloudWatch dashboard** — every number in this report was pulled ad hoc via CLI.
- **No alarms on anything except autoscaling.** Nothing alerts on 5xx rate, ALB target health, Kong
  saturation (F1), or RDS CPU. If Kong saturates at 3 a.m., nobody is paged.

---

## 6. Reproducing this test

```bash
# Load generator is already installed on the bastion (apache2-utils).
aws ssm start-session --target i-04bed8d526c55e47d

ALB=b2g-uat-alb-105245781.ap-south-1.elb.amazonaws.com
ab -t 840 -n 5000000 -c 4 -r -k "http://$ALB/cengex/web/loadtest-404"
```

Watch from the workstation:

```powershell
aws ecs describe-services --cluster b2g-UAT --services cengex `
  --query "services[0].[runningCount,desiredCount,pendingCount]" --output text

aws application-autoscaling describe-scaling-activities --service-namespace ecs `
  --resource-id service/b2g-UAT/cengex --max-results 5 `
  --query "ScalingActivities[].{start:StartTime,status:StatusCode,desc:Description}" --output table
```

**When querying metrics, always pass `--scan-by TimestampAscending`.** `get-metric-data` returns
datapoints newest-first by default; reading them as oldest-first inverts the trend and makes
successful scaling look like a regression.

Total elapsed for a full cycle is ~35 minutes, dominated by the 18-minute scale-in wait.

---

## 7. Recommended next steps, in priority order

| Priority | Action | Why |
|---|---|---|
| 1 | Add a scaling policy to `kong` and raise it to 1 vCPU | F1 — un-scalable single ingress caps the whole environment at ~400 req/s |
| 2 | Enable Container Insights | No per-task visibility; blocks real diagnosis |
| 3 | Enable `enableSchemaCache`, then load test DB-backed endpoints | F3 — current number says nothing about real application capacity |
| 4 | Raise `max` above 2 | F4 — service was saturated at its ceiling |
| 5 | Alarms on 5xx rate, target health, Kong CPU, RDS CPU | Nothing currently pages anyone |
| 6 | Investigate `pm = ondemand` → `static` | F2 — would remove the 90 ms tail on a third of requests |