# B2G on AWS — deployment runbook

How the B2G platform runs on AWS ECS Fargate, and how to rebuild it from nothing.

Every phase gives the **AWS Console click-path** and the **equivalent PowerShell verification**, so
you can click through the console to learn the components and still prove the result from a
terminal. Written for a Windows workstation.

- **Account** `022067219328` · **Region** `ap-south-1` (Mumbai)
- **Live URL** http://b2g-uat-alb-105245781.ap-south-1.elb.amazonaws.com
- **Cluster** `b2g-UAT` · **Repo** CodeCommit `b2g_demo`

> Read [Gotchas](#gotchas) before you start. Five of those cost real debugging time; they are the
> difference between this taking a day and taking a week.

---

## Contents

| Phase | What it builds |
|---|---|
| [0](#phase-0--prerequisites) | Workstation prerequisites |
| [1](#phase-1--network-and-security) | VPC, subnets, endpoints, security groups |
| [2](#phase-2--rds-and-the-database-import) | RDS, secrets, schema import |
| [3](#phase-3--source-changes) | The code changes the migration required |
| [4](#phase-4--ecr-and-image-builds) | ECR repos, CodeBuild, 15 images |
| [5](#phase-5--task-definitions-and-services) | Task definitions, Service Connect |
| [6](#phase-6--the-load-balancer) | ALB, target group, listener |
| [7](#phase-7--verification-cost-and-rollback) | Verification, cost, rollback, teardown |
| [CI/CD](#cicd-pipeline) | How code reaches production |
| [Gotchas](#gotchas) | Things that will bite you |

---

## Architecture

```
Internet
   │  HTTP :80
   ▼
ALB  b2g-uat-alb          internet-facing, 2 public subnets (an ALB REQUIRES two AZs)
   │  one target group -> kong:8000, health check GET /healthz
   ▼
Kong 3.9 DB-less          Fargate, private subnet, sg: b2g-kong-sg
   │  path routing baked into the image from docker/kong/kong.yml
   │  upstreams resolved through ECS Service Connect (*.b2g.local)
   ▼
13 Fargate services       sg: b2g-svc-sg
   9 Yii2 PHP backends                4 Angular SPAs (nginx)
   micro      .b2g.local:80/81        fe-b2g       /app
   usermgmt   .b2g.local:80/81        fe-gmarket   /market
   profilemgmt, confmgmt, digitall,   fe-website   /
   cengex, gmarketapi, mail           fe-filems    /filems
   filemst    :81 /82 /84
   │
   ▼
RDS uat-rds-db (MySQL 8.4.9, private)   ·   S3 amzn-s3-b2g-demo (filemst drive, task role)
```

**Everything is same-origin.** The SPAs and the APIs are all served through one Kong behind one ALB,
so the browser never issues a CORS preflight. That is why the per-service Yii `corsOrigins`
allowlists — which still contain ~116 hardcoded LAN IPs — never need touching.

### Routes

| Path | Upstream | Path | Upstream |
|---|---|---|---|
| `/` | fe-website | `/micro/web`, `/micro/api` | micro :80/:81 |
| `/app/` | fe-b2g | `/usermg/web`, `/api` | usermgmt :80/:81 |
| `/market/` | fe-gmarket | `/profilemg/web` | profilemgmt |
| `/filems/` | fe-filems | `/configmg/web` | confmgmt |
| `/healthz` | Kong itself | `/digitall/web` | digitall |
| `/filemst/api`, `/filemst/backend`, `/filemst/storage` | filemst :81/:82/:84 | `/cengex/web` | cengex |
| | | `/gmarketapi/web`, `/mail/web` | gmarketapi, mail |

---

## Phase 0 — Prerequisites

**Console:** nothing. This is workstation setup.

```powershell
# AWS CLI v2
winget install --id Amazon.AWSCLI -e
aws --version                      # expect aws-cli/2.x

aws configure                      # key, secret, region ap-south-1, output json
aws sts get-caller-identity        # expect account 022067219328

# CHECK THIS. If the region comes back empty, every command below fails with
# "You must specify a region" - which is easy to misread as a permissions problem.
aws configure get region           # must print ap-south-1
```

If it is empty, either set it permanently or export it for the session:

```powershell
aws configure set region ap-south-1          # permanent, for the current profile
$env:AWS_DEFAULT_REGION = "ap-south-1"       # this shell only
```

If your account uses named profiles (`aws configure list-profiles`), set
`$env:AWS_PROFILE` too, or append `--profile <name>` to every command.

```powershell

# Session Manager plugin - needed for DB tunnelling and `aws ecs execute-command`
Invoke-WebRequest "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/windows/SessionManagerPlugin.zip" -OutFile "$env:TEMP\smp.zip"
Expand-Archive "$env:TEMP\smp.zip" -DestinationPath "$env:TEMP\smp" -Force
& "$env:TEMP\smp\install.bat"
session-manager-plugin --version

# CodeCommit git helper (the repo remote uses the codecommit:// protocol)
pip install git-remote-codecommit
git clone codecommit::ap-south-1://b2g_demo
```

> If `git clone` fails with `remote-codecommit is not a git command`, the pip Scripts directory is
> not on PATH. Add `%LOCALAPPDATA%\Python\<version>\Scripts`.
>
> Windows also ships Git Credential Manager, which sits *ahead* of the AWS helper in the chain and
> will return a 403 on CodeCommit HTTPS URLs. The `codecommit://` remote sidesteps it entirely.

Docker Desktop is needed only to build images locally; CodeBuild does it in AWS otherwise.

---

## Phase 1 — Network and security

### 1.1 The second public subnet

**An ALB requires subnets in two Availability Zones.** The VPC had only one public subnet, so this
is the first blocker.

**Console:** VPC → Subnets → *Create subnet* → VPC `UAT-testing` (`vpc-0ba5292c6e629d49b`) →
Name `UAT-public-subnet-b`, AZ `ap-south-1b`, CIDR `10.0.2.0/24` → Create.
Then select it → *Route table* tab → *Edit route table association* → choose `UAT-public-rt`
(the one with the `0.0.0.0/0 → igw-…` route).

```powershell
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-0ba5292c6e629d49b" `
  --query "Subnets[].{id:SubnetId,cidr:CidrBlock,az:AvailabilityZone,name:Tags[?Key=='Name']|[0].Value}" `
  --output table
# expect 4 subnets: public-a (1a), public-b (1b), private-a (1a), private-b (1b)
```

Associating with the public route table is the step people forget — a subnet with no IGW route is
private no matter what it is called.

### 1.2 S3 Gateway endpoint (free, do it)

ECR stores image layers in S3. Without this endpoint every image pull is billed as NAT data
processing. A **Gateway** endpoint costs nothing (unlike Interface endpoints at ~$7.20/mo each).

**Console:** VPC → Endpoints → *Create endpoint* → Services `com.amazonaws.ap-south-1.s3`, type
**Gateway** → VPC `UAT-testing` → tick **both** route tables → Create.

```powershell
aws ec2 describe-vpc-endpoints --filters "Name=vpc-id,Values=vpc-0ba5292c6e629d49b" `
  --query "VpcEndpoints[].{svc:ServiceName,type:VpcEndpointType,state:State,rts:RouteTableIds}" --output json
```

### 1.3 Security groups

Four groups that chain, so no rule ever names a raw IP except the public listener:

| Group | Inbound | Purpose |
|---|---|---|
| `b2g-alb-sg` | TCP 80 from `0.0.0.0/0` | the only publicly reachable thing |
| `b2g-kong-sg` | TCP 8000 from `b2g-alb-sg` **and** from `b2g-svc-sg`; TCP 8001–8002 from `Bastion Server-SG` | gateway; admin ports never public |
| `b2g-svc-sg` | TCP 80–82, 84 from `b2g-kong-sg`; TCP 80–84 from **itself** | the 13 app services; self-rule is for east-west |
| `UAT-RDS-SG` | TCP 3306 from `b2g-svc-sg` and from `Bastion Server-SG` | database |

**Console:** EC2 → Security Groups → *Create security group* → in the rule's Source box type `sg-`
and pick the group rather than a CIDR.

> **Check egress too.** `b2g-alb-sg` was created with outbound limited to ports 80/443. The Kong
> target listens on **8000**, so the ALB could not open the connection and every health check timed
> out with no log line anywhere. See [Gotchas](#gotchas).

```powershell
# Inbound
aws ec2 describe-security-groups --group-ids sg-0084b7e74c05bf6d0 sg-02fc5488985346353 sg-0148e52978f27d5a6 sg-0d31a06531cad03bb `
  --query "SecurityGroups[].{name:GroupName,in:IpPermissions[].{p:FromPort,cidr:IpRanges[].CidrIp,sg:UserIdGroupPairs[].GroupId}}" --output json

# Outbound on the ALB group - must include 8000 to the Kong group
aws ec2 describe-security-groups --group-ids sg-0084b7e74c05bf6d0 `
  --query "SecurityGroups[0].IpPermissionsEgress[].{p:FromPort,sg:UserIdGroupPairs[].GroupId,cidr:IpRanges[].CidrIp}" --output json
```

---

## Phase 2 — RDS and the database import

### 2.1 Parameter group

**Console:** RDS → Parameter groups → *Create* → family `mysql8.4`, name `b2g-uat-mysql84` → then
Edit → set `log_bin_trust_function_creators = 1` → Save.
RDS → Databases → `uat-rds-db` → *Modify* → DB parameter group `b2g-uat-mysql84` → Apply immediately.

**Swapping the parameter group needs a reboot** even though the parameter inside it is dynamic:
RDS → Databases → select → Actions → **Reboot**. Do it before any traffic exists.

```powershell
aws rds describe-db-instances --db-instance-identifier uat-rds-db `
  --query "DBInstances[0].{status:DBInstanceStatus,pg:DBParameterGroups[0]}" --output json
# ParameterApplyStatus must read "in-sync", not "pending-reboot"
```

### 2.2 Credentials into Secrets Manager

**Console:** RDS → `uat-rds-db` → Modify → New master password → Apply immediately.
Then Secrets Manager → *Store a new secret* → Other type → key/value:
`engine, host, port, username, password, dbInstanceIdentifier` → name `b2g/uat/rds-master`.

Create a second secret `b2g/uat/db-app` for the least-privilege application user. **The ECS tasks
get this one, never the master.**

```powershell
aws secretsmanager list-secrets --query "SecretList[].Name" --output table
```

### 2.3 Import the dumps

The dumps live at `s3://amzn-s3-b2g-demo/db-backup/`. They came from `mysqldump 8.0.28` against
another RDS instance and **will not import unmodified**:

| In the dump | Why it fails | Fix |
|---|---|---|
| `SET @@SESSION.SQL_LOG_BIN=0` | needs `SUPER`; RDS `admin` has none | strip the line |
| `SET @@GLOBAL.GTID_PURGED=''` | needs `SUPER` | strip the line |
| ``DEFINER=`Sudha`@`%` `` in `bgi_b2g_reg_dev` procedures | that user does not exist here | strip the clause |
| no `CREATE DATABASE` | dumped single-db | create each schema first |

Run the import from the bastion (it is SSM-managed, so no SSH key is needed). Give its instance
role `s3:GetObject` on the backup prefix and `secretsmanager:GetSecretValue` first.

```bash
# on the bastion, via: aws ssm start-session --target i-04bed8d526c55e47d
sudo apt-get install -y mysql-client-core-8.0 jq
SEC=$(aws secretsmanager get-secret-value --region ap-south-1 --secret-id b2g/uat/rds-master --query SecretString --output text)
HOST=$(echo "$SEC" | jq -r .host); USER=$(echo "$SEC" | jq -r .username); PASS=$(echo "$SEC" | jq -r .password)
umask 077; printf '[client]\nhost=%s\nuser=%s\npassword="%s"\n' "$HOST" "$USER" "$PASS" > ~/my.cnf
MYSQL="mysql --defaults-file=$HOME/my.cnf --protocol=TCP"

for db in bgi_b2g_master_dev bgi_b2g_reg_dev bgi_b2g_digitall_dev bgi_b2g_gmarket_dev bgi_b2g_cengex_dev; do
  aws s3 cp "s3://amzn-s3-b2g-demo/db-backup/${db}.sql" /var/tmp/${db}.sql --region ap-south-1
  sed -E -e '/^SET @MYSQLDUMP_TEMP_LOG_BIN/d' \
         -e '/^SET @@SESSION\.SQL_LOG_BIN/d' \
         -e '/^SET @@GLOBAL\.GTID_PURGED/d' \
         -e 's/DEFINER=`[^`]*`@`[^`]*`//g' \
         /var/tmp/${db}.sql > /var/tmp/${db}.clean.sql
  $MYSQL -e "CREATE DATABASE IF NOT EXISTS \`$db\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
  $MYSQL "$db" < /var/tmp/${db}.clean.sql && echo "$db imported"
done

# filemst references these two; no dump exists, create them empty so the app boots
for db in bgi_jsrsv3_audit estore; do
  $MYSQL -e "CREATE DATABASE IF NOT EXISTS \`$db\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
done
```

Then the application user. Grants must span **all** schemas — usermgmt reads
`bgi_b2g_digitall_dev`, and digitall/gmarketapi reference the reg and master schemas:

```bash
$MYSQL -e "CREATE USER IF NOT EXISTS 'b2gapp'@'%' IDENTIFIED BY '<from b2g/uat/db-app>';"
for db in bgi_b2g_master_dev bgi_b2g_reg_dev bgi_b2g_digitall_dev bgi_b2g_gmarket_dev bgi_b2g_cengex_dev bgi_jsrsv3_audit estore; do
  $MYSQL -e "GRANT SELECT,INSERT,UPDATE,DELETE,EXECUTE,SHOW VIEW,CREATE TEMPORARY TABLES ON \`$db\`.* TO 'b2gapp'@'%';"
done
$MYSQL -e "FLUSH PRIVILEGES;"
```

**Verify — expected table counts:**

```sql
SELECT TABLE_SCHEMA, COUNT(*) FROM information_schema.TABLES
WHERE TABLE_SCHEMA LIKE 'bgi_%' OR TABLE_SCHEMA='estore' GROUP BY TABLE_SCHEMA;
```

| Schema | Tables | Used by |
|---|---|---|
| `bgi_b2g_master_dev` | 73 | micro |
| `bgi_b2g_reg_dev` | 70 | usermgmt, profilemgmt, confmgmt, mail, filemst |
| `bgi_b2g_gmarket_dev` | 97 | gmarketapi |
| `bgi_b2g_digitall_dev` | 19 | digitall |
| `bgi_b2g_cengex_dev` | 17 | cengex |
| `bgi_jsrsv3_audit` | 0 | filemst audit — **no dump exists** |
| `estore` | 0 | referenced, never instantiated |

### 2.4 Connecting MySQL Workbench

RDS is private and stays private. Tunnel through the bastion with SSM — **no SSH key and no firewall
change**, and it works from any network:

The inner quotes **must** be backslash-escaped. PowerShell strips plain double quotes when
handing the argument to a native executable, so the CLI receives `{host:[...]}` and fails with
`Invalid JSON: Expecting property name enclosed in double quotes`:

```powershell
aws ssm start-session --region ap-south-1 --target i-04bed8d526c55e47d `
  --document-name AWS-StartPortForwardingSessionToRemoteHost `
  --parameters '{\"host\":[\"uat-rds-db.c5myq6iqolml.ap-south-1.rds.amazonaws.com\"],\"portNumber\":[\"3306\"],\"localPortNumber\":[\"3307\"]}'
```

In Git Bash the unescaped form is the correct one:

```bash
aws ssm start-session --region ap-south-1 --target i-04bed8d526c55e47d \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["uat-rds-db.c5myq6iqolml.ap-south-1.rds.amazonaws.com"],"portNumber":["3306"],"localPortNumber":["3307"]}'
```

Close the tunnel with **Ctrl+C**. Killing the process with `Stop-Process` leaves the session
`Active` server-side; clean those up with
`aws ssm describe-sessions --state Active` then `aws ssm terminate-session --session-id <id>`.

Leave that window open. In Workbench create a **Standard (TCP/IP)** connection — *not* "over SSH",
the tunnel already exists — to `127.0.0.1:3307`, user `b2gapp`. All five databases are on this one
instance; switch schemas in the tree.

```powershell
aws secretsmanager get-secret-value --region ap-south-1 --secret-id b2g/uat/db-app `
  --query SecretString --output text | ConvertFrom-Json | Select-Object username,password
```

---

## Phase 3 — Source changes

Five changes the migration required. Each preserves XAMPP and Docker Desktop behaviour — every new
lookup falls back to the value that was hardcoded before.

**a. DB connection via environment** — `<service>/config/db-local.php` (×8)
`config/web.php` picks its environment from `HTTP_HOST`, not `APP_ENV`, so behind a load balancer
every service resolves to `local`. Conveniently `db-local.php` already named the right schema, so
only host/port/user/password are externalised. **The schema name stays a literal** so a mis-set env
var cannot cross-wire one service to another's database.

**b. Health endpoint** — `docker/php-nginx/nginx.conf` and 5× `nginx-spa.conf`
Nothing in the repo had one, and ALB target groups require it:
```nginx
location = /healthz { access_log off; return 200 "ok\n"; }
```
Exact-match, so it outranks the prefix and regex blocks and never touches the filesystem — it
answers 200 even on ports whose docroot does not exist for that service. Note there are **five**
copies of `nginx-spa.conf`; each SPA Dockerfile copies from its own build context, so the four
per-app copies are the ones that ship.

**c. Kong route table** — `docker/kong/kong.yml`
The committed file was a 3-service stub; the full table was sitting unreferenced in `kong1.yml`.
Restored all 16 services against Service Connect DNS and added a `/healthz` route backed by the
`request-termination` plugin, so the ALB health check reports on the gateway itself rather than
depending on an upstream being up.

**d. Host-agnostic frontends** — 4× `environment.docker.ts`, 4× `Dockerfile`
The bundles baked an absolute gateway URL (`b2g` had a private EC2 IP). Since every SPA and API is
published through one origin, the base is now path-only (`const KONG = ''`), so the images need no
rebuild when the public URL changes.

**e. Runtime directories** — 9 backend Dockerfiles. See [Gotchas](#gotchas).

---

## Phase 4 — ECR and image builds

### 4.1 Repositories

15 repos: `b2g/php-nginx-base`, `b2g/kong`, one per backend, and `b2g/app`, `b2g/gmarket`,
`b2g/website`, `b2g/filems-fe`.

**Console:** ECR → Repositories → *Create* → name e.g. `b2g/micro` → enable *Scan on push*.

### 4.2 Lifecycle policies

Without these, ECR grows forever and blows the ~$4/mo budget line.

**Console:** select repo → *Lifecycle Policy* → *Create rule* → priority 1, tagged, any prefix,
*More than* 5 images → Expire. Add a second rule for untagged after 1 day.

```powershell
aws ecr describe-repositories --query "repositories[].repositoryName" --output table
aws ecr get-lifecycle-policy --repository-name b2g/micro --query lifecyclePolicyText --output text
```

### 4.3 CodeBuild

**Console:** CodeBuild → *Create project* → Source **AWS CodeCommit**, repo `b2g_demo`, branch
`main` → Environment: *Managed image*, Amazon Linux, standard 6.0, **Privileged ✔** (required for
docker-in-docker) → Service role `b2g-codebuild-role` → Buildspec *Insert build commands* →
Artifacts **No artifacts** → Cache **Local**, tick *Docker layer cache* and *Source cache*.

**Build order matters:** `b2g-base-image` first — all 9 backends build `FROM` it. Then everything
else in parallel.

```powershell
aws codebuild list-projects --output table
aws codebuild start-build --project-name b2g-base-image --query "build.id" --output text
aws ecr describe-images --repository-name b2g/micro `
  --query "sort_by(imageDetails,&imagePushedAt)[-1].[imageTags[0],imagePushedAt]" --output text
```

---

## Phase 5 — Task definitions and services

### 5.1 Task definitions

**Console:** ECS → Task definitions → *Create* → Fargate, Linux/X86_64 → Task role `b2g-task-role`,
Execution role `ecsTaskExecutionRole`.

Sizing — total **4.75 vCPU / 9.5 GB** against a 5.25 / 13.5 budget:

| Size | Services |
|---|---|
| 0.5 vCPU / 1 GB | kong, micro, usermgmt, digitall, filemst |
| 0.25 vCPU / 0.5 GB | profilemgmt, confmgmt, cengex, gmarketapi, mail, and all 4 frontends |

Per container:
- **Port mappings need a `name`** — Service Connect keys off it. micro/usermgmt get `web` (80) and
  `api` (81); filemst gets `api` (81), `backend` (82), `storage` (84).
- **Secrets, not environment**, for credentials: `DB_USER` → `<secret-arn>:username::`,
  `DB_PASSWORD` → `<secret-arn>:password::`. The **execution** role needs
  `secretsmanager:GetSecretValue` to inject them.
- **Health check:** `CMD-SHELL wget -q -O /dev/null http://127.0.0.1:80/healthz || exit 1`.
  Use `127.0.0.1`, never `localhost` — see [Gotchas](#gotchas). Kong uses `kong health`.
- **Logs:** awslogs to `/ecs/b2g-uat/<service>`. **Set retention to 7 days** — auto-created groups
  default to *Never expire*.

East-west URLs are plain environment variables:
```
MICRO_BASE_URL=http://micro.b2g.local          MICRO_API_BASE_URL=http://micro-api.b2g.local:81
USERMGMT_BASE_URL=http://usermgmt.b2g.local    PROFILEMGMT_BASE_URL=http://profilemgmt.b2g.local
MAIL_BASE_URL=http://mail.b2g.local            GMARKETAPI_BASE_URL=http://gmarketapi.b2g.local
CENGEX_BASE_URL=http://cengex.b2g.local        DIGITALL_BASE_URL=http://digitall.b2g.local
BSGPACK_BASE_URL=http://micro.b2g.local/master
FILEMST_BASE_URL=http://<alb-dns>/filemst/api      # PUBLIC on purpose - see below
```

> `FILEMST_BASE_URL` must be the **public** URL. usermgmt concatenates it into download links
> returned to the browser (`params['imageurl']`), so an internal name would produce links the
> browser cannot resolve. Every other hop is server-side and uses Service Connect DNS.
>
> Each variable must be **non-empty**. The Yii code merges them with
> `getenv('X') ?: null` inside `array_filter()`, so an empty value silently reverts to the old
> XAMPP `/b2gnew/...` default and self-404s.

### 5.2 Services with Service Connect

**Console:** ECS → Cluster `b2g-UAT` → Services → *Create* → Launch type Fargate → your task
definition → **Networking: private subnets, `b2g-svc-sg`, Public IP OFF** (NAT handles egress; a
public IP in a private subnet does nothing) → **Service Connect: enabled**, namespace `b2g.local`,
and for each port alias set the *Discovery name* and *DNS* — e.g. port `web` → `micro` →
`micro.b2g.local:80`.

Kong is both a **client** (resolving the others) and **discoverable** as `kong.b2g.local:8000`.

```powershell
# describe-services accepts at most 10 names - batch it
aws ecs describe-services --cluster b2g-UAT --services micro usermgmt profilemgmt confmgmt digitall cengex gmarketapi mail filemst kong `
  --query "services[].{svc:serviceName,run:runningCount,state:deployments[0].rolloutState}" --output table
aws ecs describe-services --cluster b2g-UAT --services fe-b2g fe-gmarket fe-website fe-filems `
  --query "services[].{svc:serviceName,run:runningCount}" --output table
```

Prove name resolution from inside a task. Note the **single-quoted** `--command`: in a double-quoted
PowerShell string `$h` would be expanded by PowerShell before the container ever saw it.

```powershell
$T = aws ecs list-tasks --cluster b2g-UAT --service-name usermgmt --query "taskArns[0]" --output text
$probe = 'sh -c ''for h in micro micro-api usermgmt profilemgmt confmgmt digitall cengex gmarketapi mail filemst-api kong; do getent hosts $h.b2g.local >/dev/null 2>&1 && echo "$h RESOLVES" || echo "$h FAIL"; done'''
aws ecs execute-command --cluster b2g-UAT --task $T --container usermgmt --interactive --command $probe
```

---

## Phase 6 — The load balancer

**Console:** EC2 → Load Balancers → *Create* → **Application Load Balancer** → internet-facing,
IPv4 → **both public subnets** → security group `b2g-alb-sg`.

Target group first: EC2 → Target Groups → *Create* → target type **IP addresses** (Fargate has no
instances), protocol HTTP port **8000**, VPC `UAT-testing` → Health check path **`/healthz`**,
healthy 2 / unhealthy 3, interval 15s, timeout 5s. **Do not register targets by hand** — ECS does it.

Listener HTTP:80 → forward to `b2g-kong-tg`. Then set **Attributes → Idle timeout = 120s**: chained
east-west calls take ~5–6s and a cold task is slower while opcache warms.

Finally attach the target group to the Kong **service** (ECS → service → Update → Load balancing),
not to the target group directly.

```powershell
$TG = aws elbv2 describe-target-groups --names b2g-kong-tg --query "TargetGroups[0].TargetGroupArn" --output text
aws elbv2 describe-target-health --target-group-arn $TG --query "TargetHealthDescriptions[].TargetHealth" --output json
# expect: healthy
$DNS = aws elbv2 describe-load-balancers --names b2g-uat-alb --query "LoadBalancers[0].DNSName" --output text
curl.exe -s -o NUL -w "%{http_code}`n" "http://$DNS/healthz"
```

---

## Phase 7 — Verification, cost and rollback

### Route sweep

```powershell
$DNS = aws elbv2 describe-load-balancers --names b2g-uat-alb --query "LoadBalancers[0].DNSName" --output text
$routes = @('/healthz','/','/app/','/market/','/filems/','/micro/web/','/micro/api/','/usermg/web/',
            '/api/','/profilemg/web/','/digitall/web/','/cengex/web/','/gmarketapi/web/',
            '/filemst/api/','/filemst/backend/','/filemst/storage/','/configmg/web/','/mail/web/')
foreach ($r in $routes) {
  $code = curl.exe -s -o NUL -w "%{http_code}" --max-time 60 "http://$DNS$r"
  "{0,-24} {1}" -f $r, $code
}
```

**Expect 16 × 200.** `/configmg/web/` and `/mail/web/` return **500 on their bare root only** — a
pre-existing missing default view, not a deployment fault. Their real action routes work
(`/configmg/web/conf/confengg/ping` returns a proper 401).

### Prove data actually flows

```powershell
# 243 countries out of bgi_b2g_master_dev, through ALB -> Kong -> Service Connect -> micro -> RDS
(curl.exe -s "http://$DNS/micro/web/master/master/getcountrylist" | ConvertFrom-Json).data.Count

# Auth against bgi_b2g_reg_dev. Write the body to a file: PowerShell mangles inline JSON quoting,
# and curl's @file form avoids the escaping entirely. Use -Encoding ascii, NOT utf8 - Windows
# PowerShell 5.1 writes a BOM with utf8 and the server rejects the body as invalid JSON.
'{"username":"nobody@example.invalid","password":"wrong"}' | Set-Content -Encoding ascii $env:TEMP\login.json
curl.exe -s -X POST "http://$DNS/usermg/web/auth/login" -H "Content-Type: application/json" -d "@$env:TEMP\login.json"
# expect {"status":false,"message":"Invalid username or password"} - the app's own answer, which
# proves the usermst_tbl lookup ran. A 500 here would mean the DB connection failed.
```

### Cost

| Line | Est./mo |
|---|---|
| Fargate 4.75 vCPU + 9.5 GB | ~$171 |
| RDS db.t4g.medium + 50 GB gp3 | ~$62 |
| NAT Gateway | ~$44 |
| ALB | ~$20 |
| CodeBuild, CloudWatch, ECR, S3, Secrets | ~$18 |
| **Total** | **~$315** |

Not included and **not** in the budget: `database-2` (db.t3.xlarge, publicly accessible, ~$190/mo)
and the bastion (~$30/mo).

### Rollback

Deploys pin the image to a commit sha, so rolling back is one command:

```powershell
aws ecs list-task-definitions --family-prefix b2g-micro --sort DESC --query "taskDefinitionArns[0:5]" --output table
aws ecs update-service --cluster b2g-UAT --service micro --task-definition b2g-micro:<previous-revision>
aws ecs wait services-stable --cluster b2g-UAT --services micro
```

### Teardown

Reverse order: delete ECS services → deregister task definitions → delete ALB and target group →
delete the CI/CD EventBridge rules and CodeBuild projects → ECR repos → **snapshot then** delete
RDS → NAT Gateway (billed hourly whether used or not) → subnet and endpoints.

---

## CI/CD pipeline

**`main` is the branch that maps to ECS.** Nothing else deploys. Deployment happens when a pull
request is **merged into `main`** — not when a commit lands on a branch, and not on a direct push
to `main`.

```
developer pushes feature/xyz          (no deploy - nothing is triggered by a branch push alone)
        │
        ▼  opens a pull request into main
        │
        ▼  EventBridge  b2g-pr-validate
   CodeBuild b2g-ci      builds + smoke-tests ONLY affected services, pushes nothing to ECR,
                         posts PASS/FAIL onto the pull request
        │
        ▼  human approves (approval rule requires 1) and clicks Merge
        │
        ▼  EventBridge  b2g-main-deploy       <-- fires on the MERGE, not on the commit
   CodeBuild b2g-cd      per service: compare the sha it is RUNNING against main's tip
                         changed  -> build, push <sha>, new task def revision, roll, wait stable
                         unchanged-> skip
```

Everything is driven by [`ci/service-map.json`](../ci/service-map.json); the shared logic is in
[`ci/pipeline.py`](../ci/pipeline.py). **Adding a service is one entry in the map.**

Cascade: a change under `docker/php-nginx/` rebuilds the base image and **all 9 backends** that
derive from it. A change under `docker/kong/` rebuilds only Kong. A docs-only change deploys nothing.

### What triggers a deploy, exactly

`b2g-main-deploy` matches this event pattern:

```json
{
  "source": ["aws.codecommit"],
  "detail-type": ["CodeCommit Pull Request State Change"],
  "resources": ["arn:aws:codecommit:ap-south-1:022067219328:b2g_demo"],
  "detail": {
    "event": ["pullRequestMergeStatusUpdated"],
    "isMerged": ["True"],
    "destinationReference": ["refs/heads/main"]
  }
}
```

Three conditions must all hold, which is what makes "merge only" precise:

| Field | Why it is there |
|---|---|
| `pullRequestMergeStatusUpdated` | The only CodeCommit event that means "a pull request's merge state changed". Creating or updating a PR raises different events and is handled by `b2g-ci`. |
| `isMerged: ["True"]` | The same event also fires when a merge is *attempted and fails*. Without this, a failed merge would deploy. Note it is the **string** `"True"` with a capital T in the CodeCommit event, not a JSON boolean — a lowercase `true` silently never matches. |
| `destinationReference` | Restricts it to PRs targeting `main`. A PR merged into a release or feature branch does not deploy. |

The target passes `sourceVersion: refs/heads/main` rather than a commit id from the event. By the
time this event fires the merge has already landed, so `refs/heads/main` *is* the merge commit, and
resolving the ref avoids having to guess which of the event's several commit ids represents the
merged result. It also passes `TRIGGER`, `PR_ID`, `MERGED_FROM`, `MERGE_OPTION` and `MERGE_AUTHOR`
into the build environment, so the CodeBuild log records which pull request caused the deployment.

### What changed, and the trade-off

Previously the rule was:

```json
"detail-type": ["CodeCommit Repository State Change"],
"detail": { "event": ["referenceUpdated","referenceCreated"],
            "referenceType": ["branch"], "referenceName": ["main"] }
```

That fires on **any** update to `main`. A merge produces a `referenceUpdated`, so merges did deploy —
but so did a direct `git push origin main`, and the event carries nothing that distinguishes the two.
Matching on the pull-request event is the only way to tell them apart.

> **Consequence, stated plainly:** a commit pushed straight to `main` no longer deploys anything.
> That is the point of the change, but it means an emergency hotfix pushed directly will sit on
> `main` un-deployed and the cluster will keep running the previous image. To ship it, either open a
> pull request (even a one-line one) or start the deploy by hand:
> ```powershell
> aws codebuild start-build --project-name b2g-cd --region ap-south-1
> ```
> A manual run is safe and idempotent: `b2g-cd` compares each service against the sha it is actually
> running, so it deploys exactly what is out of date and skips the rest.

### Console walkthrough — inspecting and changing the trigger

**See the current trigger**
1. Console → **Amazon EventBridge** → *Buses* → **Rules** (region **Asia Pacific (Mumbai) ap-south-1**).
2. Click **`b2g-main-deploy`**.
3. The **Event pattern** panel shows the JSON above. The **Targets** tab shows CodeBuild project
   `b2g-cd`, and under *Additional settings* → *Configure input* → **Input transformer** you can see
   the `sourceVersion` and the environment overrides.

**Change it yourself (what I did, in console terms)**
1. EventBridge → Rules → `b2g-main-deploy` → **Edit**.
2. Step 1 *Define rule detail* — leave name and bus (`default`) alone → **Next**.
3. Step 2 *Build event pattern*:
   - Event source: **AWS events or EventBridge partner events**
   - Creation method: **Custom pattern (JSON editor)**
   - Paste the JSON above → **Next**.
     (The visual *Pattern form* builder cannot express `isMerged` or `destinationReference`, so the
     JSON editor is required here.)
4. Step 3 *Select targets*:
   - Target: **CodeBuild project** → `b2g-cd`
   - Execution role: **Use existing role** → `b2g-events-invoke-codebuild`
   - *Additional settings* → Configure target input → **Input transformer** → **Configure input transformer**
     - Input path:
       ```json
       {"pr":"$.detail.pullRequestId","src":"$.detail.sourceReference","author":"$.detail.author","mergeopt":"$.detail.mergeOption"}
       ```
     - Template:
       ```json
       {"sourceVersion":"refs/heads/main","environmentVariablesOverride":[{"name":"TRIGGER","value":"pr-merge","type":"PLAINTEXT"},{"name":"PR_ID","value":"<pr>","type":"PLAINTEXT"},{"name":"MERGED_FROM","value":"<src>","type":"PLAINTEXT"},{"name":"MERGE_OPTION","value":"<mergeopt>","type":"PLAINTEXT"},{"name":"MERGE_AUTHOR","value":"<author>","type":"PLAINTEXT"}]}
       ```
5. **Next** → **Next** → **Update rule**.

**Confirm `main` is the branch wired to ECS**
1. Console → **CodeBuild** → *Build projects* → **`b2g-cd`** → **Edit** → *Source*.
2. **Source version** must read `refs/heads/main`. This is the branch the deploy builds from; the
   EventBridge input transformer sets the same value per-run.
3. Same check on **`b2g-ci`** — its per-run `sourceVersion` is overridden to the PR's source commit,
   so its project-level value only matters for manually started builds.

**Watch a deployment happen**
1. Console → **CodeCommit** → *Repositories* → `b2g_demo` → **Pull requests** → open yours →
   **Merge** → pick a merge strategy → **Merge pull request**.
2. Console → **CodeBuild** → *Build history*. A `b2g-cd` build appears within seconds; its
   **Initiator** reads `rule/b2g-main-deploy`.
3. Open the build → **Build logs**. Near the top, `cat /tmp/plan.json` shows exactly which services
   were selected and why (`sources changed` or `base image changed`); everything else is skipped.
4. Console → **ECS** → *Clusters* → `b2g-UAT` → the rolled services show a new task-definition
   revision pinned to the merge commit's sha.

### Optionally: stop direct pushes to `main` altogether

The trigger change means a direct push no longer deploys, but it does not *prevent* one. To make the
pull-request path the only way code reaches `main`, attach a deny to the developers' IAM
group/role — CodeCommit approval rules apply to pull requests only and cannot block a push:

```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Deny",
    "Action": ["codecommit:GitPush"],
    "Resource": "arn:aws:codecommit:ap-south-1:022067219328:b2g_demo",
    "Condition": {
      "StringEqualsIfExists": {
        "codecommit:References": ["refs/heads/main"]
      },
      "Null": { "codecommit:References": "false" }
    }
  }]
}
```

**Not applied.** It would also block the merge commit for anyone whose merge is performed
client-side, and it locks out administrators, so it is a team-policy decision rather than part of the
pipeline. Apply it to the developer group only, never to the role that performs merges.

### Verified end to end

The trigger change was exercised on a real branch, pull request and merge, not just configured:

| Check | Result |
|---|---|
| Direct `git push origin main` (docs commit) | **No build started.** Build count unchanged over 75s. This is the behaviour that changed. |
| Branch push `test/merge-trigger` | No deploy, as intended — a branch push alone triggers nothing |
| Opening the pull request | `b2g-ci` fired, selected **cengex only** (1 of 14), `/healthz` and app-root boot checks passed, posted **CI PASSED** onto the PR |
| Merge attempt with no approval | Refused: `PullRequestApprovalRulesNotSatisfiedException` — the gate is real |
| After merge | `b2g-cd` fired with initiator `rule/b2g-main-deploy`, plan selected **cengex only** (`base: false`), deployed the merge commit `0f9db46c` |
| Build environment on that run | `TRIGGER=pr-merge`, `PR_ID=1`, `MERGED_FROM=refs/heads/test/merge-trigger`, `MERGE_OPTION=THREE_WAY_MERGE`, `MERGE_AUTHOR=…user/Balaji` |
| Manual `start-build` after a direct push | Deployed exactly the one stale service and skipped the other 13 |

Two behaviours worth knowing, both discovered during that run:

- **An author cannot approve their own pull request.** CodeCommit rejects it with
  `PullRequestCannotBeApprovedByAuthorException`. The single-approval rule therefore always
  requires a second person; it is not a formality that the author can satisfy alone.
- **There is an admin override**, which is how a genuinely blocked merge gets through:
  ```powershell
  $REV = aws codecommit get-pull-request --pull-request-id <id> --query "pullRequest.revisionId" --output text
  aws codecommit override-pull-request-approval-rules --pull-request-id <id> --revision-id $REV --override-status OVERRIDE
  ```
  In the console: open the pull request → **Approvals** tab → *Override approval rules*. It is
  recorded in the PR activity, so it is auditable rather than silent. This was used once to complete
  the trigger test above, since the same identity had authored the PR.

**Known limitation:** CodeCommit has no GitHub-style *required status check*. The approval rule
enforces a human approval but cannot itself require CI to be green — the approver reads the CI
comment. To make it a hard gate, add the CI role to the approver pool and require 2 approvals.

---

## Gotchas

Five things that cost real debugging time here. Each failed at **runtime**, not build time.

**1. No `runtime/` directory in the backend images.**
Yii needs `runtime/` to exist and be writable. It is not tracked in git and `.dockerignore` excludes
`**/runtime/*`, so `COPY . /app` produced images without it and the app fataled on its first log
write. The original `chown … 2>/dev/null || true` swallowed the error. Create the directories
explicitly in the Dockerfile.

**2. Health checks must use `127.0.0.1`, not `localhost`.**
nginx binds IPv4 only (`0.0.0.0:80`), but `/etc/hosts` maps `localhost` to both `127.0.0.1` and
`::1`. busybox `wget` tries IPv6 first and gets connection-refused. With `localhost` every one of
the 13 services would crashloop, and it would look like an application fault.

**3. Check load-balancer *egress*, not just ingress.**
`b2g-alb-sg` allowed outbound on 80/443 only. The target is on **8000**, so the ALB could not open
the connection — health checks timed out and Kong's access log showed nothing at all, because no
request ever arrived. If a target is unhealthy and the container log is silent, suspect the network
path, not the app.

**4. `kong:3.9` ships neither `wget` nor `curl`.**
An HTTP health check inside that container can never pass. Use `kong health`, which checks the nginx
master process.

**5. filemst does not use the shared DB loader.**
`common/config/base.php` picks a `backend/json/DB*.json` descriptor by `HTTP_HOST` and builds the
DSN from it — so it kept pointing at the on-prem LAN MySQL while all eight siblings were correctly
on RDS. `/filemst/api/` still returned 200 because the root route never touches the database, which
made it easy to miss. It now takes `DB_HOST`/`DB_USER`/`DB_PASSWORD` from the environment.

**And one for CodeBuild:** a buildspec command containing an unquoted `word: word` is parsed by YAML
as a mapping and the whole file is rejected before anything runs. Single-quote any command with a
colon in it — `- 'if ...; then echo "FAIL: x"; fi'`.

### Environment limitations

Consequences of where this runs, not defects in the deployment:

- **On-prem integrations are unreachable** from the VPC: Bridge API `192.168.1.77:8380`, AI services
  `192.168.1.216:9000/9005`, translate API `192.168.1.124`. Those features error. Needs a
  VPN/Direct Connect.
- **`b2g_filems_fe` shows partial data** — six of its nine base URLs point at external GBF services
  on `192.168.1.x`.
- **filemst audit is inert** — `bgi_jsrsv3_audit` has no dump.
- **`/subscriptnmg/web` and thawani checkout 404** — those services are not in the repo.
- **Cron jobs do not run** — `b2g_usermgmt/cron/*.bat`, profilemgmt's file sync and digitall's ESG
  sync are not containerised. They need EventBridge Scheduled Tasks.
- **Uploads outside filemst are ephemeral** — lost on redeploy. EFS is the fix if that matters.
- **`b2g_gmarketapi/modules/masterdata/Module.php` is missing from the repo** — only its controller
  was committed, so `/gmarketapi/web/masterdata/*` has been broken in every environment including
  XAMPP. Dead code: the SPAs call micro directly.
- **Secrets remain in git history** — the shared `jwtSecret`, `encryptionKey`, Keycloak admin
  secret, SMTP and reCAPTCHA keys. Only the DB credentials were moved to Secrets Manager.