Tutorial: Multi-Region CI/CD Pipeline — Build, Scan, Deploy to Azure Container Apps

This tutorial documents a multi-region CI/CD pipeline that builds container images per region on Kubernetes-based agents (Kaniko), runs unit tests, code and container scans (Sonar, Snyk, Trivy/Grype), deploys to Azure Container Apps per region, then runs DAST and technical verification tests (TVT) before creating a release tag and cleaning up.

Pipeline at a glance

The pipeline runs in 10 stages in sequence:

Order	Stage	Purpose
1	Prepare Environment Builder	Scale up K8s build-farm agents (e.g. to 10) for parallel builds.
2	Run Unit Tests	Maven unit tests (optional; can continue on error).
3	Code Base Scan	SonarQube, Snyk SAST, optional Trivy golden SBOM.
4	Build and Containerising	One job per region: Kaniko builds and pushes image to ACR per region.
5	Scan docker image via Trivy	Snyk CSS, Grype SBOM scan (optional; can continue on error).
6	Deployment api app to container app	Deploy image to Azure Container Apps per region.
7	Services scan and verify	DAST per region.
8	Technical Verify Test	TVT per region to verify deployment.
9	Create git tag	Create release tag.
10	Environment Clean Up	Scale down build-farm agents (e.g. to 3).

Example pipeline run: All stages run in sequence. Some jobs (e.g. unit tests, Grype SBOM) may be set to continueOnError: true, so the pipeline can still succeed with warnings or partial failures.

Pipeline run: Prepare Environment Builder, Run Unit Tests, Code Base Scan, Build and Containerising, Scan docker image via Trivy, Deployment, Services scan and verify, Technical Verify Test, Create git tag, Environment Clean Up

Triggers and parameters

Branches: Pipeline runs on push to master or main.

trigger:
  - master
  - main

Parameters control which regions run and which features are enabled:

Parameter	Type	Default	Purpose
`regional_codes`	object	sz, en, au, hk, eu, sgp, br, ca, us, ind	Regions used for build (Kaniko) and DAST (one job per code).
`container_app_deploy`	object	Same 10 codes	Regions where the app is deployed to Azure Container Apps.
`tvt_regions`	object	sw, uk, au, hk, eu, sgp, br, ca, us, ind	Regions for Technical Verify Test (can differ from deploy list).
`enable_sonar`	boolean	true	Run SonarQube code scan.
`enable_trivy`	boolean	false	Run Trivy scanner (golden SBOM) in Code Base Scan stage.
`enable_snyk_sast`	boolean	true	Run Snyk SAST.
`enable_snyk_css`	boolean	true	Run Snyk CSS in container-scan stage.
`enable_sbom`	boolean	true	Run Grype SBOM container scan.
`unbreak_sbom_scan`	boolean	true	Whether SBOM scan failure fails the job.
`enable_tests`	boolean	true	Run Maven unit tests.

Pool and variables

Pool: azk8s-agents — self-hosted Kubernetes agents for build/deploy stages that need K8s (e.g. Kaniko, build farm control). Jobs that use hosted agents specify pool: name: "Azure Pipelines".
Variable groups: azure_prod_acr_secret, CICD_BUILD_CONTROL (for secrets and build control).
Key variables: tag (Build.BuildId), dockerRegistryServiceConnection, MANIFEST_URI, KANIKO_MANIFEST_FILE, TRIVY_MANIFEST_FILE, DOCKERFILE_URI, CONNECT_K8S_SERVICES, CICD_NAMESPACE, BUILD_NAME, BRANCH_NAME, REPO_NAME, SYSTEM_ACCESSTOKEN, GIT_COMMIT, etc. Templates receive these so Kaniko, deployment, and other steps use the same ACR, namespace, and commit.

Stage 1: Prepare Environment Builder

Display name: Prepare Environment Builder

Purpose: Bring up the build farm (K8s agents) so enough parallel jobs can run in the Build and Containerising stage.

Job: init_agents_build_farm — runs template templates/AgentsBuildframcontrol.yaml with AGENT_SCALE: 10, plus CONNECT_K8S_SERVICES and CICD_NAMESPACE.

Stage 2: Run Unit Tests

Display name: Run Unit Tests

Purpose: Run Maven unit tests on a Microsoft-hosted agent (Azure Pipelines pool). Optional and can continue on error so the pipeline does not block on test failures.

Condition: eq('${{ parameters.enable_tests }}', true).
Job: run_unit_tests (display name: Maven Unit Tests), continueOnError: true.
Steps: Verify Java 17, Maven dependency:go-offline (with continueOnError: true), then templates/RunTests.yaml with MAVEN_OPTS: "-P prod".

Stage 3: Code Base Scan

Display name: Code Base Scan

Purpose: Static analysis and early container/SBOM scan — SonarQube, Snyk SAST, and optionally Trivy golden SBOM. All jobs can run in parallel.

Job	Condition	Template / task	Notes
SonarScan	`enable_sonar`	SonarScan.yaml	Sonar project key, host URL, token, inclusions/exclusions.
SnykSAST	`enable_snyk_sast`	SnykSAST.yaml	Uses repo name and Git commit.
Trivy Scanner Golden SBOM	`enable_trivy`	TrivyScanHostedVM.yaml	`continueOnError: true`.

Stage 4: Build and Containerising

Display name: Build and Containerising

Purpose: Build a Docker image per region using Kaniko on the K8s build farm and push to Azure Container Registry (ACR). One job per entry in parameters.regional_codes.

Loop: ${{ each value in parameters.regional_codes }}.
Job name: docker_build_and_public_aws_ecr_region_${{ value }} (display name: e.g. “Kaniko - Server: sz Build and release image”).
Template: templates/KanikoDockerized.yaml with ACR URI, REGIONAL_CODE: ${{ value }}, Dockerfile path, K8s connection, namespace, Git token/PAT, repo/branch/commit, and TAG: $(tag).

Result: One image per region in ACR (e.g. miacrprd.azurecr.io/sz-repo:$(GIT_COMMIT)).

Stage 5: Scan docker image via Trivy

Display name: Scan docker image via Trivy

Purpose: Scan the built container image — Snyk Container (Snyk CSS) and Grype SBOM. Some jobs use continueOnError: true so the pipeline can succeed even if a scan fails.

Job	Condition	Template	Notes
Snyk CSS	`enable_snyk_css`	SnykCSS.yaml	Image name (e.g. us-repo), Dockerfile path.
Grype SBOM	`enable_sbom`	GrypeSBOM.yaml	e.g. image `ca-repo`, `REGIONAL_CODE: ca`, `UNBREAK_SBOM_SCAN: ${{ parameters.unbreak_sbom_scan }}`, `continueOnError: true`.

Stage 6: Deployment api app to container app

Display name: Deployment api app to container app

Purpose: Deploy the regional image to Azure Container Apps — one job per region in parameters.container_app_deploy.

Loop: ${{ each value in parameters.container_app_deploy }}.
Job name: image_deployment_app_${{ value }} (display name: e.g. “region sz - container app deployment image”).
Task: AzureContainerApps@1 — subscription, containerAppName: amdtx-be-api-${{ value }}, resourceGroup: BackendAPIApps, imageToDeploy: "miacrprd.azurecr.io/${{ value }}-repo:$(GIT_COMMIT)".

Stage 7: Services scan and verify (DAST)

Display name: Services scan and verify

Purpose: Run DAST (dynamic application security testing) against the deployed app in each region.

Loop: ${{ each value in parameters.regional_codes }}.
Job name: Dast_scan_region_${{ value }} (display name: e.g. “Server: sz Docker image scan”).
Template: templates/DASTScan.yaml with REGIONAL_CODE: ${{ value }}. Uses Azure Pipelines hosted pool.

Stage 8: Technical Verify Test (TVT)

Display name: Technical Verify Test

Purpose: Run technical verification tests (e.g. smoke/health checks) against the deployed endpoints per region. Region list can differ from deploy list (tvt_regions vs container_app_deploy).

Loop: ${{ each value in parameters.tvt_regions }}.
Job name: TVT_test_endpoint_region_${{ value }} (display name: e.g. “Server: sw TVT Test deployment result”).
Template: templates/TechnicalVerifyTest.yaml with REGIONAL_CODE: ${{ value }}.

Stage 9: Create git tag

Display name: Create git tag

Purpose: Create a Git tag for the release (e.g. production release tag).

Job: release_git_tag — runs templates/GitTagRelease.yaml.

Stage 10: Environment Clean Up

Display name: Environment Clean Up

Purpose: Scale down the build farm to save cost when the pipeline is done.

Job: scale_down_agents_build_frame — runs templates/AgentsBuildframcontrol.yaml with AGENT_SCALE: 3 (and same K8s connection/namespace).

Flow diagram

    trigger (master/main)
            |
            v
    +-------+--------+   +----------------+   +-------------------------+
    | Prepare Env    |   | Run Unit Tests |   | Code Base Scan          |
    | (scale agents) |   | (Maven)        |   | Sonar, Snyk SAST,       |
    | 1 job          |   | 1 job          |   | Trivy golden (optional) |
    +-------+-+------+   +-------+--------+   +------------+------------+
            |                   |                           |
            +-------------------+---------------------------+
                                |
                                v
    +-------------------------------------------------------------------+
    | Build and Containerising (Kaniko per region)                      |
    | 10 jobs: sz, en, au, hk, eu, sgp, br, ca, us, ind                 |
    +------------------------------+------------------------------------+
                                   |
                                   v
    +------------------------------+------------------------------------+
    | Scan docker image (Snyk CSS, Grype SBOM)                          |
    +------------------------------+------------------------------------+
                                   |
                                   v
    +-------------------------------------------------------------------+
    | Deployment api app to container app (Azure Container Apps)        |
    | 10 jobs: one deploy per region                                    |
    +------------------------------+------------------------------------+
                                   |
            +----------------------+----------------------+
            v                      v                      v
    +----------------+    +----------------+    +------------------+
    | DAST per region|    | TVT per region |    | Create git tag   |
    | 10 jobs        |    | (tvt_regions)  |    | 1 job            |
    +-------+-+------+    +--------+-------+    +--------+---------+
            |                     |                      |
            +---------------------+----------------------+
                                  |
                                  v
                    +---------------------------+
                    | Environment Clean Up      |
                    | (scale down agents)       |
                    +---------------------------+

Pipeline YAML (key structure)

Below is the top-level structure of the pipeline: triggers, parameters, pool, variables, and stage list. Each stage uses templates under templates/ for the actual steps.

trigger:
  - master
  - main

parameters:
  - name: regional_codes
    type: object
    default: [sz, en, au, hk, eu, sgp, br, ca, us, ind]
  - name: container_app_deploy
    type: object
    default: [sz, en, au, hk, eu, sgp, br, ca, us, ind]
  - name: tvt_regions
    type: object
    default: [sw, uk, au, hk, eu, sgp, br, ca, us, ind]
  - name: enable_sonar
    type: boolean
    default: true
  # ... enable_trivy, enable_snyk_sast, enable_snyk_css, enable_sbom, unbreak_sbom_scan, enable_tests

pool:
  name: azk8s-agents

variables:
  - group: azure_prod_acr_secret
  - group: CICD_BUILD_CONTROL
  - name: tag
    value: "$(Build.BuildId)"
  - name: DOCKER_BUILDKIT
    value: 1
  - name: dockerRegistryServiceConnection
    value: "acr-prod-env"
  - name: MANIFEST_URI
    value: kubernetes-deploy/deploy.yaml
  - name: KANIKO_MANIFEST_FILE
    value: kubernetes-deploy/kaniko-job.yaml
  - name: DOCKERFILE_URI
    value: deploy/DockerfileKanikoJava17
  - name: CONNECT_K8S_SERVICES
    value: "aks-cicd-agent"
  - name: CICD_NAMESPACE
    value: "az-cicd-agent-ns"
  # ... BUILD_NAME, BRANCH_NAME, REPO_NAME, SYSTEM_ACCESSTOKEN, GIT_COMMIT

stages:
  - stage: PrepareEnvironmentBuilder
    displayName: Prepare Environment Builder
    jobs:
      - job: "init_agents_build_farm"
        steps:
          - template: templates/AgentsBuildframcontrol.yaml
            parameters:
              CONNECT_K8S_SERVICES: $(CONNECT_K8S_SERVICES)
              CICD_NAMESPACE: $(CICD_NAMESPACE)
              AGENT_SCALE: 10

  - stage: RunTests
    displayName: "Run Unit Tests"
    jobs:
      - job: "run_unit_tests"
        condition: eq('${{ parameters.enable_tests }}', true)
        pool: { name: "Azure Pipelines" }
        continueOnError: true
        steps:
          - template: templates/RunTests.yaml
            parameters:
              MAVEN_OPTS: "-P prod"

  - stage: CodeBaseScan
    displayName: Code Base Scan
    jobs:
      - job: SonarScan
        condition: eq('${{ parameters.enable_sonar }}', true)
        # ... SonarScan.yaml
      - job: SnykSAST
        condition: eq('${{ parameters.enable_snyk_sast }}', true)
        # ... SnykSAST.yaml
      # ... Trivy golden SBOM

  - stage: docker_containzer_regional_server
    displayName: Build and Containerising
    jobs:
      - ${{ each value in parameters.regional_codes }}:
          - job: "docker_build_and_public_aws_ecr_region_${{ value }}"
            steps:
              - template: templates/KanikoDockerized.yaml
                parameters:
                  ACR_URI: $(containerRegistry)
                  REGIONAL_CODE: ${{ value }}
                  # ... DOCKERFILE_URI, K8s, Git, TAG

  - stage: Golden_Container_Image_Scan
    displayName: "Scan docker image via Trivy"
    jobs:
      - job: SnykCSS
        condition: eq('${{ parameters.enable_snyk_css }}', true)
        # ... SnykCSS.yaml
      - job: "Grype_SBOM_Container_Image_Scan"
        condition: eq('${{ parameters.enable_sbom }}', true)
        continueOnError: true
        # ... GrypeSBOM.yaml

  - stage: docker_deployment
    displayName: "Deployment api app to container app"
    jobs:
      - ${{ each value in parameters.container_app_deploy }}:
          - job: "image_deployment_app_${{ value }}"
            steps:
              - task: AzureContainerApps@1
                inputs:
                  azureSubscription: "Production Subscription (...)"
                  containerAppName: amdtx-be-api-${{ value }}
                  resourceGroup: BackendAPIApps
                  imageToDeploy: "miacrprd.azurecr.io/${{ value }}-repo:$(GIT_COMMIT)"

  - stage: DAST_SCAN
    displayName: "Services scan and verify"
    jobs:
      - ${{ each value in parameters.regional_codes }}:
          - job: "Dast_scan_region_${{ value }}"
            steps:
              - template: templates/DASTScan.yaml
                parameters:
                  REGIONAL_CODE: ${{ value }}

  - stage: TVT_regional_deployment
    displayName: "Technical Verify Test"
    jobs:
      - ${{ each value in parameters.tvt_regions }}:
          - job: "TVT_test_endpoint_region_${{ value }}"
            steps:
              - template: templates/TechnicalVerifyTest.yaml
                parameters:
                  REGIONAL_CODE: ${{ value }}

  - stage: Git_tag_release
    displayName: "Create git tag"
    jobs:
      - job: "release_git_tag"
        steps:
          - template: templates/GitTagRelease.yaml

  - stage: environment_clean_up
    displayName: Environment Clean Up
    jobs:
      - job: "scale_down_agents_build_frame"
        steps:
          - template: templates/AgentsBuildframcontrol.yaml
            parameters:
              AGENT_SCALE: 3

Summary

Aspect	Detail
Trigger	Branches `master`, `main`.
Agents	Self-hosted pool `azk8s-agents` for Kaniko and farm control; Azure Pipelines for tests and some scans.
Regions	Build/deploy/DAST use `regional_codes` / `container_app_deploy`; TVT uses `tvt_regions` (can include sw, uk, etc.).
Build	Kaniko on K8s, one image per region in ACR (`{region}-repo:$(GIT_COMMIT)`).
Scans	Sonar, Snyk SAST/CSS, Trivy, Grype SBOM — toggled by parameters; some jobs `continueOnError: true`.
Deploy	Azure Container Apps via `AzureContainerApps@1` per region.
Post-deploy	DAST then TVT per region; then git tag and scale-down.

This pipeline gives you a single YAML-driven flow for build → scan → deploy → verify → tag → cleanup across multiple regions, with optional stages and configurable region lists.

quyennv.com