Automated Build & Staging Deployment
This document covers the two AWS CodePipelines that together take a
commit on the dev branch of meltano-catalog and end with a freshly
deployed and SIT-tested staging environment. The handoff between them
is automatic; a green run of the second pipeline is the gate that opens
the manual production workflow.
Trigger
Section titled “Trigger”The trigger is any commit to the dev branch of the meltano-catalog
repository on GitHub. An AWS CodeStar / AWS Connection to GitHub watches
the branch and signals the meltano-catalog-build pipeline when a new
commit lands. There is no manual approval step at this stage — every
commit to dev builds.
The mechanism is a GitHub App-based connection rather than a webhook configured per-repository, so the same connection is reused if other AWS pipelines need to observe the same repo. The connection is owned by the AWS account that hosts the build pipeline.
Pipeline 1 — meltano-catalog-build
Section titled “Pipeline 1 — meltano-catalog-build”This pipeline runs the build and produces the container images. It does
not deploy anything; its sole output is the set of images pushed to the
three cloud registries under the latest-dev tag.
Stages
Section titled “Stages”The pipeline executes the following stages in order. Each stage runs
inside CodeBuild under a buildspec checked into meltano-catalog.
Source. The pipeline checks out the commit on dev that triggered
it, via the GitHub Connection.
Test. Maven runs the Java unit tests across all meltano-* modules
in the catalog repo. The frontend test suite (running under the
configured Node toolchain) runs in parallel or sequentially depending
on the buildspec; both must pass for the build to proceed.
Build. Two container images are built from the same workspace:
The catalog image packages the Spring Boot catalog backend together
with the built frontend assets. The shelltask image packages the
Spring Cloud Task shelltask runtime that executes Meltano pipeline
bash jobs. Both images are built from Dockerfiles in meltano-catalog.
Publish. Each image is tagged latest-dev and pushed to all three
cloud registries:
| Cloud | Registry |
|---|---|
| AWS | Elastic Container Registry (ECR) |
| Azure | Azure Container Registry (ACR) |
| Artifact Registry |
The latest-dev tag is a moving tag — every successful build
overwrites it. There is no per-build immutable tag at this stage; the
provenance of any given latest-dev image is whichever commit on dev
most recently completed a successful build pipeline.
Outputs and side effects
Section titled “Outputs and side effects”The visible outputs of this pipeline are the six image pushes (two
images × three registries). The pipeline does not modify any deployed
environment; the staging environment continues to run whatever
latest-dev was deployed previously until the staging pipeline picks
the new images up.
Failure modes worth knowing
Section titled “Failure modes worth knowing”A failure in the Test stage halts the pipeline and no images are published. A failure in the Build stage (e.g. a Dockerfile change that no longer builds) halts the pipeline at that stage; tests have already passed at that point. A failure in the Publish stage is unusual but can leave the registries in an inconsistent state — for example, ECR pushed but ACR not. Re-running the pipeline will overwrite all three registries on success, so the recovery action is to fix the cause and re-trigger.
Pipeline 2 — meltano-cloud-staging
Section titled “Pipeline 2 — meltano-cloud-staging”Once the build pipeline finishes successfully, the staging pipeline is
triggered automatically. It deploys the latest-dev images to the
staging EKS cluster and runs the SIT suite against the result.
The staging environment currently lives in AWS EKS. There is a planned migration to Google Cloud (GKE); when that lands, this document and the pipeline definition will need to be updated together.
How the deployment is assembled
Section titled “How the deployment is assembled”The staging pipeline does not contain the deployment definition itself. Instead, its CodeBuild buildspec clones two additional repositories at runtime and uses them to perform the deploy:
The meltano-build repository supplies the deployment scripts and
Helm charts. These describe how to deploy the catalog and shelltask
components — the resource definitions, service wiring, and parameterised
values. The pipeline clones the appropriate branch (typically the same
branch convention as the rest of the release, i.e. dev for staging).
The meltano-config repository supplies environment-specific
configuration — Helm values overlays, secrets references, ingress
hostnames, and any other staging-specific overrides. The pipeline
clones this alongside meltano-build and points the Helm install at
the staging values file.
Cloning at runtime, rather than baking the charts and config into the
build pipeline output, means that operational changes to the deployment
(e.g. tuning a resource limit or adding a Helm value) do not require a
rebuild of the application — pushing to meltano-build or
meltano-config is sufficient and the next staging deployment will
pick them up.
Deployment target
Section titled “Deployment target”| Setting | Value |
|---|---|
| Cluster | Staging EKS in AWS |
| Image tag deployed | latest-dev from ECR |
| Helm charts | meltano-build |
| Values / config | meltano-config (staging overlay) |
| Planned change | Migrate to Google Cloud (GKE) |
Because the pipeline deploys the moving latest-dev tag, Kubernetes
needs to be told to actually re-pull and roll the pods. The Helm
chart’s image pull policy and rollout strategy are responsible for
this; see meltano-build for the specifics.
Post-deploy: SIT
Section titled “Post-deploy: SIT”The final action of the staging pipeline is to run the system
integration test (SIT) suite against the freshly deployed staging
environment. The SIT suite is owned by the meltano-sit repository,
and its purpose, gating role, and operational considerations are
covered in 02-sit-tests.md.
A green SIT run is what closes out the automated portion of the release pipeline. It is also the precondition for opening a Linear release card and starting the production workflow.