Build, Release & Deployment - Overview

This set of documents describes how the Meltano application is built, tested, deployed, and promoted from development through to production. It is an engineer-facing reference: it names the concrete repos, pipelines, registries, and clusters involved, and is intended to be useful both for someone running a release and for someone troubleshooting one.

The application is a multi-component system, with the bulk of the source living in a single repository (meltano-catalog) and a number of companion repositories that supply build glue, environment configuration, and supporting tooling.

Scope

This document covers the path from a code change merged into the dev branch of meltano-catalog through to the staging environment in AWS, and onward into the manual production release workflow that culminates in a deployment to Azure AKS. Some later production steps are not yet documented and are flagged as TODO at the end of the production checklist.

Components being built and deployed

The meltano-catalog repository contains three distinct deliverables that are produced from a single build:

The catalog backend is a Java/Spring application that exposes the REST API and hosts the connector registry. The shelltask runtime is a Spring Cloud Task component that executes Meltano pipelines as bash jobs. The application frontend is the React UI that talks to the catalog backend. The Java modules and the frontend are built together, and the build produces two container images: one for the catalog backend (with the frontend assets bundled or served alongside) and one for the shelltask runner.

High-level flow

The pipeline has three stages, each gating the next.

The automated build stage is triggered by any commit to the dev branch of meltano-catalog. It runs unit tests, builds the container images, and publishes them to all three cloud registries (AWS ECR, Azure ACR, Google Artifact Registry) under the moving tag latest-dev.

The staging deployment stage is triggered automatically as soon as the build stage finishes. It deploys the freshly built images to the staging EKS cluster in AWS, then runs the SIT (system integration test) suite against that environment. A green SIT run is the gate that opens the production workflow.

The production release stage is a manual workflow, coordinated via a Linear card created from the New Release Template. It involves merging dev → master across eight repositories, which triggers an Azure ADO build pipeline producing immutably-tagged images, followed by a manual Azure ADO Release that deploys those images to the production AKS cluster.

   ┌──────────────────────┐
   │ commit to dev branch │
   │   (meltano-catalog)  │
   └──────────┬───────────┘
              │
              ▼
   ┌──────────────────────────────────────────┐
   │ AWS CodePipeline: meltano-catalog-build  │
   │  • Maven + frontend unit tests           │
   │  • Build catalog + shelltask images      │
   │  • Push as latest-dev → ECR, ACR, AR     │
   └──────────┬───────────────────────────────┘
              │
              ▼
   ┌──────────────────────────────────────────┐
   │ AWS CodePipeline: meltano-cloud-staging  │
   │  • Deploy latest-dev to staging EKS      │
   │  • Helm charts from meltano-build        │
   │  • Env config from meltano-config        │
   │  • Run SIT suite against staging         │
   └──────────┬───────────────────────────────┘
              │ (SIT green)
              ▼
   ┌──────────────────────────────────────────┐
   │ Production release (manual, Linear card) │
   │  • PR dev → master in 8 repos            │
   │  • Azure ADO build → ACR (build-number   │
   │    tag, immutable)                       │
   │  • Manual Azure ADO Release → AKS prod   │
   │  • Further steps TODO                    │
   └──────────────────────────────────────────┘

Repository roles

The release process spans eight repositories. Their roles in the pipeline are summarised below; the production checklist covers what happens in each at release time.

Repository	Role
`meltano-catalog`	Java backend, shelltask, frontend — the primary source repo. Drives both the AWS build pipeline (on `dev`) and the Azure build pipeline (on `master`).
`meltano-www`	Marketing / public website. Merged as part of every release.
`meltano-build`	Deployment scripts and Helm charts. Cloned at deploy time by both the staging and production pipelines.
`meltano-config`	Environment-specific configuration (values, secrets references, per-cluster overrides). Cloned at deploy time alongside `meltano-build`.
`meltano-sit`	System integration test suite executed against staging by the staging pipeline (and re-run as part of release where applicable).
`meltano-docs`	Documentation site. Merged as part of every release so docs ship in lock-step.
`meltano-cli`	Command-line tool. Merged as part of every release.
`meltano-ce`	Community edition. Merged as part of every release.

Registry layout and image tagging

Container images are published to three cloud registries so that deployments can run in any of the three target clouds without cross-cloud image pulls. The same image content is pushed to all three.

For development builds (the AWS CodePipeline stage), images are tagged latest-dev and the tag is overwritten on every build. There is no build-number traceability for latest-dev images by design — they are the moving target that staging always tracks.

For production builds (the Azure ADO stage), images are tagged with the Azure build number, which is immutable. Production deployments always reference a specific build-number tag.

Where to go next

Read 01-automated-build-and-staging.md for the AWS-side automation (build + staging deploy), then 02-sit-tests.md for the gating SIT run, and finally 03-production-release-checklist.md for the manual production workflow.