System Integration Tests
The SIT suite is the gate between the automated portion of the release pipeline and the manual production workflow. It runs against the staging environment immediately after each staging deployment, and a fully green run is the trigger for the team to begin a production release.
What SIT covers
Section titled “What SIT covers”System integration tests exercise the deployed application end-to-end against a real staging environment. They go beyond the unit tests run inside the build pipeline by validating that the catalog backend, the shelltask runner, and the frontend all behave correctly when wired together with their real dependencies — the staging database, the staging object storage, the staging container registry pulls, ingress, authentication, and the Kubernetes plumbing that connects them.
The suite is owned by the meltano-sit repository, which contains both
the test cases and the harness that executes them.
When and where SIT runs
Section titled “When and where SIT runs”SIT runs as the final stage of the meltano-cloud-staging CodePipeline,
after the Helm deployment to staging EKS has completed and the
application is reachable. The pipeline’s CodeBuild step that owns this
stage clones meltano-sit, configures it against the staging
environment, and executes the suite.
Running SIT inside the staging pipeline (rather than as a separate pipeline) has two consequences worth knowing. The pipeline’s overall status reflects SIT outcome — a red SIT run shows up as a red pipeline, which is visible at a glance. And no release card should be opened unless and until the most recent staging pipeline run is fully green.
What a green SIT means — and doesn’t mean
Section titled “What a green SIT means — and doesn’t mean”A green SIT run means the most recent latest-dev images, deployed to
staging EKS using the current contents of meltano-build and
meltano-config, pass the integration suite. That is a precondition
for production release but not a guarantee of production correctness:
The staging environment is not bit-for-bit identical to production. It runs on AWS EKS today (with a Google Cloud migration planned); production runs on Azure AKS. Differences in cluster configuration, networking, managed-service versions, or data shape are not covered by SIT, and are caught — or missed — only at production deploy time.
The staging deploy uses the moving latest-dev tag, while production
uses an immutable build-number tag produced by a separate Azure ADO
build. The Azure build is fed by the master branch after PR merges,
so even though the source is the same commit hash, the production
binary is technically a re-build, not a promotion of the same artifact
that SIT exercised. Any divergence in the Azure build environment
(toolchain version, base image, transitive dependency resolution) is
not covered by SIT.
These limitations are not reasons to skip SIT — they are reasons not to treat SIT as a substitute for the production smoke checks that follow the AKS release.
When SIT fails
Section titled “When SIT fails”A red SIT run halts the release process. The expected response is to
investigate, fix, and let the next commit to dev re-run the whole
pipeline (build → staging deploy → SIT). The fix may be in
meltano-catalog (the application code), in meltano-build (the
deployment definition), in meltano-config (the staging config), or in
meltano-sit itself (a flaky or stale test). Diagnosis usually starts
by reading the failed test’s output and matching it against recent
commits to those four repositories.
A flaky SIT failure should not be papered over by simply re-triggering the staging pipeline. Re-runs are appropriate when the failure is clearly environmental (e.g. a transient registry pull failure), but real bugs that happen to be intermittent will be missed if re-running becomes a habit.
Handoff to production
Section titled “Handoff to production”When SIT is green, the team is clear to open a Linear card from the New Release Template and begin the production workflow. The checklist on that card — and what each step actually does — is the subject of 03-production-release-checklist.md.