Sebastian KaiserA personal engineering notebook

The notebook

CI/CD Across Team Boundaries

An order API can pass every test while a downstream consumer drops the customer's delivery choice. When several teams own the path from request to result, they need a way to check that the customer receives what was agreed.

An illustrative express-order change shows how shared container-delivery jobs and a small, scenario-based test setup can help teams investigate failures together, while they keep their own build tools and checks. A passing result is useful for deployment only while it still describes the combination being deployed.

1 One change, several teams

Suppose we add express delivery to an order-processing application. The order API validates the request and publishes an acceptance event. A consumer updates a query model, and the application displays the accepted order. To the customer, this is one operation. Its implementation may belong to several teams.

The API team can add the new field and pass its tests while a downstream consumer silently drops it. Even a small change therefore raises a question that no single team's build can answer: does the delivery option survive the whole trip?

Those teams may use different repositories and build tools. Some services sit behind private networks or in separate infrastructure accounts. Requiring everyone to adopt one pipeline or deploy together would make a local change depend on a much larger reorganization.

The approach here shares the recurring image-delivery work and gives teams a repeatable way to check behavior across their boundaries. It uses GitHub Actions, Helm, and Kubernetes; older systems can remain dependencies with their own deployment procedures. The order change is an illustrative example, not a report of a company deployment.

2 Start with the behavior

Before changing the pipeline, the API and consumer teams agree what accepting an express order means. The acceptance event must retain the delivery option, and the query must show the accepted order before an agreed deadline.

System view of the express-order example: application to order API, event stream, consumer and query store, with a query service returning the accepted order to the application. The test follows one unique order reference and checks that express delivery is preserved before the deadline.
Figure 1: The illustrative order system. The API publishes the delivery option through the event stream; the consumer preserves it in the query model. The scenario follows one order across that path and checks the query result before the agreed deadline. Dashed links represent query and response exchanges.

That agreement can become a scenario kept beside the code or in a shared testbench repository:

gherkinIllustrative express-order behavior with project-defined tags.Link
@domain-orders @capability-order-acceptance @usecase-express-delivery
Feature: Order delivery options

  Scenario: An accepted order retains its delivery option
    Given a unique order reference and a valid customer for this test run
    When the client submits an order with delivery option "express"
    Then its acceptance event contains delivery option "express"
    And its query shows "accepted" with "express" delivery before the deadline

The test follows one unique order reference from submission to the query result. It waits for asynchronous processing, with the deadline measured from submission. Yesterday's accepted order cannot satisfy it.

This gives the teams a concrete behavior to discuss during review. If duplicate events are also part of the requirement, they need a replay scenario as well; this one checks the first accepted result.

3 What the test pyramid leaves to the pipeline

The test pyramid still applies [1]. Request validation belongs in fast unit tests. Integration tests exercise the broker or database, and contracts check what producers and consumers expect at a boundary. The order scenario adds a check of the path those smaller tests cannot establish on their own.

Running it across repositories takes some coordination. Someone has to select the service versions, make them available to the test, and retain enough of the result for another team's deployment job to use. Teams must also agree what counts as done; Scrum makes that a shared responsibility for teams working on one product [2]. The pipeline makes their checks repeatable.

3.1 Alternatives and their limits

Mocks and interface specifications remain useful. OpenAPI and AsyncAPI describe HTTP and message contracts [3, 4], but neither establishes that a real consumer preserves the express option through to the query. Maintaining a mock image of every service would also mean maintaining a second implementation of much of the system.

At the other extreme, starting the entire platform for each change brings in services that have nothing to do with the order. A shared environment avoids some setup work, but deployments for other work can change its conditions while a test is running.

When cross-service changes are rare, a few hand-written journey tests may be enough. A maintained testbench becomes worthwhile when teams repeatedly reconstruct the same setup.

4 Build the part of the platform you need

The express-order scenario needs the API, event stream, consumer, and query service with its store. Authentication or customer services join that setup only if the request depends on them. The rest of the platform can stay out.

This is the test slice: the scenario together with the services, configuration, and data needed to exercise it. Teams maintain that setup explicitly. Tags can group scenarios by domain, capability, or use case, but selecting a tag does not discover dependencies or assemble an environment.

The smaller setup makes the behavior easier to investigate. If the delivery option disappears, the engineer can follow the request through a manageable set of services. It still needs real dependencies where their behavior matters; a fixture returning a convenient response cannot establish what the actual service will return.

4.1 Optional implementation with spex

One implementation uses spex to translate constrained scenarios into Kubernetes Test TooL (KUTTL) workspaces [8]. Teams maintain a catalog of supported operations, so a readable step still needs an implementation behind it.

Scenario selection and environment setup remain separate [9]. A suite chooses the scenarios; profiles describe the services to install, and bindings identify the endpoints to call. The order profile brings up the participating services with the candidate API image and pinned companion versions. A replay check might use a smaller setup without the API.

The same arrangement can run locally in a temporary Kind cluster [10] or against existing development services. Shared targets need additional care, as discussed below.

5 Reuse delivery jobs across applications

With the behavior defined, the API team's repository can call a reusable GitHub Actions workflow [5]. Compilation stays in the team's Dockerfile, and the team supplies its test commands. The shared jobs handle the image's journey between build, test, publication, and deployment.

The workflow first builds an image or reuses one whose declared build inputs have not changed. It passes an exported image archive and checksum between runners, so test jobs use the selected image rather than build another one. Publication waits for every enabled check to pass. A later release copies the published image to the destination registry and verifies its digest without rebuilding it. Figure 2 follows that image through build and release.

Reusing an image saves a build; it does not excuse running the required tests. It also depends on declaring the inputs correctly and refreshing mutable base images or external dependencies when needed.

Artifact-preserving delivery with checksummed image handoff, publication gated by enabled checks, no-rebuild promotion, and deployment supported by the run record and relevant conditional checks
Figure 2: Follow the checksummed image handoff through gated publication and promotion; conditional checks support deployment decisions.

Keeping policy separate from build identity matters in small ways too. A pull request once failed before building because a release-branch naming rule was applied to its feature branch. We changed the check to validate the target branch while keeping the source commit as the build identity. Developers could use ordinary feature names without weakening the release policy.

Access follows the job's needs. An isolated build need not reach private services; the order test may need an approved runner and scoped credentials. GitHub OpenID Connect can supply short-lived cloud credentials [6], but the platform still has to decide which jobs may use them. Untrusted changes must not gain private access, and permission to test is separate from permission to deploy [7].

6 Know what a passing test covered

Suppose the scenario passes. The event retained express delivery and the query returned the accepted order before the deadline. That result describes the API, consumer, and settings used in that run.

The pipeline keeps those versions and settings with the test reports and image identity. The record needs to describe what actually ran, not merely what a job intended to install. Otherwise, the next team cannot tell whether the result applies to its proposed deployment.

6.1 If deployment approval uses a shared target

An isolated slice makes that easier. On a shared target, a consumer deployment may overlap the test. Both teams may be doing reasonable work, while the passing assertions leave them uncertain about which combination worked.

If the result will support deployment approval, the platform needs to prevent competing changes for the duration of the run and invalidate the result if relevant deployments or settings change. That control must work across repositories and reject stale workers at the target. A lock checked only by a job cannot stop it from resuming after its permission has expired.

These are requirements on the surrounding platform, not guarantees supplied by spex or a reusable workflow. Where target changes cannot be controlled or observed, the result has that limitation. An isolated setup is preferable when the necessary dependencies can be recreated.

6.2 Keep reports tied to the image

Shared jobs retain reports from trusted runs alongside the selected image; spex can supply JUnit results for CI [8]. Deployment jobs need to verify where a report came from rather than trust an arbitrary file claiming success. Captured diagnostics must also leave out credentials.

I want the reports to give both teams enough context to investigate the same failure. Did the API reject the request? Did the event lose the field? Did the consumer fail to update the query? A wrong endpoint or unavailable broker needs a different fix. The report should help them locate the break in the journey before deciding where a change belongs.

7 Deploy the combination that was checked

Now suppose the API image stays unchanged, but another team releases a new consumer. The earlier passing test no longer answers the same question.

For the example below, A2 is the API image digest; C1 and C2 are different consumer image digests. Express delivery remains enabled and the other inputs stay fixed.

Table 1: An illustrative release decision for an unchanged API image.
Stage Combination Decision
Recorded pass A2 with C1 Retain coverage of A2/C1
Later proposal A2 with C2 Hold pending coverage of A2/C2
Rolling proposal A2 with C1, C2 Cover both paths and their coexistence

A mixed rollout matters too. A test that happens to reach C1 says nothing about the C2 path or their coexistence. Before deployment starts, the release owner needs results that cover the combinations customers can actually encounter. The deployment controls must keep the target consistent with that decision.

The test environment need not match production, but the assumptions behind the test must hold in both. For the express-order scenario, that includes compatible event formats, a consumer that preserves the delivery option, and processing that meets the agreed deadline. Where possible, make those assumptions executable checks. Before using the result to approve a rollout, check whether any environment differences could invalidate it. If they could, rerun the affected checks against the proposed setup.

Different endpoints or credentials can be expected; they still need to reach the intended services. A changed processing mode or feature flag may alter the behavior itself. An external service that can change without an observable version also limits what an earlier test can establish. The release owner must decide whether a fresh target check is needed or whether to accept the remaining risk explicitly.

7.1 Roll out and recover the express-order change

For this change, the teams can first deploy a consumer that handles existing events and preserves the new field, then introduce the producer that sends it. If old consumers remain active, the new producer must work with them too, or the option must stay disabled until they have left service. Older stored orders and replayed events belong in those compatibility checks.

The release coordinator gets agreement on that sequence and how long old formats remain supported. Operators need authority to stop the rollout; someone must also be accountable for resolving disagreements and approving exceptions. An exception records a decision to accept risk, not a passing test.

During activation, the operations owner watches the acceptance-to-query delay and consumer backlog against agreed stop conditions. Disabling the option may contain a problem, but orders already accepted still need processing or repair. Redeploying an earlier image cannot undo database writes or recall events.

If an express order later appears as standard delivery, the immediate task is recovery. Once the cause is understood, a regression scenario can preserve what the teams learned for the next change.

8 Keep the shared setup worth maintaining

Service teams own the behavior and maintain the scenarios that check it. Workflow maintainers own the shared delivery machinery; platform owners control access and deployment writes. Those roles may overlap, but the responsibilities need to be clear.

Responsibility map separating service checks, shared workflow and testbench maintenance, target controls, release coordination, and named exception approval. Roles may overlap; passing checks alone do not grant deployment authority.
Figure 3: Responsibilities in the proposed delivery arrangement. Teams contribute behavioral checks, maintainers provide repeatable execution, operators control targets, and release roles coordinate agreements and accountable decisions.

Start with shared build and test jobs, then add a slice for a recurring cross-service check. The catalog and setup need maintenance as services change, and generated workspaces must remain readable enough to debug. Versioned workflows also need an upgrade policy that includes their separately referenced actions.

Shared targets can become a queue. Adding runners will not help when builds must wait for the same resources; the remedy may be a smaller shared scope or more isolation. Tests also need to leave existing infrastructure intact and clean up only what they created.

Judge the shared jobs and the testbench separately. Compare the build and setup work saved with the time spent maintaining and debugging them. If teams spend more time interpreting generated failures than they save in setup, simplify the testbench before adding more scenarios.

9 References

  1. H. Vocke, The Practical Test Pyramid, February 26, 2018.
  2. K. Schwaber and J. Sutherland, The Scrum Guide, November 2020.
  3. OpenAPI Initiative, OpenAPI Specification v3.1.1, October 24, 2024.
  4. AsyncAPI Initiative, AsyncAPI Specification v3.0.0.
  5. GitHub, Reuse workflows.
  6. GitHub, OpenID Connect.
  7. GitHub, Secure use reference.
  8. pruefwerk, spex: Scenario-driven KUTTL workspace generator for Kubernetes acceptance testing, README snapshot, revision 8033f19.
  9. pruefwerk, Scenario Repository Guide, revision 8033f19.
  10. Kubernetes SIGs, kind.