Pipeline Engine: CI/CD That Generates Compliance Evidence at Every Step

A federal contractor ships a critical patch on Thursday afternoon. The code passes tests, the vulnerability scan clears, and the deployment succeeds. Three weeks later, an assessor asks for evidence that the change was authorized, tested, scanned, and approved before production deployment. The engineering lead spends half a day pulling screenshots from three different dashboards, exporting scan reports, and finding the Slack thread where the change was verbally approved.

If your CI/CD pipeline does not generate compliance evidence, you are running compliance and engineering as two separate programs --- one that builds software and one that retroactively proves the first one did its job correctly. That separation is expensive, fragile, and unnecessary.

Advisedly's pipeline engine unifies both programs. Every build, test, scan, and deployment generates structured, signed compliance evidence mapped to the controls it satisfies --- automatically, as the work happens, not weeks later when someone asks.

Why Now: EO 14028 Section 4 and the End of Screenshot Evidence

Executive Order 14028, Section 4, mandated that software producers provide evidence of secure development practices to federal buyers. NIST translated those requirements into SP 800-218, the Secure Software Development Framework (SSDF), which maps specific practices --- build provenance, automated testing, vulnerability management, change control --- to verifiable, repeatable outputs.

OMB M-22-18 operationalized those requirements into procurement language. FedRAMP 20x (Phase 2 pilot announced underway in December 2025) expects continuous evidence streams, not quarterly screenshot packets. CISA BOD 26-04 (issued June 10, 2026) mandates risk-based vulnerability management with documented remediation timelines.

The direction is unmistakable: compliance evidence must be machine-generated, cryptographically attributable, and reproducible. A CI/CD pipeline that runs tests but does not record the results in an auditable, framework-mapped format is doing half the job. The pipeline engine closes that gap by treating evidence generation as a first-class build artifact --- not an afterthought bolted on by a compliance team that was never invited to the architecture discussion.

How the Pipeline Engine Works

Pipeline Definition

Pipelines are defined in .advisedly/pipeline.yml at the root of your repository:

stages:
  - name: build
    steps:
      - name: install-dependencies
        run: npm install
      - name: compile
        run: npm run build
        compliance:
          controls: [SA-11, CM-3]
          evidence_type: build_log

  - name: test
    steps:
      - name: unit-tests
        run: npm test
        compliance:
          controls: [SA-11]
          evidence_type: test_results
      - name: security-scan
        run: advisedly scan
        compliance:
          controls: [RA-5, SI-2]
          evidence_type: security_scan

  - name: deploy
    steps:
      - name: deploy-staging
        run: ./deploy.sh staging
      - name: smoke-test
        run: npm run test:e2e
        compliance:
          controls: [CA-2]
          evidence_type: test_results
      - name: deploy-production
        run: ./deploy.sh production
        compliance:
          controls: [CM-3, CM-5]
          evidence_type: deployment_record

The compliance block on each step is the key differentiator from generic CI/CD. It declares which framework controls the step satisfies and what evidence type it produces. The pipeline runner captures the step output, hashes it, signs it, maps it to the declared controls, and persists it as a structured evidence artifact --- all before the next step begins.

The Pipeline Runner Architecture

The pipeline runner executes in its own container --- isolated from both the application build and the platform API. This separation serves two purposes: security (the runner process cannot tamper with its own evidence) and reliability (a failed build cannot corrupt the evidence pipeline).

Each pipeline run receives a unique run ID. Every artifact generated during the run --- build logs, test results, scan findings, deployment records --- is tagged with that run ID, creating a complete audit chain from commit to production. The runner itself is stateless; pipeline state lives in the platform database, which means runner containers can be replaced, scaled, or restarted without losing evidence continuity.

The runner writes evidence artifacts to the platform evidence vault with SHA-256 content hashes. Once written, evidence is immutable --- protected by the same WORM audit trail that covers all platform operations. An auditor can request the evidence artifact for any pipeline run and independently verify its content hash matches the recorded hash.

Compliance Evidence Generation

Every pipeline run produces structured compliance artifacts:

Pipeline Step	Evidence Generated	Framework Mapping
Build	Build provenance (inputs, outputs, environment)	SA-11, SA-15 (SLSA compliance)
Unit tests	Test results with pass/fail rates	SA-11 (Developer testing)
Security scan	Vulnerability findings with severity and disposition	RA-5, SI-2 (Scanning, flaw remediation)
SAST/DAST/SCA	Static/dynamic analysis results	SA-11(1)(5)(8)
Code review	Review approval records	CM-3 (Change management)
Deployment	Deployment record with actor, timestamp, artifact hash	CM-3, CM-5 (Access restrictions for change)
Smoke tests	Post-deployment verification results	CA-2 (Control assessments)

Evidence types span the full CI/CD lifecycle: build_log, test_results, security_scan, deployment_record, code_review, approval_gate, artifact_signature, and sbom. Each maps to the controls it satisfies across whatever frameworks apply --- 800-53, 800-171, CMMC, FedRAMP, ISO 27001, or any combination from the 500+ supported frameworks.

SLSA Build Provenance

API and platform container images receive DSSE-signed SLSA v1.0 provenance statements and OpenVEX documents. Air-gap release tarballs and their product SBOMs receive detached Ed25519 signatures. These outputs are separate paths; this product does not claim a SLSA assurance level. The image attestations provide evidence that can support supply-chain review, and their signing key is held in the platform vault under envelope encryption.

Quality Gates: Stop Bad Builds Before They Become Compliance Findings

Quality gates define conditions that must be met before a pipeline proceeds to the next stage. Unlike generic CI/CD gates that simply block promotion, Advisedly's gates generate negative evidence when they fire --- documenting what was blocked, why, and the control requirement it would have violated.

Gate	Condition	Controls Satisfied
Test coverage	Minimum code coverage threshold	SA-11
Security scan	No critical or high vulnerabilities unresolved	RA-5, SI-2
SBOM generation	Bill of materials produced and signed	SR-3, SR-4
Approval	Required reviewer sign-off recorded	CM-3, CM-5
Compliance check	No blocking compliance findings against declared controls	CA-2

When a gate blocks promotion, the evidence artifact records the failure state: which gate failed, what threshold was violated, and what action was taken (blocked, overridden with justification, or escalated). Override events require a justification field and are flagged in auditor packets. An assessor can see not just that gates exist, but how often they fire, how often they are overridden, and by whom --- a compliance narrative that screenshots can never provide.

Staged Promotion

The pipeline supports staged promotion with evidence at every transition:

1. Build --- Compile and package; generate build provenance
2. Test --- Run automated tests; record results as evidence
3. Stage --- Deploy to staging; record deployment artifact
4. Verify --- Run smoke tests and compliance checks; generate verification evidence
5. Promote --- Deploy to production after all gates pass; record final deployment

Each stage transition is a compliance event. The full promotion chain --- from commit to production --- is a single auditable sequence with cryptographic linkage between stages.

How Pipeline Evidence Flows Into Auditor Packets

Pipeline evidence does not sit in a separate system waiting to be manually collected. It flows automatically into the organization's auditor packet:

• Change management section: Pipeline-enforced approval records, override justifications, and the full promotion chain from commit to production
• Vulnerability management section: Scan results from pipeline security stages, with disposition (fixed, accepted, mitigated) and remediation timelines
• Configuration management section: Build provenance, deployment records, and artifact hashes proving what was deployed matches what was built
• Testing section: Automated test results, coverage metrics, and post-deployment verification outcomes

When an assessor pulls the auditor packet for a given assessment period, every pipeline run during that period is already summarized, attributed, and mapped to controls. The evidence collection phase of audit preparation --- the four-to-eight-week manual effort that most organizations dread --- compresses to a review of auto-generated artifacts.

What Makes This Different from Generic CI/CD

GitHub Actions, GitLab CI, Jenkins, and Azure Pipelines are build orchestration tools. They execute steps, produce logs, and report pass/fail. They do not:

• Map step outputs to compliance framework controls
• Generate signed SLSA provenance attestations
• Produce structured evidence artifacts with content hashes
• Feed results into an auditor packet generator
• Record gate failures as negative compliance evidence
• Maintain WORM-protected evidence chains

Bolting compliance onto generic CI/CD means building a second system that watches the first system and retroactively documents what it did. That second system is always out of sync, always incomplete, and always the first thing that breaks when the pipeline changes. The pipeline engine eliminates the gap by making evidence generation intrinsic to execution --- the pipeline does not complete a step until the evidence artifact is persisted.

Key Takeaways

• Every pipeline step that carries a compliance block generates signed, framework-mapped evidence automatically --- no manual collection, no screenshots, no shared drives.
• SLSA build provenance with Ed25519 signatures satisfies EO 14028 Section 4 and NIST SP 800-218 requirements without a separate attestation workflow.
• Quality gates produce negative evidence when they fire, giving assessors visibility into not just what passed but what was blocked and why.
• The pipeline runner is architecturally isolated from the build environment, ensuring evidence integrity even if the application build is compromised.
• Evidence flows directly into auditor packets, compressing the weeks-long manual evidence collection phase into automated artifact assembly.

Frequently Asked Questions

Can I use the pipeline engine alongside my existing CI/CD?

Yes. The pipeline engine can run as a wrapper around existing build scripts or as a standalone pipeline. Organizations commonly start by adding compliance blocks to steps that map to their most urgent framework requirements, then expand coverage as they retire manual evidence collection for additional controls.

What frameworks does the evidence map to?

The 500+ compliance frameworks supported by the platform. Each step's compliance block declares the controls it satisfies; the platform's cross-framework crosswalk automatically maps those controls across every framework the organization has in scope for a given information system.

How is evidence protected from tampering?

Every evidence artifact receives a SHA-256 content hash at creation time. Artifacts are stored in the platform's WORM-protected evidence vault --- write-once, read-many, with database-level triggers that prevent modification or deletion. The hash chain is independently verifiable offline.

Does this work in air-gapped environments?

Yes. The pipeline runner, evidence generation, and SLSA signing all operate without external network dependencies. Verification bundles are produced alongside release artifacts for fully disconnected validation. The public key and signed envelopes are the only two pieces needed to verify provenance.

What happens if a gate is overridden?

Override events require a justification field, record the overriding actor, and are flagged in auditor packets. Assessors see both the gate failure and the documented rationale for proceeding --- which is often stronger compliance evidence than a gate that simply never fires.

How Advisedly Helps

The pipeline engine integrates compliance evidence generation directly into the CI/CD workflow so that every build, test, and deployment produces audit-ready artifacts mapped to your active framework controls --- eliminating the weeks of manual evidence collection that typically precede each assessment cycle. Reach out to begin@advisedly.ai

<!-- LI hook: Your CI/CD runs compliance and engineering as two programs. -->