Customer Secret Vault: BYOK Encryption for Integration Credentials

In February 2024, attackers breached Change Healthcare through a remote-access portal using a single set of compromised credentials on an account with no multi-factor authentication. From that one foothold, they reached systems processing claims and payments for a huge share of the U.S. healthcare system, and the company took its platforms offline nationwide. The blast radius was total because nothing compartmentalized what that one credential could reach. That is the lesson for any system holding secrets: when one key or one credential protects everything, one compromise exposes everything.

Why Now: CMMC and the Key Custody Question

The CMMC program rule at 32 CFR Part 170 (effective December 16, 2024) requires defense industrial base contractors to demonstrate cryptographic protection of CUI under NIST 800-171 control 3.13.11 (SC.L2-3.13.11 --- CUI Encryption). Auditors no longer accept "we encrypt at rest" as a sufficient answer. They ask who controls the key, whether the key can be rotated without downtime, whether access is audited, and whether the customer can revoke the vendor's ability to decrypt. If the answer to any of those questions is "the vendor handles it," the control is at best partially satisfied.

If you cannot revoke the vendor's access to your keys, you do not have BYOK --- you have marketing. The term has been diluted by vendors who offer "customer-managed keys" that still require the vendor to hold a fallback decryption path. Real BYOK means the vendor physically cannot decrypt your data without your key material, and revoking the key renders the data permanently inaccessible.

The Problem with Single-Key Encryption

The baseline approach to credential storage in security platforms: encrypt with one key, store that key as an environment variable. It technically qualifies as encryption at rest. It checks a box on an audit questionnaire. It fails every serious threat model.

If that key leaks --- misconfigured deployment, compromised CI/CD pipeline, insider with runtime access --- every secret is exposed simultaneously. No blast-radius containment. No way for individual customers in a multi-tenant platform to maintain independent key control. No log of who decrypted what, when, or why.

For organizations operating under NIST 800-171, CMMC, FedRAMP, or DFARS, this is not theoretical. Assessors probe how integration credentials are protected. "We encrypt them" starts the conversation, not ends it.

Envelope Encryption: Containing the Blast Radius

Advisedly's Customer Secret Vault uses envelope encryption --- a two-layer key architecture that limits the damage of any single compromise.

Layer 1 --- the Data Encryption Key (DEK). Each secret is encrypted with its own unique DEK using AES-256-GCM. The DEK is generated at encryption time, used once, and never stored in plaintext.

Layer 2 --- the Key Encryption Key (KEK). The DEK is itself encrypted (wrapped) by a KEK before storage alongside the ciphertext. The KEK never touches the secret data directly.

Compromising encrypted secret data alone is useless --- you need the wrapped DEK. Compromising the KEK alone is useless --- you still need the encrypted secrets. Compromising any single DEK exposes exactly one secret, not the entire vault.

This is the architecture used by AWS, Azure, and GCP for their own managed encryption and the pattern recommended by NIST SP 800-57 for key management in federal information systems.

AES-256-GCM: Authenticated Encryption

The cipher choice matters. AES-256-GCM (Galois/Counter Mode) is an Authenticated Encryption with Associated Data (AEAD) construction.

Confidentiality: AES-256 provides 256-bit symmetric encryption. Approved for Top Secret classification under CNSA Suite.

Integrity and authenticity: GCM produces an authentication tag alongside ciphertext. Decryption verifies the tag before releasing plaintext. Tampered ciphertext, altered IVs, or modified metadata cause explicit rejection --- not corrupted plaintext.

Encryption without authentication is vulnerable to bit-flipping attacks, padding oracles, and ciphertext manipulation. GCM closes those vectors in a single operation. Each encrypted secret stores its IV and authentication tag in dedicated fields, keeping each secret independently decryptable and re-encryptable --- which makes per-secret, transactional key rotation possible.

Bring Your Own Key (BYOK)

For customer organizations, BYOK is required, not an optional upgrade. You supply the KEK. You control where it lives. You decide when it rotates. If you revoke it, the platform physically cannot decrypt your secrets --- not because of policy, but because of mathematics.

Customer organizations bring the KEK through one of these provider modes:

AWS KMS --- wrap and unwrap operations happen through the KMS API. Key material never leaves AWS.
Azure Key Vault --- wrap and unwrap stay within the Key Vault boundary.
GCP Cloud KMS --- key material stays in Google infrastructure.
HSM via PKCS#11 --- for strictest requirements, the KEK lives on a hardware security module.

In every mode, the platform never sees your KEK in plaintext. It sends the DEK to your KMS or HSM for wrapping and unwrapping. If your KMS denies access --- revoked grant, rotated key, disabled key --- the platform returns an error. It does not fall back to a platform-managed key. It does not cache plaintext. It fails closed.

HSM and PKCS#11: FIPS-Validated Key Storage

Some environments require FIPS 140-2 Level 3 or FIPS 140-3 validated key storage. "The key is in a cloud KMS" is insufficient when accreditation demands hardware-backed cryptographic modules with tamper-evident physical security.

The vault includes a native PKCS#11 adapter interfacing directly with hardware security modules:

Cloud HSMs --- AWS CloudHSM, Azure Dedicated HSM, GCP Cloud HSM
On-premises network HSMs --- Thales Luna, Entrust nShield, Yubico HSM 2, and any PKCS#11-compliant module

Wrapping mechanisms supported: AES Key Wrap with Padding (RFC 5649), AES Key Wrap (RFC 3394), and AES-GCM wrapping for FIPS 140-3 alignment. Configurable per organization based on HSM capabilities and accreditation requirements.

HSM session management: one initialization per process, session pooling per slot, clean finalization on shutdown. The HSM PIN is never stored in environment variables or logged --- pulled from the vault's own internal meta-secret store, itself envelope-encrypted.

For air-gapped deployments where no cloud KMS is available, PKCS#11 provides the same BYOK guarantee using on-premises hardware.

Key Rotation Without a Maintenance Window

Key rotation is where most secret management systems collapse. Naive rotation re-encrypts every secret in one batch --- hours of downtime, a window where the whole vault is mid-migration.

Advisedly's rotation works incrementally, one secret at a time, each inside its own database transaction. Decrypt under old key material, re-encrypt under new key material, update key version, audit the operation --- all within a per-row transaction. No all-or-nothing cutover. No maintenance window. No blast-radius expansion during rotation.

Rotation can be automated on schedule or triggered manually when policy demands it. Every re-encryption is captured in the audit trail.

Audited Access: Every Decrypt Leaves a Trail

Encryption without audit is security theater. If you cannot prove who accessed a secret, when, and from what context, your encryption protects secrets from everyone except insiders with legitimate platform access.

Every vault operation --- create, read, update, rotate, delete, expiration --- is logged in a dedicated, append-only audit table purpose-built for secret access events.

The critical constraint: the audit row is written inside the same database transaction as the decrypt operation. Plaintext cannot be returned without the audit record being committed first. If the audit write fails, the decrypt fails. No race condition. No window where a secret can be revealed without evidence.

This satisfies NIST 800-171 control 3.3.1 (AU.L2-3.3.1 under CMMC) --- creating and retaining system audit logs to enable monitoring, analysis, investigation, and reporting of unlawful or unauthorized system activity.

Access Controls and Organizational Scoping

Secrets are scoped to organizations and tagged by integration type. A scanner credential for one organization is not visible to another organization's administrators, even within the same platform instance.

The reveal operation is deliberately friction-laden: time-limited modal that auto-clears after 30 seconds, no bulk export, every reveal a discrete audited event requiring explicit user action. Tier-based limits control per-organization secret counts, preventing unbounded growth.

Air-Gap Compatibility

For disconnected, classified, or air-gapped environments, the vault operates identically without external cloud dependency. Cloud KMS providers are replaced with local key management --- on-premises HSM via PKCS#11 or operator-configured local key mode.

The encryption primitives, audit trail, access controls, and rotation mechanics are the same. Only the KEK location changes. The crypto layer is decoupled from the key storage layer, so swapping providers requires zero changes to how secrets are encrypted, stored, or audited.

Key Takeaways

Envelope encryption (per-secret DEK wrapped by customer-controlled KEK) contains blast radius to a single secret per compromise
BYOK is mandatory for customer organizations --- the platform cannot decrypt without your key material, and revoking the key renders data permanently inaccessible
AES-256-GCM authenticated encryption prevents both disclosure and undetected tampering
Per-secret transactional rotation eliminates maintenance windows and all-or-nothing cutover risk
Every decrypt is audit-logged in the same transaction --- no window exists where a secret can be revealed without evidence
FIPS 140-2/3 validated key storage via native PKCS#11 adapter for environments requiring hardware-backed crypto

Frequently Asked Questions

What happens if we lose our KEK?

The secrets become permanently inaccessible. This is by design --- it is the mathematical guarantee that makes BYOK meaningful. Organizations should follow their KMS provider's key backup and disaster recovery procedures. For AWS KMS, multi-region keys provide geographic redundancy. For HSMs, key backup ceremonies with split-knowledge custodians are the standard approach.

Can we rotate the KEK without re-encrypting every secret immediately?

Yes. When a new KEK version is provisioned, the system begins incremental re-wrapping. Secrets encrypted under the previous KEK version remain readable (the old version is not destroyed until re-wrapping completes). The rotation process walks through secrets transactionally, and progress is visible in the operator dashboard. Typical rotation for a vault with 500 secrets completes in under 60 seconds.

How does the vault handle multi-tenant isolation?

Each organization's secrets are scoped by organization ID at the database level. Cross-organization access is architecturally impossible --- queries are filtered by org context before hitting the database. Even platform administrators cannot reveal another organization's secrets without that organization's KEK, which is customer-controlled.

Is there a secret count limit?

Tier-based limits apply: each subscription tier defines a maximum secret count per organization. Updates to existing secrets do not count against the cap. The limit exists to keep audit trail volume manageable and to prevent unbounded storage growth, not as an artificial paywall.

Does the vault work with secrets that need to be read by automated processes?

Yes. Scanner credentials, SIEM API tokens, and integration keys are read by background workers during automated operations (vulnerability scans, sync pushes, connector polls). Each automated read is audit-logged identically to a human read. The audit trail distinguishes automated reads (system context) from manual reveals (user context) so operators can identify anomalous access patterns.

How Advisedly Helps

The Customer Secret Vault provides AES-256-GCM envelope encryption with mandatory BYOK, FIPS-validated HSM support, per-secret transactional rotation, and same-transaction audit logging --- giving defense industrial base contractors the cryptographic control and audit evidence that CMMC SC.L2-3.13.11 assessors demand. begin@advisedly.ai