Log Management: Retention, Search, and Compliance

First developed as part of our SIEM capability whitepaper, May 2026. Expanded and updated for this site.

When Maersk's IT team rebuilt 45,000 endpoints after NotPetya inflicted $300M in damages, the investigation hinged on logs --- and the logs they had not retained were the ones that would have shown the initial lateral movement from the compromised MeDoc update server into the domain controller. The forensic team could reconstruct what happened after the wiper fired, but the critical 48-hour window when the attacker moved from a single accounting server to global domain admin lived only in DNS query logs and DHCP lease records that had already rotated off. The lesson was not that Maersk lacked a SIEM --- it was that their log management strategy had a gap between what they kept and what they needed.

Why Now

FedRAMP's 20x pilot program is tightening AU-family controls significantly. Draft guidance now requires organizations to demonstrate not just that they generate audit logs, but that they retain specific log categories with cryptographic integrity verification and can produce them within defined time windows during assessment. The days of "we keep everything for a year" as a sufficient log management answer are ending. At the same time, OMB memorandum M-21-31 requires federal agencies to retain specific log categories at defined maturity tiers and centralize them for analysis --- meaning log pipeline architecture is becoming a federal compliance requirement, not just a best practice.

The Three Functions of Log Management

Log management serves three distinct functions, each with different requirements:

Security monitoring --- Real-time analysis for threat detection (needs speed)
Incident investigation --- Forensic search during and after incidents (needs depth and breadth)
Compliance evidence --- Audit trail demonstrating control effectiveness (needs retention and integrity)

A log management strategy must address all three. Most failures come from optimizing for one at the expense of the others --- the organization with sub-second search on 30 days of firewall logs that cannot produce authentication records from 90 days ago when an assessor asks.

Retention Requirements by Framework

Framework	Minimum Retention	Notes
FedRAMP	90 days online, at least 1 year archived	Must be retrievable for audits and investigations
NIST 800-171	Not specified (org-defined)	Common practice: 1-3 years; assessors expect at minimum 1 year
HIPAA	6 years	Aligns with HIPAA record retention; includes access logs for ePHI
PCI DSS v4	1 year (3 months immediately accessible)	Applies to all CDE activity
SOC 2	Audit period + reasonable retention	Typically 1 year; auditors want full period coverage without gaps
DFARS 7012	90 days post-incident (evidence preservation)	Incident-specific; baseline retention must exceed MTTD + investigation window
CMMC Level 2	Same as 800-171 (direct mapping)	Assessment methodology validates actual retention, not policy claims

When multiple frameworks apply, retain for the longest required period. Most organizations settle on 1 year hot storage (searchable) and 3-7 years cold storage (archival). But here is the contrarian take: storing every log for 7 years is not a security strategy --- it is a storage bill. The retention period that matters is the one that covers your mean dwell time plus your investigation window. If your MTTD is 14 days and your investigations close within 30, then 90 days of searchable hot storage covers 100% of your security use cases. The 7-year cold tier exists purely for compliance --- fund it accordingly, and do not pretend it serves security operations.

What to Log: A Priority Framework

Not all logs are created equal. The question is not "what can we log?" but "what must we log to detect the attacks we face and satisfy the auditors who assess us?"

Tier 1: Must Log (Security and Compliance Critical)

Authentication events --- successful and failed logins across all systems, including service accounts
Authorization changes --- role assignments, permission grants, group membership modifications
Administrative actions --- user creation/deletion, configuration changes, policy modifications
Data access events --- CUI/PII/PHI access, downloads, bulk queries, export operations
Network security events --- firewall allows/denies, IDS/IPS alerts, VPN connections
System events --- service starts/stops, OS patch installations, boot sequences, scheduled task changes

Tier 2: Should Log (Investigation Support)

DNS queries --- internal resolution patterns reveal C2 beaconing and data exfiltration before any other telemetry source
DHCP leases --- IP-to-device mapping is critical for attribution; without it, firewall logs showing 10.0.4.17 are meaningless
Web proxy requests --- URL access patterns, especially to uncategorized domains
Email metadata --- sender, recipient, subject, attachment hashes (not body content)
File access --- reads, writes, deletes on sensitive shares and repositories
Process execution --- command-line arguments, parent-child relationships, unsigned binary execution

Tier 3: Expensive to Log (Evaluate Against Threat Model)

Full packet capture --- enormous volume, but invaluable for forensics on encrypted C2 channels
Application debug logs --- noisy, but occasionally the only record of logic-layer attacks
Database query logs --- every SQL statement creates massive volume; consider logging only DDL and privileged DML
Cloud API calls --- CloudTrail/Activity Log completeness vs. cost at scale

The Log4Shell vulnerability (CVE-2021-44228) demonstrated why Tier 2 matters: organizations that logged DNS queries detected exploitation within hours because the JNDI callback to attacker-controlled LDAP servers was visible in DNS before it appeared anywhere else. Organizations without DNS logging learned they were compromised days or weeks later.

Tiered Storage Architecture

Design storage tiers around access patterns, not just retention periods:

Tier	Retention	Purpose	Search Speed	Typical Technology
Hot	30-90 days	Real-time monitoring, active investigation	Sub-second	SIEM primary index, Elasticsearch hot nodes
Warm	90 days - 1 year	Investigation and compliance queries	Seconds to minutes	Object storage with index overlays, frozen indices
Cold	1-7 years	Compliance retention, historical reference	Minutes to hours	Blob storage (compressed, encrypted at rest)
Archive	7+ years (if required)	Legal hold, long-term regulatory	Hours to days	Immutable storage with lifecycle policies

The critical design decision is the hot-to-warm transition. Too aggressive (7 days hot) and your analysts cannot investigate without waiting for rehydration during active incidents. Too generous (180 days hot) and your storage costs dominate the budget. Calibrate against your MTTD: if your average detection takes 14 days and investigation takes another 14, then 45 days of hot storage covers the vast majority of cases with margin.

Log Pipeline Design

A production log pipeline has seven stages, and failures at any stage create gaps that compound downstream:

Collection --- Agents, forwarders, and API integrations on source systems. The agent must survive the system it monitors --- if the attacker can kill the logging agent, your visibility dies first.
Transport --- Encrypted, reliable delivery via syslog-TLS, HTTPS, or message queues. Buffer locally on the source if the transport is unavailable --- never drop logs silently.
Parsing --- Extract structured fields from raw log data. Failed parses must land in a dead-letter queue, not disappear.
Enrichment --- Add context: asset owner, business criticality, geolocation, threat intel hits. This is where raw events become investigable events.
Routing --- Direct events to the appropriate tier based on source, type, and criticality. Route decisions happen once; re-routing after storage is expensive.
Storage --- Write to the target tier with appropriate indexing for the expected query patterns.
Alerting --- Real-time rules fire on the stream before storage, not after. Detection latency should be bounded by parse time, not index time.

The pipeline must be monitored as infrastructure. A silent failure in collection --- a misconfigured agent, a rotated credential, a full disk on a forwarder --- creates a gap that only becomes visible when you need those logs for an investigation and discover they stopped flowing three weeks ago.

Integrity and Tamper Protection

Compliance frameworks (NIST 800-53 AU-9, CMMC AU.L2-3.3.8, FedRAMP AU-9) require protection of audit logs from unauthorized modification. An attacker who achieves admin access will attempt to cover their tracks by modifying or deleting logs. Your architecture must make this impossible or at minimum detectable:

Write-once storage for cold and archive tiers (WORM policies, immutable blob storage)
Hash chains --- each log batch includes a cryptographic hash of the previous batch, creating a tamper-evident chain
Separate infrastructure --- log storage must not be on the systems being logged, and administrative access must require separate credentials
Access controls --- limit who can read logs (investigation team) versus who can administer the log platform (platform engineers), and log both activities
Forward to an independent system --- outbound SIEM forwarders create a second copy under separate administrative control

The integrity requirement is not theoretical. In the SolarWinds compromise, the attackers deliberately covered their tracks --- disabling audit logging (via auditpol) before acting, re-enabling it afterward, and removing the tooling and artifacts that recorded their lateral movement. Organizations that forwarded logs to a separate, independently administered system retained the evidence.

Cost Management

Log management costs are driven by three factors that multiply against each other:

Volume --- More logs = more collection bandwidth, parsing compute, and storage
Retention --- Longer retention = more storage (linear growth)
Search speed --- Faster search = exponentially more expensive infrastructure (hot storage costs 10-50x cold per GB)

Practical Cost Controls

Filter noise before ingestion. Health check pings, load balancer keep-alives, and debug-level application logs that fire every second consume volume without producing security or compliance value. Filter at the collection layer, not after indexing.
Compress aggressively on cold tiers. Log data compresses 10-20x. A 7-year retention policy on compressed cold storage costs a fraction of what most organizations assume.
Summarize high-volume sources. Aggregate firewall allow events into 5-minute windows (source, destination, port, count). Keep full-fidelity records only for denies and alerts.
Implement per-source volume monitoring. A misconfigured application that starts logging every HTTP request at debug level can double your daily volume overnight. Set volume thresholds and alert on deviations.
Budget by function. Security monitoring (hot tier) gets the expensive fast storage. Compliance retention (cold tier) gets the cheap durable storage. Do not fund compliance retention at security monitoring speeds --- assessors do not need sub-second search.

Log Source Coverage Gaps

The most dangerous log management failure is not retention or search speed --- it is missing sources. You cannot detect what you do not collect. Common gaps that enable breaches:

Build and CI/CD pipelines --- SolarWinds was invisible because nobody logged build system activity
SaaS application audit logs --- OAuth token abuse lives in identity provider logs that many organizations never ingest
Container orchestration --- Kubernetes API server logs, admission controller decisions, pod lifecycle events
Cloud control plane --- IAM policy changes, network ACL modifications, storage bucket permission grants
DNS --- the single most valuable log source for detecting C2 and data exfiltration, and the most commonly missing
Service mesh / API gateway --- east-west traffic between microservices, where lateral movement actually happens

Conduct a log source gap analysis quarterly: list every system category in your environment, confirm each has an active log collection path, and validate that logs are actually arriving (not just configured). The gap you discover today is the gap an attacker exploits tomorrow.

Key Takeaways

Log management serves three functions (monitoring, investigation, compliance) --- each with different speed, depth, and retention requirements
Retention periods should be driven by framework requirements and your MTTD + investigation window, not arbitrary multi-year policies
Tiered storage is not optional: hot storage for detection speed, cold storage for compliance duration
Log integrity (hash chains, WORM, separate infrastructure) is a compliance control and a security control simultaneously
Missing log sources are more dangerous than short retention --- you cannot investigate what you never collected
Cost management starts at collection (filter noise) and compounds through architecture (tier appropriately)

Frequently Asked Questions

How do I determine the right retention period for my organization?

Start with your compliance obligations --- identify the longest mandatory retention across all applicable frameworks. Then validate against your security needs: your retention must exceed your MTTD (mean time to detect) plus your typical investigation window. If you detect breaches in an average of 14 days and close investigations within 30, your hot tier needs at minimum 45 days. Cold storage handles the compliance tail. Most multi-framework organizations land on 1 year hot/warm and 3 years cold as the pragmatic default.

What is the biggest mistake organizations make with log management?

Treating it as a storage problem rather than a coverage problem. Organizations invest in expensive search infrastructure for the logs they already have while missing entire categories of log sources. The SolarWinds breach was invisible not because of retention policy but because build pipeline telemetry was never collected. Before optimizing search speed or extending retention, confirm you are actually collecting logs from every system category in your environment.

How do I handle log management costs that keep growing?

Volume growth is the primary cost driver, and it compounds. First, filter noise at the collection layer (health checks, debug logs, repetitive allow events). Second, implement tiered storage so only security-relevant recent data lives on expensive hot infrastructure. Third, compress cold storage aggressively (10-20x compression is typical for log data). Fourth, monitor per-source volumes and alert on unexpected spikes. A misconfigured application can double daily ingest overnight.

Do I need to keep logs from systems that have been decommissioned?

Yes, for the retention period applicable at the time those logs were generated. Decommissioning a system does not eliminate your obligation to retain its historical audit records. This is a common audit finding: organizations delete log archives when they decommission the source system, then cannot produce evidence when assessors ask about historical access patterns. Archive logs from decommissioned systems to cold/archive tiers with appropriate lifecycle policies.

How do I prove log integrity to an auditor?

Demonstrate three things: (1) logs cannot be modified after write (WORM storage, append-only indices, or hash chains), (2) administrative access to the log platform is separate from administrative access to the systems being logged, and (3) you can detect if any gap exists in the log stream (sequence numbers, heartbeat events, or volume-based anomaly detection). The strongest posture forwards logs to an independently administered secondary system --- if both copies agree, integrity is demonstrated by cross-reference.

How Advisedly Helps

Advisedly consolidates log management and SIEM into a unified platform with tiered storage, compliance-aligned retention policies, and sub-second search across hot data. The platform maps every log source to the specific compliance controls it satisfies --- AU-2, AU-3, AU-6, AU-12 and their equivalents across 500+ frameworks --- so your security telemetry automatically generates audit evidence without a separate export-and-reconcile process. For organizations with existing log infrastructure, Advisedly integrates as the compliance mapping layer through outbound SIEM forwarders, adding framework evidence generation to your current SIEM investment without rip-and-replace. Contact begin@advisedly.ai to design a log management architecture that satisfies your compliance obligations and your security operations team simultaneously.