Building Detection Rules That Actually Work

On December 9, 2021, a proof-of-concept for CVE-2021-44228 -- Log4Shell -- hit Twitter. Within hours, mass exploitation began. Organizations that had a detection rule looking for JNDI lookup strings in inbound HTTP headers caught the first attempts within minutes and blocked them. Organizations without that rule -- or whose SIEM was so overloaded with noise that the alert was buried -- spent days to weeks discovering they were compromised. Some learned from breach notifications sent by vendors, not their own detection stack.

The difference was not budget or team size. It was whether someone had written a precise, tested detection rule and deployed it in an environment where analysts could actually see it fire.

Detection engineering is closer to software engineering than checkbox compliance. A rule must be written, tested against historical data, validated for false positive rate, deployed, monitored, and maintained. This article covers the full lifecycle -- from threat model to production deployment to retirement -- with the operational rigor that separates detection programs that catch real intrusions from those that generate noise.

Why Now

The environment for detection engineering has shifted fundamentally:

CVE explosion: Over 29,000 CVEs published in 2023. Each represents a potential detection rule. Without a systematic approach, organizations either write too many rules (drowning in alert fatigue) or too few (missing real exploitation).
Federal detection mandates: CISA binding operational directives and federal zero-trust guidance increasingly push agencies toward measurable detection coverage mapped to MITRE ATT&CK techniques. Detection is no longer "best effort" -- it is a measurable, auditable obligation.
Living-off-the-land dominance: Attackers increasingly use legitimate system tools (PowerShell, WMI, certutil) that have no malware signature. Detection must target behavior patterns, not file hashes -- requiring more sophisticated rule logic and more careful tuning.

The contrarian truth: a detection rule with a 50% false positive rate is worse than no rule at all. It does not just fail to detect threats -- it actively trains analysts to ignore the category, creating a blind spot that persists even after the rule is fixed. Bad rules do not merely waste time; they build organizational muscle memory to dismiss.

Anatomy of a Detection Rule

Every effective detection rule has five components:

1. Data Source

What log data feeds the rule. A rule can only detect what it can see. If you need to detect lateral movement via RDP, you need Windows Security Event Logs with Logon events (Event ID 4624/4625). If you need to detect exfiltration via DNS tunneling, you need full DNS query logs, not just firewall connection logs.

Common blindness: organizations write rules against data sources they think they have but do not actually collect reliably. Before writing any rule, verify that the data source is ingesting at the expected volume and latency.

2. Logic

The condition that triggers the rule. Five logic types cover the detection landscape:

Threshold: More than N events in T time (e.g., 10 failed logins in 5 minutes)
Pattern: Specific event sequence (e.g., successful login followed by immediate privilege escalation)
Anomaly: Deviation from baseline behavior (e.g., user accessing 10x more files than their 30-day average)
Indicator match: Match against known-bad indicators (IPs, hashes, domains from threat intelligence feeds)
Absence: Expected event did not occur (e.g., no heartbeat from critical system in 10 minutes)

Pattern and anomaly rules catch advanced threats. Threshold and indicator-match rules catch commodity attacks. A mature detection program uses all five.

3. Context Enrichment

Additional data attached to the alert so analysts can triage without tool-switching:

Asset criticality: Domain controller vs. test workstation
User risk: Privileged admin vs. standard user
Threat intelligence: Source IP in active threat feeds
Historical activity: Has this user/host triggered this rule before?
Vulnerability state: Is the targeted system missing a relevant patch?

An alert without context takes 10 minutes to triage. An alert with context takes 30 seconds.

4. Severity Classification

How urgently the alert needs attention:

Severity	Criteria	Response Target
Critical	Active compromise likely, immediate business impact	Page on-call, respond within 15 min
High	Probable malicious activity, sensitive data at risk	Priority queue, respond within 1 hour
Medium	Suspicious activity warranting investigation	Standard queue, respond within 4 hours
Low	Informational, minor anomaly	Investigate during business hours

Severity must match actual risk. A rule that fires frequently on ambiguous activity should not be marked Critical regardless of the theoretical severity of what it detects. Over-classification erodes trust in severity levels across the entire program.

5. Response Guidance

What the analyst should do when the rule fires:

Investigation steps (what to check, in what order)
Containment actions (what to isolate or block)
Escalation criteria (when to invoke incident response)
False positive identification criteria (what patterns indicate benign activity)

A rule without response guidance produces an alert that generates a question ("what do I do with this?") rather than an action. Rules are not complete until an analyst can follow the attached guidance without asking a senior engineer.

The Detection Rule Lifecycle

Phase 1: Threat-Driven Prioritization

Do not start by asking "what can I detect with my logs?" Start by asking "what threats matter most to my organization?" Then determine what data you need and what rule logic would detect each threat.

Use MITRE ATT&CK to systematically identify techniques relevant to your threat model. Map your current detection coverage against the framework. The gaps are your rule development backlog, prioritized by:

Business impact if the technique succeeds undetected
Likelihood based on your threat intelligence
Data source availability (can you actually detect it with current logging?)

Phase 2: Rule Development

Write the rule logic with explicit attention to:

Precision over recall: A rule that catches 70% of true positives with a 5% false positive rate is better than one that catches 95% of true positives with a 40% false positive rate. Missed detections are bad; buried detections are worse.
Testability: Can you reproduce a true positive in a test environment? If you cannot trigger the rule deliberately, you cannot validate it.
Documentation: Write the rule's purpose, expected behavior, known limitations, and tuning guidance alongside the logic. Future maintainers (including you in six months) need this context.

Phase 3: Validation Against Historical Data

Before production deployment, run the rule against 30-90 days of historical log data:

True positive rate: Does it catch known-bad activity from past incidents, red team exercises, or penetration tests?
False positive rate: How often does legitimate activity trigger it? Target below 10%.
Volume: How many alerts per day/week? Is this workload manageable per analyst?
Edge cases: What legitimate workflows look similar to the attack pattern?

If the false positive rate exceeds 10%, tune before deployment. Adjustments: tighten thresholds, narrow time windows, add excluding conditions for known-good patterns, increase minimum event counts.

Phase 4: Staged Deployment

Deploy rules in stages:

Shadow mode (1-2 weeks): Rule fires internally but does not alert analysts. Monitor volume and false positive rate.
Low-severity (1-2 weeks): Rule generates alerts at reduced severity. Analysts review without SLA pressure.
Production severity: Rule fires at intended severity with full SLA. Monitor for the first 30 days.

Staged deployment catches environment-specific false positives that historical data did not reveal.

Phase 5: Ongoing Tuning

Detection rules are not write-once artifacts. Establish a recurring tuning cycle:

Weekly: Review top 10 highest-volume rules. Is volume appropriate or indicating a tuning problem?
Monthly: Review false positive rates by rule. Tune or disable consistently poor performers.
Quarterly: Review detection coverage against ATT&CK. Identify gaps. Develop new rules.
After every incident: Did existing rules detect this incident? If not, why? What rules are needed?

Phase 6: Retirement

Rules that have not fired in 90 days need review. Either:

The threat no longer applies (retire the rule)
The data source stopped feeding events (fix the data pipeline)
The rule logic has drifted from the current environment (update the rule)

Dead rules consume SIEM resources and obscure coverage metrics. Remove them.

Detection as Code

Treat detection rules like software: version-controlled, peer-reviewed, tested in CI/CD, and deployed through a pipeline.

Why Detection as Code

Audit trail: Every rule change is tracked with who, when, and why
Peer review: Rule logic is reviewed by a second engineer before deployment (catches false positive risks)
Rollback: Bad rules can be reverted instantly
Testing: Automated tests validate rule behavior against synthetic events before deployment
Consistency: Rules follow a standard schema across the team

SIGMA for Portability

SIGMA rules provide a vendor-neutral YAML format for detection rules. Writing rules in SIGMA means they deploy across different SIEM platforms without rewriting logic. The SIGMA ecosystem includes:

Community-maintained rule repository (thousands of rules covering ATT&CK techniques)
Converters for Splunk, Elastic, Microsoft Sentinel, and others
Validation tooling that checks rule syntax and logic errors

For organizations running multiple SIEM platforms or considering migration, SIGMA prevents vendor lock-in at the detection layer.

CI/CD Pipeline for Detection Rules

A detection engineering pipeline includes:

Lint: Validate rule syntax and required fields (data source, severity, response guidance)
Unit test: Run rule against synthetic events that should trigger (true positive) and should not trigger (true negative)
Integration test: Deploy to staging SIEM, inject test events, verify alert generation
Performance test: Measure rule execution time against production-volume event rates
Deploy: Push to production SIEM with appropriate staging

Layered Detection

No single rule catches everything. Layer detection at multiple stages of an attack:

Kill Chain Stage	Detection Approach	Example Rule
Initial access	Authentication anomalies	Failed auth spike from single IP then success
Execution	Process behavior	Word.exe spawning PowerShell with encoded command
Persistence	Configuration changes	New scheduled task created by non-standard process
Privilege escalation	Token/group changes	Token manipulation or admin group modification
Lateral movement	Cross-system authentication	RDP from non-admin workstation to server
Collection	File access patterns	Bulk access to restricted file shares
Exfiltration	Outbound anomalies	Large outbound transfer to IP with no prior history

Each layer catches attackers who evade prior layers. Defense in depth applies to detection as much as it applies to prevention.

Detection Rule Metrics

Track these to assess your detection program's health:

Metric	Target	Action if Missing
True positive rate per rule	Above 80%	Tune or retire rules below 50%
Mean time from fire to triage	Under 15 min (Critical)	Reduce alert volume or add analysts
Rules with zero fires in 90 days	Under 10% of rule set	Review data sources and retire dead rules
Alert volume per analyst per shift	20-40 actionable alerts	Tune thresholds or add correlation
ATT&CK technique coverage	Expanding quarterly	Develop rules against priority gaps
Mean rule age	Track, no fixed target	Older rules need more frequent review
Time from CVE publication to detection rule deployment	Under 48 hours for Critical CVEs	Pre-build templates; Log4Shell was 0-day to exploit in hours

Key Takeaways

Detection engineering is software engineering -- rules require development, testing, peer review, staged deployment, and maintenance
A detection rule with a 50% false positive rate is worse than no rule: it trains analysts to ignore the category and creates a persistent blind spot
Start with threats (MITRE ATT&CK gaps in your environment), not with available data -- then verify the required data sources actually exist and ingest reliably
Validate every rule against 30-90 days of historical data before production deployment; target below 10% false positive rate
Stage deployments: shadow mode (1-2 weeks) then low-severity (1-2 weeks) then production severity
Detection-as-code (version control + CI/CD + SIGMA) provides audit trail, peer review, rollback, and automated testing
Layer detection across all kill chain stages -- each layer catches attackers who evade prior layers
Retire rules with zero fires in 90 days after investigating whether the silence indicates a dead rule or a broken data pipeline

Frequently Asked Questions

How quickly should we deploy a detection rule for a new critical CVE like Log4Shell?

Under 48 hours from publication for critical CVEs with known active exploitation. The practical approach: maintain rule templates for common exploitation patterns (JNDI injection, deserialization, SSRF, command injection) and adapt them to specific CVE indicators when advisories drop. Log4Shell detection rules (JNDI strings in HTTP headers) could be written in 30 minutes by a team with templates ready. The 48-hour target covers writing, testing against synthetic events, validating against your specific environment, and staged deployment. For CVEs in the CISA KEV catalog, treat deployment as an operational SLA.

How do we measure detection coverage objectively?

Map your active detection rules to MITRE ATT&CK techniques. For each technique in your threat model, classify coverage as: (a) no detection rule exists, (b) rule exists but untested, (c) rule exists with validated true-positive rate above 50%, (d) rule validated and exercised in red team or tabletop. Report coverage as the percentage of in-scope techniques at level (c) or (d). Most organizations start at 20-30% and should target 60-70% within 12 months. 100% coverage is neither achievable nor necessary -- prioritize by threat model relevance.

Should we write our own detection rules or use community/vendor rule sets?

Both. Community rule sets (SIGMA repository, vendor-provided rules) provide broad coverage quickly but are tuned for no specific environment -- expect 30-50% false positive rates without customization. Use community rules as a starting point, validate against your historical data, tune thresholds to your baselines, and add allowlists for your specific legitimate patterns. Then write custom rules for your unique threat surface: your specific applications, your specific network topology, your specific user behavior patterns. Custom rules typically achieve the lowest false positive rates because they are written against known-good baselines.

What is the relationship between detection rules and SOAR playbooks?

Detection rules generate alerts. SOAR playbooks define the response. Each detection rule should map to a playbook that specifies exactly what an analyst does when the rule fires: investigation steps, triage criteria, containment actions, escalation triggers. Without this mapping, alerts generate questions instead of actions. The maturity progression: (1) rule fires, analyst improvises response, (2) rule fires, analyst follows documented playbook, (3) rule fires, SOAR executes playbook steps automatically with analyst approval at decision points, (4) rule fires, fully automated response for high-confidence scenarios.

How do we handle detection rules for encrypted traffic we cannot inspect?

You cannot detect threats in traffic you cannot see, but you can detect behavioral patterns around encrypted sessions. Focus on metadata: connection frequency, timing, duration, volume, destination reputation, certificate properties, JA3/JA4 fingerprints. A detection rule that fires on "encrypted connection to rare domain using uncommon TLS fingerprint with periodic beaconing interval" catches C2 traffic without decrypting it. For environments where TLS inspection is deployed, layer content-based rules on top of the metadata rules. Neither approach alone provides full coverage.

How Advisedly Helps

Advisedly provides a detection rule library aligned with MITRE ATT&CK, with rules pre-tuned for common environments and a testing framework that validates new rules against historical data before deployment. The platform tracks detection coverage across ATT&CK techniques, identifies gaps in your rule set, measures true positive and false positive rates per rule, and provides tuning recommendations based on your specific alert volumes. For organizations adopting detection-as-code, Advisedly integrates with version control workflows and supports SIGMA-format rule import across your existing SIEM infrastructure. Contact begin@advisedly.ai to evaluate your detection coverage against your actual threat model.