Detection-as-code with Sigma rules that are measured, not just written. Every rule is scored for precision, recall and false-positive rate against labelled events, tested against real adversary telemetry, and blocked from merging if it drops below a bar it declares for itself.
🔎 Live dashboard: every detection, its scores and its test status. Hover any number for how it was calculated. 📖 How it works: the full method, written for a reader with no security background as well as one with.
- 🎯 20 Sigma rules, each ATT&CK-mapped and scored against hand-labelled benign look-alikes: 16 Windows rules also proven to fire on a pinned public EVTX capture of the real attack, and 4 Entra ID rules on a declared, weaker evidence tier. Cloud telemetry has no public captures, so those are field-validated against Microsoft's published table schema at compile time instead (ADR 0005).
- ⚖️ Two independent engines check every rule: Hayabusa over real EVTX, and a SQL evaluator over labelled events. They must agree; disagreement fails the build.
- 🚧 Every rule declares the score it must hit. CI fails when one drops below its own bar, and a lowered bar is rejected unless it carries a written justification.
- 🔁 Every published number is generated, then rebuilt in CI and diffed against what is committed, including the unflattering ones and including this file.
- 🪓 Noisy signal is split, not deleted. PowerShell obfuscation on its own fires on two thirds of ordinary admin scripting (measured), so that rule is
informationaland used for hunting; a second rule alerts athighonly when obfuscation meets an execution or download sink, at an FP rate of 0.11 on the first rule's own false alarms (ADR 0006). - 🔌 Deployable output: rules compile to source-bound Splunk SPL, Microsoft XDR KQL, and Azure Monitor KQL for the Entra rules.
One command reproduces all of it from a clean checkout (needs uv):
uv sync --all-extras && uv run detkit ci # ~20s, the same sequence CI runsA devcontainer is included if you'd rather not install anything locally.
Read FP rate first: it is the share of benign events that alerted, and unlike precision it cannot be improved by writing more attack cases. Precision moves with the malicious-to-benign ratio of the case set, which is authored here, not observed.
20 rules, 172 labelled events. Generated by detkit eval; CI fails if this table drifts from the corpus.
| rule | level | precision | recall | FP rate | bar it must clear |
|---|---|---|---|---|---|
entra_audit_privileged_role_assigned |
high | 1.00 | 1.00 | 0.00 | p≥0.90, r≥1.00, fp≤0.10 |
proc_creation_win_impacket_wmiexec_output |
high | 1.00 | 1.00 | 0.00 | p≥0.90, r≥1.00, fp≤0.10 |
security_win_dcsync_replication |
critical | 1.00 | 1.00 | 0.00 | p≥0.90, r≥1.00, fp≤0.10 |
security_win_service_installed |
high | 1.00 | 1.00 | 0.00 | p≥0.90, r≥1.00, fp≤0.10 |
posh_ps_obfuscated_payload_execution |
high | 0.83 | 1.00 | 0.11 | p≥0.80, r≥1.00, fp≤0.15 |
proc_creation_win_encoded_powershell |
high | 0.83 | 1.00 | 0.14 | p≥0.80, r≥1.00, fp≤0.20 |
proc_creation_win_firewall_disabled_netsh |
high | 0.75 | 1.00 | 0.14 | p≥0.70, r≥1.00, fp≤0.20 |
security_win_kerberoasting_rc4_request |
high | 0.75 | 1.00 | 0.17 | p≥0.70, r≥1.00, fp≤0.25 |
entra_signin_legacy_auth_success |
medium | 0.80 | 1.00 | 0.20 | p≥0.80, r≥1.00, fp≤0.20 |
sysmon_proc_access_lsass_dump |
high | 0.80 | 1.00 | 0.20 | p≥0.75, r≥1.00, fp≤0.25 |
entra_audit_illicit_consent_grant |
high | 0.80 | 1.00 | 0.25 | p≥0.75, r≥1.00, fp≤0.25 |
security_win_asrep_roasting |
high | 0.75 | 1.00 | 0.25 | p≥0.70, r≥1.00, fp≤0.30 |
security_win_scheduled_task_created |
high | 0.75 | 1.00 | 0.25 | p≥0.70, r≥1.00, fp≤0.30 |
proc_creation_win_certutil_download_decode |
high | 0.67 | 1.00 | 0.33 | p≥0.60, r≥1.00, fp≤0.40 |
posh_ps_defender_tamper |
high | 0.71 | 1.00 | 0.40 | p≥0.65, r≥1.00, fp≤0.45 |
entra_audit_app_credential_added |
high | 0.60 | 1.00 | 0.40 | p≥0.55, r≥1.00, fp≤0.40 |
security_win_user_added_to_privileged_group |
high | 0.60 | 1.00 | 0.50 | p≥0.55, r≥1.00, fp≤0.55 |
posh_ps_susp_encoded_powershell_scriptblock |
informational | 0.50 | 1.00 | 0.67 | p≥0.45, r≥1.00, fp≤0.70 |
security_win_eventlog_cleared |
high | 0.60 | 1.00 | 1.00 | p≥0.55, r≥1.00, fp≤1.00 |
security_win_local_user_created |
informational | 0.43 | 1.00 | 1.00 | p≥0.40, r≥1.00, fp≤1.00 |
Called out rather than buried:
- 🔴 Two rules still alert on 100% of benign events.
security_win_eventlog_clearedkeeps that bar deliberately: a cleared log carries no evidence of why, so no discriminator exists, and the argument for shipping it anyway is the base rate.security_win_local_user_createdcannot make that argument, so it is demoted to an audit record. Both justifications live in their case files. - 🔧 Two rules that did have signal were fixed rather than excused.
security_win_service_installedwent from 0.43 precision / 1.00 FP rate to 1.00 / 0.00, andsecurity_win_scheduled_task_createdto 0.75 / 0.25, by keying on what the service or task actually executes. - 🕵️ Measurement found a real evasion.
powershell.exeaccepts-eas an abbreviation of-EncodedCommand; the rule tested only the longer spellings. Closing it took recall from 0.80 to 1.00 at the cost of one false positive in seven, a trade the harness refused to accept until it was written down.
- 🧪 The benign events are authored, not captured. They prove a rule's logic discriminates; they say nothing about alert volume on a live estate.
- ☁️ The Entra rules are not proven against a real tenant. Their logic is measured and their queries are schema-validated, but nothing here replays them against captured telemetry, because none is published.
- 🧩 Sentinel coverage is partial, and it is a platform limit. Splunk gets all 16 Windows rules, source-bound; the Entra rules compile to Azure Monitor KQL bound to
SigninLogs/AuditLogs. Microsoft XDR gets theprocess_creationsubset: Sentinel'sSecurityEventtable has noPreAuthTypecolumn, so the Kerberos rules cannot bind there without per-ruleEventDataparsing. - 🏗️ Nothing here provisions a SIEM. Rules compile to deployable, source-bound queries; running them in a real estate is out of scope. See ADR 0004 on why there is no Terraform.
detections/ is my own work and carries every gate its telemetry can support. vendored/ is a pinned, attributed copy of the public SigmaHQ Windows corpus (~2,400 rules, DRL 1.1), not my work and held to none of those gates. It runs through the same conversion pipeline at scale and shades the coverage map blue; green cells are authored. The headline number is the green one.
uv run detkit validate # metadata + fixture + ATT&CK tag discipline
uv run detkit eval # precision / recall / FP rate per rule
uv run detkit convert # compile to Splunk + XDR, check every query is bound
uv run detkit probe <rule> # run a rule against its evidence (EVTX, or cases for cloud rules)
uv run detkit dashboard # rebuild site/index.html + site/about.html
uv run detkit readme # rewrite the table above from evals/results.json
uv run pytest -v # unit + detection tests (fetches samples, runs Hayabusa)detkit ci installs the Hayabusa release pinned in .hayabusa-version, verified against its recorded SHA-256, because an unverified binary download is not a reproducible build. Locally, without Hayabusa, the detection tests skip so you can still work on the tooling; CI sets DETKIT_REQUIRE_HAYABUSA=1, which turns any skip into a failure. A detection suite that did not execute must never report green.
The corpus is closed at 20 rules and the repository is in maintenance: the gates keep running, the pinned ATT&CK release and Hayabusa version get bumped when they move, nothing new is planned. Two remaining ideas were declined in writing rather than left on an open list: more Entra rules would repeat a pattern already proven four times, and a Splunk container replaying the same pinned captures would restate gates that already pass, since what a live SIEM would really add is production volume this project does not have. ADR 0007 makes both arguments.
- 📖 How it works: the pipeline, both test layers, every metric and what it hides, all the CI gates, and the objections this invites.
- 🔄
docs/detection_lifecycle.md: hypothesis → rule → fixtures → tests → conversion → ATT&CK → ship. - 🏛️
docs/adr/: the architecture decisions, including the rejected options. - 🗺️
coverage/navigator_layer.json: loadable in the ATT&CK Navigator ("Open Existing Layer" → "Upload from local").
MIT for the rules, tooling, tests and docs authored here. vendored/ is SigmaHQ's corpus under the Detection Rule License 1.1; see vendored/LICENSE.Detection.Rules.md. The split is spelled out in NOTICE.