The scenario and numbers are illustrative; the arithmetic was verified by script.
"This rule is useless, it's all false positives." Maybe the rule is good, and the math is the problem.
What you'll learn
- The confusion matrix as a shared language
- The base-rate fallacy, with numbers
- Tuning that doesn't add false negatives
- Why every exclusion needs FN verification
The story at a glance
Week 1: a rule for WINWORD -> powershell -enc floods the queue, mostly managed-software automation. Week 2: analysts start ignoring it. Week 3: the real attack is buried. They detected it and still couldn't respond.
Confusion matrix
| Attack | Normal | |
|---|---|---|
| Alert | TP | FP |
| No alert | FN (most dangerous) | TN |
Base rate, in numbers
1,000,000 events/day, 10 real attacks, zero misses, false positive rate 0.1%:
- TP = 10, FP = 999,990 x 0.1% = about 1,000
- Precision = 10 / 1,010 = 0.99%
Cut the false positive rate to 0.001% and precision rises to about 50%. Both figures were computed and verified in a script.
Tuning rules of thumb
- Baseline before you exclude.
- Exclusions: narrow and specific (full path of the managed product, decoded content, principal, signature), never "exclude powershell.exe".
-
-encis a signal, not a verdict: always decode. - Correlate in sequence and time window.
- After each exclusion, replay an attack and confirm it still alerts.
Operate with metrics
Precision, alert volume per day, top-firing rules, and a periodic review of exclusions.
Record why it was a false positive
"Looks like FP" is not a conclusion. Write the evidence, and escalate when unsure.
Next: running all this as a system, not heroics.
Adapted from chapter 32 of my Japanese book on server defense.
