The Infinite False Positive Phenomenon
Insanity is doing the exact... same thing... over and over again expecting... things to change...
Farcry 3 (2012)
while (true) {False Positive;}
During my time working as a Blue Team analyst in a Security Operation Centre (SOC), by far one of the most insufferable aspects of the job would be these annoying, persistent False Positives that fired over and over and over ….. again. They could be False Positives or Benign Positives technically.
Aside: In the realm of threat detection, when a threat is identified, analysis must be performed to determine if this identification is genuine or a mistake. See below for an explanation of these outcomes.
I use the terms “alert”, “threat detection” and “use case” fairly interchangeably.
Overview of triage outcomes
Benign and False Positives are inevitable. If you aren’t getting any from your threat detections, they are probably too specific and risk missing actual True Positive events. To reiterate; BPs and FPs are not inherently bad.
But recurring BPs and FPs are bad.
They cause unnecessary noise, taking Blue Team focus away from handling important tasks and they contribute to alert fatigue, whereby an analyst’s sharpness for a particular alert dulls with each unnecessary BP/FP.
I’m sure every SOC has those moments of “Oh it’s just another _. Nah we get heaps of those, it’s nothing.“
Begs the question: why alert on it in the first place? Obvious solution: don’t alert on it!
This continuous alerting on known recurring BPs and FPs is what I coin…
The Infinite False Positive Phenomenon
Yes, it can apply to BPs as well. But for brevity, I will refer to just FPs.
It constitutes a SOC that is aware of a flaw in their threat detections, but this flaw just never gets solved and never will get solved. So the BPs and FPs keep popping alerts. And it just becomes part of the day-to-day job.
what’s the solution?
Like I said: DON’T ALERT ON IT. As in, fix or tune the threat detection so it will not trigger for events or conditions which are explicitly linked to these recurring BPs and FPs. But a SOC can operate in a way where the tuning never gets completed successfully.
roadblocks to tuning
No accountability
By far the biggest reason that recurring FPs just stay dormant, is that even if the team identifies a threat detection requiring tuning, or an analyst is tasked with tuning an alert, there is no formal accountability for it apart from a verbal agreement.
As soon as work picks up and things get busy, this verbal agreement is forgotten by all, and the task of tuning the alert fades into the void of the underworld.
Therefore it is vital that meaningful accountability (not just verbal agreement) be attributed to the tuning task. This means - create a ticket and assign it to the SOC or analyst’s queue. Unless the tuning work is done, such that the ticket is resolved by someone, there is always a reminder of accountability to perform the required tuning task and it will never fall victim to “we got busy, I forgot” - since the ticket will be in a queue no matter what.
Poor habits
When a FP or BP is identified as the outcome of an investigation, there should almost always be a Post-Incident Activity of tuning the threat detection, if this event has a recurring nature.
A habit should be encouraged, from on-boarding of analysts to just general ticket approach, to perform tuning whenever appropriate. The easiest trigger for this is to tune upon a False Positive outcome.
if ( false positive ) => { TUNE }No follow-up
Analysts may occasionally forget to abide by the above best practice of always performing tuning following a FP outcome. Therefore, a higher up eg. Team Lead has to take responsibility of monitoring the alerts with the highest number of False Positives - as they likely require tuning. This starts with the awareness of which alerts are causing the highest False Positive counts, which could be achieved through a SIEM dashboard or a Case Platform dashboard.
False Positives by Alert - Weekly
The Team Lead can delve into the top three alerts with the highest number of False Positives for the week during a weekly meeting with the operational team potentially more if in a quiet period. A discussion can be opened with the analysts that perhaps understand the reason for the FP outcome of an alert so the SOC is aware of it.
Then single analysts can be tasked with the tuning of a particular alert, with the workload being distributed across the SOC team. Remember - accountability! They are tasked through the assignment of a ticket, not a verbal agreement.
I chose to suggest a weekly FP analysis as this isn’t too frequent to burden the team unnecessarily, but is frequent enough that recurring FPs get dealt with before they cause alert fatigue.
Lack of skills
At the end of the day, even with these best practices and habits above, if the tuning activity isn’t completed successfully, the Infinite False Positive Phenomenon remains. This really boils down to two situations:
1. SOC team lacks particular skills
2. Requirement for better upskilling culture
If an entire SOC team is hired where everyone lacks the skills to complete the tuning task successfully, this is a mammoth of an HR failure. Not saying it could never happen. But clearly better technical interviews and vetting of candidate’s SIEM and programming skills need to be performed.
Assuming at least some individuals in the SOC team have the required aptness to tune successfully, analysts which lack those skills should be encouraged to shadow them and also be proactive in their upskilling ambitions:
Read up on documentation
Complete paid or free training
Experiment with SIEM querying
…in the end
By ensuring meaningful accountability of tuning tasks, forming solid tuning habits upon FP outcomes, having frequent follow-ups into tuning requirements that may have been forgotten and ensuring the SOC team has the skills required to tune successfully - the Infinite False Positive Phenomenon should be defeated.
This will drastically improve alert fatigue and subsequently boost morale. I can tell you firsthand there is nothing more undermining than a SOC which doesn’t implement proper tuning practices and leaves analysts smashed by recurring FPs. The phone calls, admin, meaninglessness of the alerts - will leave you questioning “Am I in cybersecurity or admin?”
Following best tuning practices is a vital but often overlooked operational aspect of a SOC. I really hope that this blog post can assist teams who have found themselves in my boat :)
