Why Security Analysts Don’t Trust Alerts

Photo by Lisa Zoe on Unsplash

Think about what it means to be a security analyst. Do you have a picture in mind? The variety is amazing. Some work in mature security operation centers with rows of dual monitors and the glow of mechanical backlit keyboards. Some are not even a security analyst by title. The title may be “IT Specialist” and yet take on security responsibilities. And there is everything in between.

Across this variety there is a universal truth. Even with all the technology at our disposal, threat investigations remain time consuming and challenging. Ponemon’s 2020 Cost of a Breach Report captures the average time to identify and contain a data breach was 280 days, nearly identical to the average of 279 days in 2019.

This seems counterintuitive when you consider that as security analysts, we have access to more data than ever before:

• Our endpoints are heavily monitored and protected with advanced heuristics and behavioral modelling
• Our networks are heavily monitored with ubiquitous technologies like netflow
• It’s easier than ever to collect and search across custom application logs
• We have access to contextual data about our internal systems and users
• We have access to threat intelligence, both public, in focused industry groups, and paid-for curated content inserted directly into our security controls

And perhaps that’s part of the problem. We have more data than ever without any insight as to whether or how this data applies to a specific threat investigation.

I began to understand why people outside cybersecurity do not recognize the challenges of threat investigation during a recent conversation. They watch police dramas like CSI and see a detective enter a crime scene and immediately they can recognize the crime (a dead body) and the murder weapon (he was shot). Instead, imagine if during the CSI episode the phone keeps ringing off the hook, each reporting a crime, and each time they begin to investigate only one in a hundred calls is there actually a dead body.

In reality, we show up to the ‘crime scene’ without knowing if any crime occurred. The data we use to figure out ‘what happened’ is an odd mix of alert, audit, and meta data generated by a blend of both deterministic and non deterministic methods.

Take the case of an authentication failure:

Table 1: Examples of Authentication Failure across Data Types

The alert is the easiest to read but trusted the least. This results in security analysts investigating each alarm to qualify it (did a crime actually occur?). Investigation requires digging through meta data and audit data to find out what actually happened. This is usually performed by conducting pivot searches — e.g. show everything that happened on 192.168.1.13 or activities related to user ‘root’ around this particular time.

Here is what that might look like:

Figure 1: Operating System Process Data

And as a security analyst, our job is to scour through the log and audit data and make a determination of what may be connected to your investigation while trying to ignore the larger volume of superfluous data. It is time consuming, tedious work relying on experience and intuition. Even a highly experienced security analyst with previous exposure to investigating successful attacks won’t necessarily recognize the next attack.

So with all the data now available to security analysts it obscures true threat indicators during the investigation. If we can’t trust alert data and only have a slightly better trust in meta data, the solution to reducing investigation time is not more data!

If we accept the premise that alerts can never be fully trusted and will always require investigation, we must focus on improving the threat investigation to reduce threat recognition and containment time. This solution doesn’t add more and more data to the security analysts, but instead provides focus, direction and guidance on trusted, ground truth data. Trusted data that is focused to the audit and contextual data that pertains to the specific investigation, removing all the superfluous endpoint, network, user data and context that only obscure threat activities. With a clear view of trusted data, security analysts are enabled to make faster determinations to qualify threats and have faster recognition to a true threat’s scope and root cause.