The Curse of the Medium-Priority Alert

What comes back to bite us when we stop looking?

Photo by David Clode on Unsplash

As a security analyst, it is likely that a lot of your time is spent on high priority alerts, especially any that occur with regular frequency. Your alert criteria will be highly tuned for these frequent, high-priority alerts providing a high level of confidence when triggered. Your playbooks will be fairly mature for these common attacks, such as phishing attempts or common malware behaviors. You may have even already invested in developing automation for these playbooks.

But what about the medium-priority alerts?

Even medium-priority alerts that occur with any regular frequency are likely less tuned, and when reviewed have less available data, raising the probability of false positives or capturing minor IT operational issues rather than threat activity. Playbooks for these alerts, if they exist, are likely generic descriptions of common investigation tactics.

If your alerts follow a typical bell curve, medium-priority alerts represent your largest volume. Hence, the medium-priority conundrum: Do you investigate?

If it is a credible threat indicator early in its attack life cycle then it is an optimal time to mitigate before incurring any damage. But is it worth the time to investigate over other higher priority alerts? What amount of time should you invest?

The problem with investigating a medium-priority alert is a lack of corroborating evidence. It is possible that this is an early threat indicator, but if the malware/threat actor is intentionally pausing before taking its next step there is little supporting evidence yet of a threat. Adding the user/host to a watch list may catch followup activity only if the malware/threat actor takes known steps and in a time interval that aligns to your security controls’ analytics.

This is effectively the ‘wait and see’ approach. It leaves a nagging feeling for any security analyst: Is this going to come back to bite me?

A concern with the ‘watch list, wait and see’ approach is the difficulty in connecting alerts. Two weeks later not many analysts will recognize a new high priority alert involving a different user account on a different system is actually connected to the previous medium-priority alert without a lot of digging. Most SIEMs perform investigations at the log level without providing any context to previous alerts. And if the threat actor deliberately takes actions to obfuscate their steps, crude aggregation methods such as compiling alerts from the same user/system will not be effective anyway.

Since breaches rarely occur in a single step, establishing connections across alerts is extremely valuable. Attacks evolve over time and leave bread crumbs along the way. Tracking the relationship across low, medium, and high priority alerts surfaces “constellations” of alert activity that more accurately recognizes qualified threat activity than investigating a single high priority alert. However, due to the threat actor’s lateral movements and changes in user accounts, security analysts cannot rely on crude alert aggregation methods.

Linking alerts together requires a linkage at the operational level. For example, take these two medium-priority alerts:

1. User-String: ‘nmap’ used on multiple apache http access requests on server: web-server
2. Multiple authentication request failed for user ‘rob’ followed by successful authentication on server: File-Server

On the surface these alerts do not seem at all related. Exploring the web server and system logs reveals:

1. External IP requests index.php using user-string nmap <medium-priority alert #1>
2. External IP requests http access for webpage joomla/configuration.php
3. index.php re-written
4. External IP request http access for index.php
5. User ‘www-data’ runs command ‘cat /var/www/apachw2/joomla/configuration.php
6. User ‘www-data’ runs mysql -u joomla
7. ‘Rob’ fails then successfully authenticates from ‘web server’ to server ‘fileshare’ <medium-priority alert #2>

Seeing two connected medium-priority alerts changes the situation completely. It now appears that both the web server has been compromised with a reverse shell, mysql has been compromised to some degree, and the user account ‘rob’ is compromised and has access to an internal file share. This is clearly the early stages of an alert, however many organizations would not have dedicated the time to investigate the medium-priority alerts. The two alerts have different sources, destinations, and users. But they are connected via a string of operational transactions.

Using operational transactions to connect alerts changes security operations’ prioritization to focus on ‘constellations’ of alerts that substantiate threat activity over single high priority alerts. Similarly, alerts with no connections to other alerts may be dismissed as a false positive.

With operation activity binding alerts together, security teams are in a stronger position to recognize early threat indicators to prevent data breaches and larger-scope compromises.