Why Log Analysis is Still Our Primary Security Investigation Tool
It started with a moderately prioritized Security Information and Event Management (SIEM) alert triggered by the firewall allowing external traffic right after the Intrusion Detection System (IDS) blocked an attack from the same external IP. And now the tabs on your browser are growing as you review and compare firewall logs, IDS logs, host activity logs, user authentication logs, VirusTotal lookups, Active Directory user lookups, Configuration Management Database (CMDB) lookups, and views from other internal and external tools.
The security vendor’s demonstration portrayed investigations from an alert look so easy. They only briefly showed log tables and mostly advanced their investigation by clicking on visualizations such as pie and trend charts. During the demo, they quickly identified the host visible in a narrow pie slice is related to the user in a smaller pie slice on a different chart on a different page. And they completed the investigation in less than 2 minutes. But actual investigations rarely resemble sales demos. The pie charts, event counts, and trend charts only give hints and not enough credible, detailed information to make true determinations to qualify the threat, understand its entry point and its current scope.
In sales pitches, many sales engineers suffer from the Curse of Knowledge. They know the conclusion of the story before it begins, leading to a set of unnatural steps during the investigation. In hindsight, why did they double-click on that pie chart? Would you have done the same?
Instead, security analysts compare log tables to see what actually happened and to manually connect activities based on timestamps to understand the sequence of related events. Log analysis is not the glamorous side of blue team activity. It’s challenging, tedious work. But it delivers the level of detail required by a security analyst to make critical decisions during the investigation. Using the previous IDS/FW alert as an example, the analyst needs to answer:
- What process established the network connection out to the external IP?
- What service owns that process and who is the user?
- Was it spawned from an interactive or non-interactive shell?
- When did that service initiate?
- What files/configurations does that service rely on and have these recently changed?
- Is that service installed on other machines or unique to this one? Is that user logged in anywhere else (as anyone else)?
- And the list goes on….
Note, these activities may be separated by hours, days, even weeks before the alert triggered.
To answer these questions, security analysts perform pivot searches e.g. searches on the system log data, network traffic, or user activity before and after the alert. This results in pages of search results which mix both superfluous and pertinent data to the investigation. Event classifications and user risk scores help to identify broad areas to review, but ultimately security analysts use log and audit data to ultimately understand what happened and in what order. The sequence of the log data will determine if there are qualified threat indicators and if so, what is its root cause and scope.
Security analysts use audit and log data because it is a trusted source of information, whereas security events, detected anomalies, and visualizations infer results that introduce a probability of error.
Alerts generated by correlation rules or behavioral anomalies will always need to be validated.
While the security industry will (and should) continue to focus on better threat detection, security analysts will also benefit from a tool focused on the investigation that:
- Establishes the causal connections of activities leading to and following a security alert.
- Bases causal connections with ground truth data, rather than inferred data, to become trusted sources of relevant information.
Establishing causal connections based with ground-truth data leading to and following a security alert removes the most challenging and tedious process performed by the security analyst, enabling security analysts to make faster and more accurate determinations to mitigate qualified threats.
