The Imperative for Real-time Speed and Unlimited Lookbacks

In the nonstop turbulence that characterizes cybersecurity, Chief Information Security Officers (CISOs) play a pivotal role in piloting their organizations through an increasingly sophisticated array of threats. One of the most critical components in their arsenal is the Security Information and Event Management (SIEM) system. SIEMs are the central nerve centers that aggregate, analyze, and correlate security data from various sources to quickly detect and respond to threats. However, to stay ahead of adversaries and mitigate potential breaches, CISOs must understand the need for real-time speed in log queries and the ability to support unlimited lookbacks across all internal telemetry correlated to external threat data.

The challenge of real-time speed

Milliseconds matter when it comes to detecting and responding to security incidents. Attackers are relentless, increasingly subtle/sophisticated, and move swiftly to infiltrate an organization’s network, exfiltrate sensitive data, or launch damaging attacks. To effectively counter these threats, CISOs need a SIEM system that can process and query log data in real-time and at scale.

Data Volume and Velocity:  Organizations generate vast amounts of log data from various sources, including firewalls, intrusion detection systems, and endpoint devices. This data must be ingested, parsed, and analyzed in real-time to separate signal from noise, identify anomalies, and prioritize potential threats. Being able to immediately identify subtle patterns in large data sets and draw prioritized correlations is an area where AI-enabled SIEMs should excel.

Advanced Threats: This is not just about processing a high volume of incidents. Zero-day exploits, and APTs (Advanced Persistent Threats) can often deliver sophisticated multifaceted attacks that require immediate attention. A SIEM that can immediately identify significant threat patterns of behavior to deliver a holistic view (from threat intelligence through to the Security Operations Center) of real-time query results enables security teams to respond promptly and in context to emerging threats, minimizing potential damage.

Operational efficiency: Real-time log query capabilities enhance the efficiency of security operations. Analysts can immediately identify false positives, investigate incidents, make informed decisions, and quickly distribute critical information where it is needed (e.g. telling firewalls to block messages with a specific profile), reducing the mean time to detect and respond.

Achieving real-time speed in log queries involves several considerations:

Scalability: The SIEM platform should be capable of scaling horizontally to handle the increasing volume and variety of log data generated by the organization, particularly when dealing with multiple, distributed log sources. Dynamic scalability also requires automated load-balancing to avoid performance bottlenecks and facilitate the prioritization of critical security events. This is particularly important given the complexity and distributed nature of both IT and OT systems.

Data Ingestion Optimization:  Given the volume of data being ingested in a SOC, correlating real-time (or streaming) analytics to the external threat environment allows security teams to analyze and act on contextualized security events as they occur. Working from a cloud-native (and serverless) architecture also allows SOCs to instantly scale resources on demand, while lossless compression algorithms can help reduce data storage requirements without compromising integrity. Efficient log data ingestion mechanisms, such as log shippers can also minimize delays in processing and indexing incoming data. 

Indexing and search optimization: Query languages are complex, often vendor-specific, and require years of experience to develop proficiency. Advanced indexing and search algorithms using an NLP-based interface are quickly becoming table stakes for accelerating query performance. Techniques like caching or parallel processing can also significantly enhance the speed of log queries. The SIEM system must also provide efficient search and retrieval mechanisms for historical data, enabling analysts to retrieve information quickly. Ideally within seconds across petabytes of data.

Machine learning integration: The volume of incidents that need to be processed (with all the multiple steps involved) is well past any human’s ability - and has been for years. Leveraging machine learning models for anomaly detection and threat scoring across hundreds of thousands or millions of incidents in real time can enhance the SIEMs' ability to identify emerging threats quickly. As AI is rapidly adopted across the industry, this is becoming a standard operating procedure for most organizations.

Unlimited lookbacks: a strategic necessity

While real-time speed is crucial for handling immediate threats, the ability to support unlimited lookbacks across internal telemetry and correlate this to external threat data is equally vital for comprehensive threat analysis, historical context, and compliance requirements.

Historical analysis: Cybersecurity incidents often involve long, complex attack chains spanning weeks or months. To understand the full scope of an incident and identify its origins, CISOs need the ability to look back at historical data. The normal lookback period for most SIEMs is 90 days, while the actual scope of an event can easily go back well past that. Without unlimited lookbacks, critical pieces of the puzzle may be missing. This is also a core component of data storage and retention policies. Implementing and enforcing a robust data retention policy (which is likely to vary by industry) combined with cloud-scale storage solutions is essential for maintaining historical log data.

Compliance and Auditing: Many industries and regulatory bodies require organizations to retain security data for extended periods. Compliance audits often require the ability to review historical logs to demonstrate adherence to security standards and incident response procedures. Rather than sending your auditors on a snipe hunt, a properly configured SIEM can generate historical audit trails on demand. This is particularly useful for non-optional short timeframe-sensitive compliance mandates like SEC Form 8K.

Threat Intelligence integration: Threat intelligence feeds can include indicator-based, vulnerability-based, campaign (TTPs) or attribution-based, OSINT, and even Dark Web data, providing valuable information about emerging threats. The ability to correlate historical data with threat intelligence feeds can help organizations proactively defend against known attack vectors. This takes the question “Who is attacking us?” and extends it to “And where exactly are we exposed to this particular threat?”.

Striking the right balance

CISOs face the challenge of striking the right balance between real-time speed and unlimited lookbacks. While real-time capabilities are critical for immediate threat detection and response, the ability to access historical data is equally essential for comprehensive threat analysis, compliance, and long-term security strategy.

Tiered data storage: One strategy to address this challenge is implementing tiered data storage. Real-time data can be stored in high-performance storage systems, while historical data can be moved to cost-effective, long-term storage solutions. Or, as an alternative:

Datalake:  Ideal for large volumes of raw diverse data that will be subject to analytics, and offers better flexibility for a broad range of data types without the need for upfront structuring. If you plan to throw AI (specifically LLMs) at large data sets, this is a good option.

Retrospective analysis: CISOs should encourage their security teams to conduct retrospective analysis of historical data periodically. This practice (particularly around dark data) involves revisiting historical logs to uncover previously undetected threats or patterns that may have gone unnoticed.

Incident response playbooks: Developing automated incident response playbooks that incorporate both real-time and historical data analysis can help streamline and accelerate response efforts for various types of incidents.

Conclusion

CISOs must recognize the strategic imperative for real-time speed in log queries and the ability to support unlimited lookbacks across all internal telemetry and external threat data. Real-time capabilities are vital for immediate threat detection and response, while unlimited lookbacks empower organizations to conduct comprehensive and contextualized threat analysis, meet compliance requirements, and strengthen long-term security strategies.

To achieve these objectives, CISOs must collaborate closely with their security teams and SIEM vendors, continually assess their SIEM architectures' scalability and performance, and implement robust data management and storage strategies, By striking the right balance between real-time speed and unlimited lookbacks organizations can bolster their cyber security posture and stay one step ahead of evolving threats in an increasingly digital world.