Security information and event management (SIEM) came together from what seemed like a natural pairing between SIM and SEM solutions. But instead of producing a great combination, like a good peanut butter and jelly sandwich, SIEM as it’s known today is a complex and burdensome term with a lot of legacy baggage.
While SIEM often describes a tool for SOC analysts that collects event data and log information (SIM) and then analyzes and prioritizes the data collected from security events (SEM), almost every SOC has several so-called “SIEM” tools to optimize the event prioritization and response efforts.
Ultimately, the SEM functions of SIEM solutions are falling short, and SOC analysts need something better to serve their SEM needs.
Read on to understand how this merger occurred and the challenges of today’s SEM.
The first concept of SIEM was created in the late ‘90s as network and security teams looked for ways to consolidate the event log information from their various devices to a central location.
The centralized logging function of SIEM started as Security Information Management (SIM). As technology evolved, central data collection wasn’t enough. SOC analysts wanted to start applying logic to the data to define patterns of known bad activity and be alerted in more real time that this activity was happening instead of reacting only to post mortem analysis. This gave rise to Security Event Management (SEM).
SEM allowed the data to be analyzed with matching lists, mapping network traffic, grouping critical systems, and even joining various events and alerts together to look for a sequence of activity instead of individual events. SEM also introduced features for managing investigations through case management and response workflows. This included grouping related alerts to a “case, “assigning cases to a user or team, and then tracking the activity and information until the investigation could be closed with an outcome.
The SIM portion of SIEM provides organizations with data log collection and aggregation. This serves as a central repository for all event logs that are generated in an environment. From there, there are many things that can be done with the data. The core goal of the SIM is data retention and reporting to meeting compliance requirements. Many regulations require organizations to maintain all log messages from three months to several years, depending on the requirements, something SIM tools are well suited for. In addition, most SIMs come pre-packaged with out-of-the-box compliance reporting aligned with common compliance regulations like SOX, PCI, & HIPAA to name a few. This works by summarizing activity from various systems to demonstrate adherence to standards when the auditors arrive. [PT1] [AM2] Lastly, centralized logging makes it possible to conduct post mortem investigations though generally limited to logs alone. Providing centralized search on IPs, hosts, users, files and more.
It’s worth noting that early in the SIM and SEM “merger,” data rates were still very low, and data was only consumed from a few devices. The typical data feeds to a SIEM were IDS, firewall and router/switch traffic. Keep in mind that in the early- to mid-2000s, big data and distributed processing technologies didn’t exist in a readily consumable fashion. The primary databases were relational and were not designed to process data in the manner needed for SIEM.
The explosion in data volumes came from broader adoption and availability of the internet and technologies to support the business. As data sources started to expand, the volume of events being collected grew exponentially. Today’s advanced security tools along with more verbose logging from operating systems and other applications are a common part of the security monitoring ecosystem.
So where did it all go wrong? My grandfather used to say “right tool, right job,” meaning you don’t use a screwdriver as a hammer and vice versa, which seems like common sense but it’s not always so simple.
First, let’s explore the technology and landscape, SIEM is not new, starting in the late 90s as stated above these tools were built on legacy technologies to solve the problems of that day. While incumbent vendors have tried to pivot underlying technologies to evolve from Oracle to proprietary databases, it was clear they couldn’t build one tool to do both jobs. So you end up with a single vendor perhaps but all SIEM vendors of today have numerous products for customers to buy, deploy, and manage.
Don’t agree? Let’s look at some examples.
Splunk has Splunk Enterprise for SIM, and Splunk Enterprise Security (ES) for SEM, ArcSight has the ArcSight Data Platform (Formerly Logger) for SIM and Enterprise Security Manager (ESM) for SEM. How about LogRhythm? They still have Data Collectors, Processors and Indexers for SIM and a separate AI Engine for SEM cleverly connected together via a Platform Manager web console.
Note that all these solutions are buried in technical debt and spend more time trying to port what was to work today than iterate and innovate on what should be. These are not small or easily portable widgets but instead complex and monolithic code bases making this task all but impossible unless starting anew with a purpose-built solution.
A modern SEM tool should understand the data presented to it and almost think like an analyst if you need information to complete an investigation that data should be collected for you and presented in a clean and easily digestible form via orchestration and automation. Modern SEM tools should not stop at the data either but strive to understand the relationships between the data and be able to provide the much-needed context around that data to tell a story to the analyst narrating what was observed. It should not overwhelm them with mountains of alerts and 40 tools to pivot through in order to aggregate information.
Should you have to buy a separate network monitoring tool, user behavior tool & SOAR tool just to accomplish the mission of reducing the risk of your organization and providing a highly capable security operations function. I would argue most certainly not. The security industry has shown time and time again that throwing more tools at the problem and having point solutions as band-aids is a broken model. We need to follow Steve Jobs adage and “Think Different”, even think smarter, and approach this problem in a new light. Leveraging the latest technologies and working from the mindset of the analyst to solve the problems SEM originally set out to solve, not in the way it has been done but the way that it should be.
Because of their distinct functions and differing goals, by and large, the industry today views log management and event management as two, distinct functions. Although SIM is often lumped in with SIEM, it is a separate functional product than the SEM. Hence there is no I in SEM.
Organizations can and should decouple their SEM to truly meet their needs. Meaning, SOC analysts can maintain SIM for its log management function and should adopt a SOC platform that meets their intended goals for a SEM solution: faster, easier incident discovery and triage.