If you’re in the IT security industry, you’ve certainly heard that IDS is dead. It’s funny to hear technology personified this way. Someone call 1110001111! The thought of a security technology expiring can be daunting, are we losing the battle? What’s the cause of death? We can include in the list of causes the high numbers of false-positives, lack-of security analysts, non-performant hardware etc. However in this post we’ll examine IDS evasion as one of them that contributed to its demise. If you’ve been around the security industry for a while, you’ll surely remember some of the techniques that we’ll explore in the Cat and Mouse section. After the nostalgia party, let’s change perspectives towards the future and take an optimistic look to what’s in store for the cyber warriors out there.
IDS can mean a lot of different things. Before we get started let’s disambiguate a bit. This article deals with network intrusion detection systems (NIDS). You know, the devices that hang out on the edge of your network, snooping on everything and sending you tons of alerts. It’s the one screaming at you to pay it attention, like the car alarm in the parking lot at 2:00am.
Circumventing IDS has been a game played for decades. Below I’ll reflect on some of the techniques that were successfully used to sidestep IDS detection. This is important to understand; the ease of avoiding security controls is a primary reason for the death of a security technology.
An obvious way to sidestep a network IDS is to simply scramble up the data in such a way that’s easy for the recipient to deal with, but difficult for the IDS. For instance, if a signature is looking for the phrase, “Command: Download all credit-card data”, we can simply break it into chunks, (“ownl”, “all cr”, “edit-c”, “Comm”, “ard da”, “and: “, “D”, “oad ”, “ta”) before sending it along out of order. At first this was done at layer four, the chunks were TCP/UDP datagrams. This is effective because the receiver of this message can easily reassemble it. If the receiver can’t, signaling for a retransmission is trivial. A network IDS that is listening to every conversation on the network cannot easily keep up and perform the reassembly required. IDS preprocessors were introduced to deal with this. With a layer-four statefull preprocessor, the IDS could reassemble the datagrams in the right order before checking for any pattern matches. We’re all fixed, right? Not quite yet, why not break the datagrams into smaller pieces and scramble them too? IP fragmentation is another technique to do the same thing, but now at layer 3.
This type of fragmentation technique requires low-level access to the operating system to perform, making it a heavier lift. It’s why we only see this technique used for attack traffic, vs. command-and-control, where much easier channels are available. It’s also easy to prevent: IP fragmentation on modern networks is very unusual and typically indicates a network configuration problem. Blocking IP fragments is safe to do and will prevent layer 3 fragmentation from being effective.
Web directory traversal was a popular way to allow a hostile web client to reach outside the bounds of the remote server’s filesystem. The technique allows attackers to not only access files that were not intended for anonymous web-client access, but also interact with servers in naughty ways. The technique manifested as a web request with a large number of parent directory paths chained together. Generally it looked something like this:
Those dot-dots are a relative link pointing to the parent directory; stringing together several parent links would allow the hostile actor to “traverse” the directory tree to areas of the server’s filesystem typically not for public consumption. The beauty here is that the parent of the root directory is itself. So, the bad guys don’t need to know how deeply nested the web content directories are,as there is no such thing as too many parent directory references.
To identify the activity, IDS signatures were created that looked for a series of “../” to alert security analysts that an attack was underway.
It wasn’t long before a disturbingly easy way around detection was discovered and used. By encoding the individual characters as their percent-encoded equivalents (i.e. ‘../’ becomes ‘%2E%2E%2F’), the cat and mouse game begins. Upon discovering this new technique in the wild, a signature is written looking for a series of (%2E%2E%2F)’s. We’re safe right? Nope. Now we encode the percent sign (%) with a technique called ‘double encoding’. After our defenses are fortified against double encoding; we should be good right? Nope, now let’s use Unicode…
Those in the industry often complain about IDS false-positives, but false-negatives, which by their nature are difficult to measure in an operational environment, are more highly likely and much worse for you then false-positives. So what’s the solution here? Think about it, what can we do so we’re not dependent on these strict signatures? We’ll get to that later in “The Future” section of this post.
Performing reconnaissance against your target is the first step for most nefarious activities. When this reconnaissance involves probing IP addresses and ports, your IDS is there to help you identify it. That is, unless it’s very slow. IDS detects network host reconnaissance with rate over time detections. If you move slow enough, you’ll slip right by. The IDS threshold can be loosened, however there’s a consequence. If you loosen the threshold too much, everything begins to look hostile; your IDS becomes paranoid. In other words, you will increase the false-positive rate so that the detection is rendered useless, like a whooping car alarm from the 90’s.
Establishing communications to external agents from a compromised machine is essential to hostile operations. Industry jargon calls this Command and Control (C2). C2 is the channel by which commands are conveyed, and sometimes information is extracted. Early C2 channels included static (always the same) IP addresses, and domain names. Although the protocols used varied, HTTP (TCP/80) became an early favorite as it’s almost always allowed outbound from your network, and is surrounded by loads of beautiful noise.
IP’s and domains were easy to understand and guard against by network defenders. They can be monitored, blocked, and sinkholed. IDS signatures will often include loads of IP addresses and domain names of known ‘bad’ assets. Even when protection in this way is effective, it’s a management nightmare. Most security operations I’ve encountered may have very good processes for reacting to new, known-hostile, IP addresses and domain names, but have little in the way of expiring these threat indicators. To find out if an organization has good management of these signatures, simply ask, “What’s your oldest hostile domain or IP address you watch for?” If they say, “I don’t know” or “This one from 1998…”, or “our Threat Vendor handles all of that”, you may have a problem.
The state of the art for attackers is now to use something much more unpredictable when establishing command and control: DGA domains. DGA stands for Domain Generation Algorithm. An infected host will use DGA’s to persist their C2 connection by using a quasi-unpredictable domain name that changes frequently. This avoids signature detection all together. IDS has no hope to identify these domains and the IP addresses associated with them; the hostile assets simply don’t live long enough to allow the IDS signature creation process to be effective.
We’ve talked about a few of my favorite evasion techniques, but the biggest problem with IDS is that most places are using the same signature set that can be easily tested against. If you’re a bad guy trying to avoid detection, it is very easy to manufacture a communication pattern (attack, C2 technique, etc.) since many organizations use the same signature set. Because of this, even novice hostile content authors can iterate over the weapon until it comes out squeaky clean. Got ETPRO? -> You just installed a lock on your network that every bad guy has an exact copy of to practice their picking techniques on. Sure, having an IDS with common signatures deployed is better than nothing, but not by much.
Like a phoenix from the ashes, the next evolution of security technology is rising. And it involves this subtle, but important, paradigm shift: The artifacts of active network monitoring should be a compact, detailed summary of ALL activity. They should not presume to detect attacks by emitting alarms while ignoring ALL other connections that did not match a pattern. They should emit high-confidence metrics of all activity occurring across the monitoring interface.
It’s what we do with the network summaries where the future detection paradigm becomes extremely compelling. We can apply statistical analysis, machine learning algorithms, and enhancements to the data. Most of these enhancements only become valuable within the frame of reference of the network being monitored, thwarting the single-lock problem. In addition, we can apply Artificial Intelligence to automate analysis, discovering patterns of interest from the mountain of data so fast as to compare it’s value with an entire SOC’s worth of trained analysts. (In fact, this is what JASK’s primary mission is, and we’re doing it).
Threat Intelligence now becomes much more powerful. Since most indicators of compromise from threat intelligence are contained within the network summaries (i.e. IP addresses, host/domain names, email addresses, file hashes, etc.), they can be examined after the fact. Intelligence indicating a hostile IP address yesterday offers little help to an IDS today. By storing these traffic summaries, we can apply threat intelligence over the period when it was known to be hostile, further empowering the analytics and analysts (AI or human).
So, let’s look now at the new paradigm against the attacks mentioned previously. First, the web directory traversal attack:
Here’s a sample of what a modern sensor would collect and report back of EVERY web connection. This is the “traffic summary” mentioned in this article.
Many enhancements can be attached to this summary, sourced from machine-learning algorithms, statistical analysis, static matches, etc. Here’s a few to get the juices flowing:
You should start to realize how difficult it’s going to be for an adversary to avoid tripping all these enhanced features around the attack. Many of them source from statistical inference (e.g. the one’s with ‘rare’ in the name are unusual considering what that host/service usually does). Let’s look at the other example we mentioned earlier: Command and control to a DGA domain.
Here are some enhancements that can be applied to this traffic summary:
What’s not shown above is the millions (or billions) of traffic summaries surrounding these two with nothing of particular interest bubbling up. It’s the AI’s turn now to stitch these two artifacts together, to begin telling a story of host compromise. Exciting, right? I think so.
So it seems IDS is dying. But it’s not dead yet. Many mature security operations rely on the IDS heavily. It’s still very good at identifying known remote access trojans (RATs), and being a configuration point by which human analysts can apply control. We looked at only a small sample of different IDS evasion strategies, and when those hopefully begin to grasp the problem and then possibly a solution. There’s a smarter way to perform detection that doesn’t rely on the IDS signature lifecycle; by collecting data on everything, and allowing offline analysis apply genius algorithms to enhance the data, enabling AI to connect the dots for us.