Implementing Cisco IDS and IPS

IDS and IPS Functions and Operations

There are two types of intrusion systems that are typically deployed in networks today: intrusion detection systems (IDS) and intrusion prevention systems (IPS). Each type of system consists of either hardware or software that first detects network anomalies. How a particular system responds to anomalies determines its true role. An IDS device does not sit in the path of active network traffic. Instead, traffic is copied out to the IDS device for inspection. If the IDS device determines that a series of packets does not have the best of intentions, it sends an alert to a management station for further action. The IDS can also actively configure network devices (such as routers and firewalls) to block or quarantine the mischievous packet flows. Remember that the IDS itself cannot block any packets, because it does not sit in the data path. So, if the IDS discovers an issue, the first few packets are already in the network. At best, the IDS can block further packets from getting into the network and reaching their target. An IPS, on the other hand, sits directly in the path of network traffic. All packets must travel through the IPS device as they cross the segment that the IPS lives on. When the IPS detects some sort of anomaly, it can both alert a management station and block the questionable packets. Because all packets are flowing through the IPS, you do not need to configure other network devices to block the bad packets. An IPS is useful for detecting viruses, worms, malicious applications, and vulnerability exploits, none of which should be permitted into the network at all. Because the IPS can immediately block packets deemed bad, it can shield the network from such exploits. As a quick review: ■ Virus—A virus is one type of malicious code that tries to propagate itself across a network. It is normally attached to other programs and executes a particular unwanted function on a user workstation when that program executes. A virus propagates itself by infecting other programs on the same computer. It can do serious damage, such as erasing files or erasing an entire disk. It can also be a simple annoyance, such as popping up a message. A virus cannot spread to a new computer without human assistance, such as opening an infected file in an e-mail attachment or through file sharing. 150x01x.book Page 567 Monday, June 18, 2007 8:52 AM 568 Chapter 23: Implementing Cisco IDS and IPS ■ Worm—A worm is another type of malicious code that executes arbitrary code and installs copies of itself in the memory of the infected computer. It can then spread to and infect other hosts from the infected computer. Like a virus, a worm is also a program that propagates itself. Unlike a virus, a worm can spread itself automatically over the network from one computer to the next. Worms simply take advantage of automatic file sending and receiving features found on many computers, or it uses its own e-mail code to send infected files to mail recipients. ■ Trojan horse—A general term that refers to a program that appears desirable but actually contains something harmful; for example, a downloaded game that contains code to erase files. The malicious contents could also hold a virus or a worm. ■ Vulnerability exploit—An attack that specifically targets a known device vulnerability. The IDS and IPS can be used together to provide tighter network security. An IPS blocks traffic that it considers unsafe. If the traffic in question is legitimate, however, then the IPS could be doing more harm than good. Such traffic is commonly called “gray area” traffic. Instead of forcing the IPS to make a yes or no decision on it, gray area traffic can be sent off to the IDS for further inspection and analysis.

Categories of IDS and IPS

There are two ways to categorize IDS and IPS systems. The first category is the scope of the system. The two IDS and IPS scopes are ■ Network ■ Host An IDS or IPS can sit in the network (as a hardware appliance or a software module in an existing network device) and thus provide protection to the entire network or segments of the network. Such systems are called either network intrusion detection systems (NIDS) or network intrusion protection systems (NIPS). One appliance can monitor multiple hosts, and additional hosts can be added without increasing the number of NIDS or NIPS devices. A network-based system can monitor and detect buffer overflows, network reconnaissance, and denial of service (DoS) attacks. The ability to see attacks against the network offers the opportunity to determine the extent of the attack. As a reminder, consider the following attacks: ■ Denial of Service (DoS)—Where one device overloads the network access to or CPU utilization of a target system ■ Distributed Denial of Service (DDoS)—Where multiple devices overload the network access to or CPU utilization of a target system 150x01x.book Page 568 Monday, June 18, 2007 8:52 AM Categories of IDS and IPS 569 ■ Recon—An attempt to gather important network information from a device, such as accounts, passwords, host names, other IP addresses in use, and operating systems (to name a few) Network-based systems can only detect and prevent intrusive activity. If a single packet does sneak through, the network-based system cannot assess the success or failure of the attack. Also, network-based systems cannot inspect encrypted traffic. As networks continue to grow, the need for additional network sensors could become cost prohibitive. The second scope category of an IDS or IPS is at the host level. Such systems are typically software modules that reside on a workstation or server and provide anomaly detection and prevention services for that single device. These systems are called either host intrusion detection systems (HIDS) or host intrusion prevention systems (HIPS). When encrypted traffic flows across a network, only the HIPS or HIDS can see the plaintext contents of the packets. Virtually all implementations of host-based intrusion systems are HIPS versus HIDS. Cisco Security Agent is an example of a HIPS. Another way to categorize IDS and IPS systems is the approach they take to identify malicious traffic. The three different identity approach mechanisms are as follows: ■ Signature-based—Signature-based systems match for a specific byte pattern or content in a packet. Such pattern matching is typically combined with particular IP address, protocol, and/ or port combinations to perform very precise matches. Attacks, such as Trojan horses, tend to change port numbers regularly, which can invalidate a pure signature-based system. Also, because signature patterns are preprogrammed into an IDS and IPS device, day-zero attacks (attacks that exploit a vulnerability in a new system patch) are difficult to defend against. ■ Policy-based—Policy-based systems use algorithms to examine strings of packets to determine patterns and behavior. For example, such an approach might detect a ping sweep, whereas a signature-based system would see only individual ping packets. Additional restrictions, such as IP addresses, protocols, or ports, can be applied (as can be done with signature-based systems). Some policies, such as restricting the ability to browse certain websites, typically involve communication with some type of blacklist database to ensure upto-date information. ■ Anomaly-based—Anomaly-based systems look for behavior that deviates from the “norm.” This implies that some definition of “normal” is dynamically learned by (statistical) or preprogrammed into (nonstatistical) the system before it can detect anything abnormal. Such systems tend to work well in small networks, where normal behavior can be easily defined. However, in larger networks, the definition of normal can easily be too expansive and complex to adequately define. 150x01x.book Page 569 Monday, June 18, 2007 8:52 AM 570 Chapter 23: Implementing Cisco IDS and IPS A honeypot is a common term heard in the threat prevention and protection environment. A honeypot is simply a sacrificial network device. Such a machine is left on the network to attract attackers, and thus pull the malicious packets away from important network resources. Packet flows captured on the honeypot device can be used to analyze the attack and construct an appropriate defense. Because the desire is to collect the devious packets, honeypots tend to be considered IDS instead of IPS. It is important to remember that any device that is used as a honeypot is not returned to the general network population, because the honeypot device is quite likely infected with all sorts of interesting malware.

IDS and IPS Signatures

For the purpose of IDS and IPS, a signature is a pattern of data or traffic that should cause a reaction when it passes through the IDS or IPS. As described earlier in this chapter, this action can be either to send some type of alert or to actively block the offending traffic. An IDS and IPS uses microengines to match signatures against packets and packet flows. In other words, each signature constitutes its own microengine. There are four categories of IDS and IPS signatures: ■ Exploit—An exploit signature typically identifies malicious traffic by matching a traffic pattern. Usually, each exploit has a unique signature (either packet flows or packet contents). Thus, each attack requires a signature for detection. If the exploit is modified in any way, a new signature is needed to be able to match the modified attack. ■ Connection—A connection signature is aware of valid network connections and protocols. The behavior of accepted connections and protocols is known in advance, and any actions that occur beyond the normal circumstances are considered suspect. Note that normal is also a subjective definition of acceptable network traffic and behavior. ■ String—String signatures typically use regular expressions to match patterns. This is similar to how exploit signatures work, except a regular expression can be used to match many conditions, whereas an exploit signature usually matches a single exploit. ■ DoS—DoS signatures examine behavior typical of a DoS attack. Because there are many forms and flavors of DoS attacks, there are a variety of DoS signatures used. As with exploit signatures, a behavioral change in a DoS attack would require an update to the DoS signature engine. A Cisco IOS router can act as an inline NIPS device. By default, there are 100 signatures embedded in Cisco IOS Software (132 total with all of the subsignatures). Additional signature definition files (SDFs) can be downloaded from Cisco.com. Individual signatures within an SDF can be enabled and disabled