Saturday, May 29, 2010

Security Engineering Is Not The Solution to Targeted Attacks

Recent publicity and lessons from the school of hard knocks have significantly increased the visibility of targeted attacks. Many organizations react to targeted attacks by pouring on yet more of the traditional reactive security measures that didn’t work in the first place. Many also institute draconian rules and procedures for their users. While stepping up security infrastructure and user awareness training is often necessary, it can never completely solve the targeted attack problem, at least not without inflicting unacceptably unreasonable and probably impractical restrictions on the organization’s personnel and IT infrastructure.

There’s been a fair amount of buzz about Michal Zalewski’s article entitled Security engineering: broken promises. He does a very good job of summarizing some of the open issues with security engineering. I do think he’s probably a little pessimistic, missing some opportunities to give credit and I think it’s unfair to claim security engineering has failed for not developing a unified model that can ensure security. However, he’s pulled together a lot of different facets of security engineering in a short article. The field of security engineering does need to continue to seek to eliminate vulnerabilities that are being exploited widely, and do it in an efficient manner. Much of his discussion can be generalized beyond software security to general information security.

While Zalewski didn’t address or mention APT, I’ve heard similar (but usually not so complete or well worded) rants about the failings of security best practices in regards to APT. It really pains me hear people trash security engineering, especially in the context of Aurora and similar attacks. I’ve also heard a fair amount “sky is falling” and “security best practices can’t keep you safe from APT”. Blaming security engineering for failing to stop targeted attacks doesn’t make sense when it was never a requirement of most systems. Furthermore, we don’t want security engineering alone to solve this problem anyway.


Engineering is about applying science to provide solutions that meet well defined parameters. These parameters involve all sorts of things like functionality, cost, reliability, etc. Many of these parameters are conflicting, at least apparently. Because we live in a world with scarce resources, engineering seeks to provide the optimum value for all the various parameters.

While security has some unique characteristics, it can be viewed as another parameter of a system. While I agree that if done right, security doesn’t have to be as painful as we often make it, security does often conflict with other parameters such as flexibility, cost, and functionality. As such, a wise engineer only invests as much effort in making a system secure as is required.

It amazes me that physics envy rages so strong in some people’s hearts and minds that they actually lose sight of the imperfections of both theoretical and applied physics. People who expect a comprehensive model to cover all aspects of security, much like the ever nebulous theory of everything, have a long time to wait, very possibly infinitely long, but even that is probably impossible to prove. Furthermore, many of the simple physics models are hard to actually apply in the real world due the many different phenomena that need to be modeled simultaneously. The massless, frictionless, point objects that we hear so much of in physics exercises must only exist in a vacuum, because I’ve never seen them. Practical application of physics isn’t as easy as the models often make it appear. That’s alright though. Using classic Newtonian physics works in a great many situations and helps me understand the world around me. Scientific models always have limitations in their applicability, but that doesn’t negate their value. While it quite often literally requires a group of rocket scientists, very often using a mix of multiple models and simulations, we’ve been able to do a great many things based on physics without having a theory of everything.

Success of Security Engineering

Formal methods are hard. Formally verifying a system is only practical on the most simple of systems. However, it has been done. Ex. flight control systems or highly secure classified systems. We refrain from formally verifying all systems we build not because we can’t or don’t know how, but because it’s just too hard. It requires too much effort and restricts the functionality and flexibility of the resulting systems too much for most people’s tastes. Most people couldn’t, or at least wouldn’t want to do their day to day work on one of these highly verified, and therefore, highly restricted systems.

While not my favorite, using risk mitigation strategies is very effective in certain circumstances. It’s useful where the risk can be quantified and accurately predicted. A prime example of this is the risk associated with identity theft. Many financial institutions effectively apply risk mitigation calculations to determine whether a given measure which will reduce losses due to identity theft will cost more to implement than just accepting the losses. As long as the losses can be accurately calculated a priori, this method is very valid.

Again, I realize that there is plenty room for improvement in the field of security engineering. Regardless, for most threats, security engineering is rather successful overall. We know how to make systems more secure than they are now, but we prefer not to. So if systems aren’t as secure as they should be, it’s usually because we didn’t design them to be secure. Sure, part of engineering is finding solutions that satisfy multiple parameters at the same time. These advances will continue to make security more compatible with ease of use and flexibility. Improved standards will continue to raise the minimum bar of security, while minimizing the additional cost of doing so. However, I believe most systems are secure enough, or at least as secure as we wanted them to be.

It should be noted that the adequacy of the security provided by most systems is not provided solely by the system itself but is supported by external factors such as legal protections. For example, the physical security of most houses is only good enough to make it difficult, well maybe even only inconvenient, for would-be burglars. The vast majority of the deterrence comes in fear of getting caught. Furthermore, insurance provides a very cost effective means of protecting your investments despite the remote risk of burglary.

Security Engineering Can’t Solve Targeted Attacks

The biggest problem with targeted attacks isn’t that security engineering couldn’t provide effective solutions. Our current systems aren’t secure enough to protect us from targeted attacks because we haven’t asked them to be that secure. Furthermore, I don’t think we want them to be that secure. Even if it was possible to make a machine that was 100% secure, I doubt it could ever be used for much of consequence while maintaining that level of security due to weaknesses in the environment, people, and processes.

Let’s return to the example of the residential physical security. Imagine if you took away the deterrence offered by law enforcement. It’s hard to imagine, but let’s say would-be attackers had basically no external deterrence and the only thing between them and your possessions in your house was you and your house. You’d have to go to some very extreme measures to keep your house secure. Simple locks and even an alarm system wouldn’t cut it. Basically in absence of any other deterrent, to defeat a rational burglar the defenses on your house would have to cost the attacker more to circumvent than the value that he could gain from sacking the house. This is a tough asymmetric situation where your defenses have to be perfect and the persistent burglar only has to find one weakness or one weak moment. He can try over and over again, as failed attempts don’t cost him much. It doesn’t take much imagination to see how living in a house like this wouldn’t be much fun.

Ok, now pretend you have something valuable to a small set of burglars. Let’s say you have something like a highly coveted recipe for cinnamon rolls. Let’s say a small set of burglars really want to make their own sweat buns instead of buying yours, and possibly sell them to your customers. The problem with this is that you can’t take out an insurance policy on your roll recipe very easily. How could you quantify the cost of exposure? How could you prove the secret was really lost if you suspect it was? How many times would insurance compensate you—only on the first loss or on all subsequent losses? Insurance just doesn’t work in this case. Insurance policies work great for easily replaceable items like televisions, cars, etc, but they just don’t work well for things like trade secrets.

Targeted attacks are much like the scenario laid out above. Sadly, there is little to no deterrence. Usually the information targeted is highly valuable, but not easily quantifiable. Lastly, while it technically would be possible to engineer defenses that would be effective, very few people really want to live the resulting vault in fort knox, let alone pay for the construction.

Alternatives to Security Engineering

So if it’s not feasible to pursue a pure engineering solution to defend against targeted attacks, what is to be done? First of all, a lot of other non-technical solutions should be pursued. I’ll refrain from discussing legal, political, diplomatic, military, etc. solutions because most of us only have a minor influence on these domains and my experience is pretty thin in these areas. However, I do think it’s clear that in many cases, non-technical solutions would be the most effective solutions to the problem. It should also be clear by the empty public statements made by many leaders and decision makers in this realm that non-technical solutions on an international scale are going to take a while, if they ever come.

Security engineering is part of the solution. In many cases, we do need to engineer more secure solutions. We need to make security cheaper and easier. However, even with the best minds on the problem, this will only help so much. While our users need to improve their resilience to social engineering, in many cases, targeted attacks are so done so well, that I couldn’t fault a user for being duped.

Previously I discussed how keeping targeted attacks secret kills R&D. In that case, I wasn’t speaking of security engineering as applied to all IT systems, but was referring to the small subset of IT infrastructure dedicated primarily to security (ex. IDS, SIMS, etc). In that post I echoed the claim of others that threat focused response or security intelligence is one of the most effective approaches to responding to targeted attacks. Surely, this incident response approach will require some engineering of tools to support this approach, in addition to the general security engineering that will come out of proper incident response. Correctly prioritizing your engineering resources to deal with targeted attacks will often result in allocation of resources to tools that support an intelligence driven response.

I often imagine that a well functioning threat focused incident response team facing targeted attacks is much like the wolf and sheepdog cartoon. While the sheep aren’t particularly well protected, and really can’t be if they are to graze successfully, the sheepdog watches for the ever present wolf. The sheepdog keeps track of the wolf and counters his efforts directly, instead of trying to remedy every possible vulnerability. I recognize that as the sheepdog is invariably successful, this comparison is a little more ideal than reality will probably ever be. However, focusing a concentrated intelligence effort on a relatively small group of highly sophisticated attackers makes a lot of sense as long as the group of advanced attackers is small and the effort to defend against them is much higher than against other vanilla threats.

I’ve done both security engineering and engineering for security intelligence. Both have their place. Both have their success stories and both have numerous opportunities for improvement. However, blaming security engineering for the impact of targeted attacks is a herring as red as they come. A world where security engineering actually tried to solve highly targeted and determined attackers would not be a fun place in which to live. In absence of other solutions, an intelligence driven incident response model is your best bet. If I haven’t been able to convince you of this, then all I have to say is that Chewbacca lives on Endor and that just doesn’t make sense… Blaming security engineering for target attacks: that does not make sense.

Thursday, May 20, 2010

Panel and Preso at SANS 4n6 and IR Summit

I’m honored to have been asked to be part of the SANS 2010 What Works in Forensics and Incident Response Summit. I’ll be part of a panel discussion on network forensics and will be presenting on the topic of “Network Payload Analysis for Advanced Persistent Threats”.

The agenda includes some presentations and panel discussions by a large number of the thought leaders in the field of incident response and digital forensics. This is an excellent opportunity to hear from those with experience responding to highly targeted attacks. I'm really looking forward to participating.

Saturday, May 1, 2010

Vortex Howto Series: Parallel NRT IDS

To fulfill all the major tasks I promised when I began this series of vortex howto articles, this installment will focus on scaling up the network analysis done with vortex in a way that leverages the highly parallel nature of modern servers. While the techniques shared in this post are applicable to all the uses of vortex demonstrated so far, it’s especially applicable to near-real time network analysis, a major goal of which is to support detections not possible with conventional IDS architectures, including high latency and/or highly computationally expensive analysis. If you are new to NRT IDS and its goals, I recommend reading about snort-nrt especially this blog post which explains why some very useful detection just can’t be done in traditional IDS architectures. As we’re going to build upon the work done in installment 3, I highly recommend reading it if you haven’t.

Many of us learned about multiprocessing and its advantages in college. In cases where you have high latency analysis, which often is caused by IO such as querying a DB, multiprocessing allows you to efficiently keep your processor(s) busy while accomplishing many high latency tasks in parallel. Traditionally, if you want to do computationally expensive tasks that can’t be done on a single processor, you have two options: use a faster processor or use multiple processors in parallel. Well, if you haven’t noticed, processor speeds haven’t increased for quite some time, but the number of processors in computers has increased fairly steadily. Therefore, as you scale up computationally expensive work on commodity hardware, your only serious choice is to parallelize. While the hard real time constraints of IPS make high latency analysis impossible and computationally expensive analysis difficult, if you are satisfied with near real-time, it’s a lot easier to efficiently leverage parallel processing.

Note that throughout this article, I’m not going to make a clear distinction between multi-threading, multi-processing, and multi-system processing. While text books make a stark differentiation, modern hardware and software somewhat blur the differences. For the purposes of this article, the distinction isn’t really important anyway.

Vortex is a platform for network analysis, but it doesn’t care if the analyzer you use is single or multi-threaded. Vortex works well either way. However, xpipes, which is distributed with vortex does make it easy to turn a single threaded analyzer into a highly parallel analyzer even if, or especially in the cases where, the analyzer is written in a language that doesn’t support threading.

Xpipes borrows much of its philosophy (and name) from xargs. Like xargs it reads a list of data items (very often filenames) from STDIN and is usually used in conjunction with a pipe, taking input from another program. While xargs takes inputs and plops them in as arguments to another program, xpipes takes inputs and divides them between multiple pipes feeding other programs. If you are in a situation where xargs works for you, then by all means, use it. Xpipes was written to be able to fit right between vortex and a vortex analyzer without modifying either, thereby maintaining the vortex interface. Xpipes spawns multiple independent instances of the analyzer program and divides traffic between the analyzers, feeding each stream to the next available analyzer. In general, xpipes is pretty efficient.

Slightly simplifying our ssdeep-n network NRT IDS from our last installment we get:

vortex -r ctf_dc17.pcap -e -t /dev/shm/ssdeep-n \
-K 600 | ./ | logger -t ssdeep-n

To convert this to a multhreaded NRT IDS, we would do the following:

vortex -r ctf_dc17.pcap -e -t /dev/shm/ssdeep-n \
-K 600 | xpipes -P 12 -c './ | logger -t ssdeep-n'

Now instead of a single instance of the analyzer we will have 12. Our system has 16 processors so this doesn’t fully load the system, but now a larger fraction of the total computing resources are used. Taking a look at this in top is as follows:

top - 12:56:25 up 102 days, 19:35, 4 users, load average: 17.30, 16.94, 9.
Tasks: 295 total, 7 running, 288 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.5%us, 54.7%sy, 0.1%ni, 28.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%
Mem: 74175036k total, 73891608k used, 283428k free, 338324k buffers
Swap: 76218360k total, 155056k used, 76063304k free, 72417572k cached

10345 66128 3492 R 22.6 0.0
10346 66128 3452 S 22.3 0.0
10322 66128 3456 R 21.9 0.0
10336 66120 3440 R 21.9 0.0
10337 66124 3464 R 21.9 0.0
10343 66128 3488 R 21.9 0.0
10330 66128 3464 R 21.6 0.0
10342 66128 3444 S 21.6 0.0
10351 66120 3476 S 21.6 0.0
10326 66132 3476 S 20.9 0.0
10340 66124 3448 S 20.9 0.0
10329 66120 3452 S 19.9 0.0
10302 350m 297m S 11.3 0.4 vortex
5 0 0 S 0.3 0.0 migration/1
32 0 0 S 0.3 0.0 migration/10

Beautiful, isn’t it?

If run to completion, the multithreaded version finishes in minutes while the single threaded version took hours.

As should be clear from the above, the -P option specifies the number of children processes to spawn. Typical values of this range from 2 to a few less than the number of processors in the system for highly computationally expensive analyzers. For high latency analyzers you can use quite a few more but there is an arbitrary limit of 1000.

One of the coolest features of xpipes is that it provides a unique identifier for each child process in the form of an environment variable. For each child process it spawns, xpipes sets the environment variable XPIPES_INDEX to an incrementing integer starting at zero. Furthermore, since the command specified is interpreted by shell, XPIPES_INDEX can be used in the command. Imagine that instead of using logger to write a log, we want to write directly to file. If you try something like:

$ vortex | xpipes -P 8 -c "analyzer > log.txt"

You would find that log file gets clobbered by multiple instances trying to write to the file at the same time. However, you could do the following:

$ vortex | xpipes -P 8 -c "analyzer > log_$XPIPES_INDEX.txt"

You’d end up with 8 log files, log_0.txt through log_7.txt which you could cat together if wanted. Similarly, if you want to lock each analyzer to a separate core, say 2-10, you could do something like the following:

$ vortex | xpipes -P 8 -c "taskset -c $[ $XPIPES_INDEX + 2 ] analyzer"

I think you get the idea. Just having a predictable identifier available to both the interpreter shell and the program opens a lot of doors.

Note that if you want to specify the command on the command line you can do so with the -c option. This can admittedly get a little tricky at times because of multiple layers of quoting etc. Alternatively, xpipes can read the command to execute from a file. For example:

$ echo 'analzyer "crazy quoted options"' > analyzer.cmd
$ vortex | xpipes -P 8 –f analyzer.cmd

That’s the basics of parallel processing for NRT IDS the vortex way. So while vortex takes care of all the real time constraints and heavy lifting of network stream reassembly, xpipes takes care of multithreading so all your analyzer has to do is analysis. While vortex’s primary goal has never been absolute performance, I have seen vortex used to perform both computationally expensive and relatively high latency analysis that would break a conventional IDS.

This largely fulfills the obligation I took on when I started this series of vortex howto articles. I hope this has been helpful to the community. I hope that someone who has read the series would be able to use vortex without too much trouble if a situation ever arose where it was the right tool for the job.

If there are other topics you would like discussed/explained, feel free to suggest a topic. For example, I’ve considered an article on tuning linux and vortex for lossless packet capture, but I think the README and error messages cover this pretty well. I’ve also considered discussing the details of the performance relevant parameters in vortex and xargs, but most of these work very well for most situations without any changes.

Again, I hope this series has will help people derive some benefit from vortex. I also want to reiterate my acknowledgments to Lockheed Martin for sharing vortex with the community as open source.