What Laika BOSS means to NIDS

I’ve long advocated for analysis of files transferred through the network. I’d like to explain how I feel Laika BOSS relates to NIDS. Just as we need NIDS to decode network protocols, apply signatures, and collect metadata on network traffic, so too we need a file-centric analysis system. Laika BOSS is this system, or at least an instance of this class of system. It performs recursive file format decoding, signature matching appropriate for files (yara), and metadata extraction. Laika BOSS is the widget to which NIDS should offload analysis of payload objects. Laika BOSS is the polished implementation of the file analysis portion of the vision that Ruminate IDS advanced.

As explained in my plea for NIDS to adopt an interface for external analysis of payload objects, the motivation for this separation comes down to specialization/Unix philosophy and general need for file analysis capabilities. There is a market for really good NIDS. I see the need for really good file-centric IDS (FIDS). They should be plumbed together, but the file analysis systems need to support more than just NIDS sources. If Laika BOSS’s relatively simple zeroMQ message format becomes the de-facto interface NIDS use to provide files for external analysis, I’ll be flattered.

Some NIDS will be tempted to try to build Laika BOSS style analysis into their core engine. While some steps towards greater payload visibility will always be welcome, going too far down this path will be a distraction. This will ultimately be similar to NIDS that have tried to bring in external log analysis/SIM functionality. A NIDS specific file analysis framework will always lag in content compared to a general purpose file IDS. More importantly though, there is still much to be done in the realm of network protocol analysis. The security community is much better served with NIDS engines that focus on support for protocols such as RDP, SMB, and HTTP2 over file formats such as JAR, SWF, and PDF. If a NIDS project wants to implement their own FIDS, I'd recommend keeping them separate, similar to Snort/Razorback.

In addition to opportunity cost, today’s NIDS just can't support the depth of analysis on files that some users want. This isn’t to say that current NIDS aren’t making an effort to provide these features. Both Bro and Suricata are both making strides towards this goal. I think the lua based rules that Emerging Threats has developed for Suricata, including functionality such as decompressing a JAR and analyzing constituent files, has given the community a taste of what must be.

However, today’s NIDS, at least as currently architected, can’t support the depth of analysis that is needed for today’s complex file formats. It’s not just the complexity of desired workloads that is a problem, it’s also inability to scale to ever increasingly parallel and economic platforms. It’s easy to pay for servers totaling dozens of CPU cores, but current NIDS architectures can’t scale well beyond about 16 threads, let alone operate on cloud resources. Suricata is the only mainstream NIDS that I know of that supports dynamic load balancing, but in practice this doesn’t scale well, with most high speed implementations using static load balancing for relatively independent worker threads. (And no, mechanisms such as the AF_PACKET rollover option do not help if they balance packets from the same stream to different workers, breaking stream reassembly).

The Laika BOSS white paper reports statistics from a real network using Laika BOSS configured with many modules including CPU intensive detection mechanisms for some file formats. Most object scan times are under 50ms but the average time spent processing a portable executable is 83s. My previous studies on this topic have convinced me that this level of variance in throughput will never be feasible in the current NIDS paradigm because as one scales up the number of NIDS threads, the limit on scalability remains the worst case load distribution. However, many want to be able to do this depth of analysis and are more than happy to pay for commodity CPUs to do it. Most users are willing to put depth of analysis in front of IPS capabilities and the associated real time constraints. If NIDS provide files to Laika BOSS asynchronously, the additional layer of dynamic load balancing makes this level of analysis feasible.

We’re very close to being able to provide payload files to external systems in current NIDS. Both Suricata and Bro can write files out to a filesystem. Furthermore, lua scripts in Suricata have access to reassembled file payloads. Bro has a new file analysis framework. There are a few limitations with all of these approaches. In many cases, providing network layer context for a file is not straightforward. In other cases, access to the file is provided in chunks instead of in a single buffer. However, I hope that in the near future, we’ll see efficient integration of Laika BOSS with NIDS, allowing the offloading of analysis of all objects passing through the network. In the meantime, there are organizations using Vortex/Ruminate derivatives to provide network payloads in a scalable, if not ugly way.

I heartily endorse Laika BOSS. I’m proud that Lockheed Martin has made this tool open source. I feel it has a good shot at becoming the file analysis engine that the community needs. I believe that the ease of creating modules and the general interface suitable for many use cases will make it popular among users. I hope that NIDS engines are soon able to efficiently integrate with Laika BOSS to offload file-centric analysis.