tag:blogger.com,1999:blog-471870110572762709.post6651048799254428804..comments2020-01-08T01:12:19.558-08:00Comments on SmuSec: Gnawing on HTTP 206 Fragmented Payloads with RuminateCharles Smutzhttp://www.blogger.com/profile/05098439824931378207noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-471870110572762709.post-75170304800194667112011-01-24T17:36:23.808-08:002011-01-24T17:36:23.808-08:00Victor,
Yes, I believe the type of issues that yo...Victor,<br /><br />Yes, I believe the type of issues that you mentioned and to which I alluded could be interesting academic problems. Back in the days that most attacks/evasion operated at layers 3 and 4, these sorts of corner cases were all the rage in academia. I don’t understand why the layer 7 and embedded payload object equivalents aren’t being discussed as rigorously today. While I’m starting with the normal case of HTTP 206, I hope Ruminate helps promote research into these types of corner cases at layer 7 and above.<br /><br />That being said, in a practice, I think it’s critical to be able to adequately handle the normal cases before worrying about the exceptional cases. For example, it’s often more important in the real world to be able to routinely operate on adequate amounts of reassembled/decoded data (classic flow depth problem) than to be able to handle extremely rare cases (such as TCP defrag corner cases). While I’m nowhere near there yet with HTTP 206 defrag in Ruminate, one pragmatic approach that likely won’t earn academic accolades because it isn’t sexy but just might help in the real world is to handle ambiguous cases by simply trying multiple possible reconstructions. Ex. if you are reassembling fragments, and the two fragments overlap, and the overlapping fragments actually differ, then try both possible reconstructions.<br /><br />Circling back to my comments to Richard above, while some other examples could be cited, layer 7 defrag is most critical if you are doing things that require looking at the reassembled payload objects—like Ruminate or VRT’s Razorback, http://labs.snort.org/razorback/, are attempting to do. If you are just going to apply a short signature or string match to a payload, full reassembly really isn’t as important. If you want to undo file format encoding, such as compression in PDF, SWF, or DOCX, then reassembly is critical. If you want to inspect PEs, throwing all or some of them in a sandbox, full reassembly is essential. This sort of analysis is what Ruminate is all about, hence my interest in HTTP 206 defrag-—both the normal and corner cases.Charles Smutzhttps://www.blogger.com/profile/05098439824931378207noreply@blogger.comtag:blogger.com,1999:blog-471870110572762709.post-77714855626171611402011-01-23T23:54:39.471-08:002011-01-23T23:54:39.471-08:00Thanks Charles, for doing and publishing this rese...Thanks Charles, for doing and publishing this research and also for the pcaps. The reassembly process looks fairly straightforward, except for the multiple sources cases.<br /><br />Without having looked at the issue at all yet, I'm wondering about evasion possibilities by exploiting server/client specific handling of corner cases. In IP-defrag and TCP-stream reassembly in Suricata we are spending a lot of effort to do it right based on the target OS (Snort has shown the way here). Would similar issues be possible here? IE handling 206-frag overlaps differently from FF, Webkit from some download manager, etc?Victor Julienhttps://www.blogger.com/profile/13971188851017199245noreply@blogger.comtag:blogger.com,1999:blog-471870110572762709.post-51202293356472481872011-01-21T15:46:28.980-08:002011-01-21T15:46:28.980-08:00Richard,
libHTP, http://sourceforge.net/projects/...Richard,<br /><br />libHTP, http://sourceforge.net/projects/libhtp/, which Suricata uses, appears to be very promising for parsing HTTP transactions and exposing HTTP payload data. In fact, when I get to the point in Ruminate where I worry more about efficiency, robustness, etc than new features and chasing crazy ideas, I’ll seriously consider swapping out the Perl implementation using HTTP::Paser with something using libHTP. Perl/Python binding for libHTP would be a welcome facilitator. However, as far as I understand, reassembling the payload fragments across multiple HTTP transactions is out of scope for libHTP. It should be clear from the architecture of Ruminate that to me it makes a lot of sense to abstract away the inter layer 7 transaction processing (206defrag) from the regular layer 7 transaction processing (http_parser). Traditional NIDS aren’t concerned with comprehensive payload object extraction and reassembly because their detection models don’t need/support it. Ruminate needs to have access to complete payload objects to be able to analyze them. One thing I have give Suricata credit for is pushing the envelope on layer 7 decoding, e.g. gzip content encoding a la libHTP.<br /><br />Vivek,<br /><br />I don’t know why some clients request byte 1 over and over again. I know my desktop does this routinely though when viewing PDFs in firefox. For the moment, I’m focusing my efforts on building the network instrumentation necessary to understand what’s going on.Charles Smutzhttps://www.blogger.com/profile/05098439824931378207noreply@blogger.comtag:blogger.com,1999:blog-471870110572762709.post-33278048574371450532011-01-21T03:36:25.751-08:002011-01-21T03:36:25.751-08:00Thanks, great post. I had no idea 206 was being us...Thanks, great post. I had no idea 206 was being used so much. <br /><br />I looked at the traces and as you pointed out found strange stuff.<br /><br />bytes 1-1 was asked for in almost every chunk - wonder why ?<br /><br />"bytes=1-1,52049-57363,483154-484799,57364-62892,485870-485915,62893-75939"<br /><br />some of the ranges are contiguous so can be collapsed like 52049-57363 , 57364-62892 , 62893-75939 can be requested as 52049-75939 ! Why were they not ? Maybe the browser wanted a multipart/bytes back ?<br /><br />Nice work. I will try out your code over the weekend.Vivek Rajanhttp://www.unleashnetworks.com/blognoreply@blogger.comtag:blogger.com,1999:blog-471870110572762709.post-65731194384542284132011-01-20T18:05:17.330-08:002011-01-20T18:05:17.330-08:00Hey Charles, have you looked at the HTP library fr...Hey Charles, have you looked at the HTP library from Suricata to see how they handle this?<br /><br />http://www.openinfosecfoundation.org/index.php/download-suricataRichard Bejtlichhttps://www.blogger.com/profile/13512184196416665417noreply@blogger.com