One of the best web interfaces for visualizing Subversion repositories (as well as providing
integrated project management and ticketing functionality) is the
Trac Project, and all Cipherdyne software projects
use it. With Trac's success, it also becomes
a target for those that would try to subvert it for purposes such as creating spam. Because
I deploy Trac with the Roadmap, Tickets, and Wiki features
disabled,
my deployment essentially does not accept user-generated content (like comments in tickets
for example) for display. This minimizes my exposure to Trac spam, which has become a
problem significant enough for various spam-fighting strategies to be developed such as
configuring mod-security and
plugin from the Trac project itself.
Even though my Trac deployment does not display user-generated content, there are still
places where Trac accepts query strings from users, and spammers seem to try and use these
fields for their own ends. Let's see if we can find a few examples. Trac web
logs are informative, but sifting through huge logfiles can be tedious. Fortunately,
simply sorting the logfiles by line length (and therefore by web request length) allows many
suspicious web requests to bubble up to the top. Below is a simple perl script
line_len.pl that sorts
any ascii text file by line length, and precedes each printed line with the line length
followed by the number of lines equal to that length. That is, not all lines are printed
since the script is designed to handle large files - we want the unusually long lines to
be printed but the many shorter lines (which represent the vast majority of legitimate
web requests) to be summarized. This is an important feature considering that at this
point there is over 2.5GB of log data specifically from my Trac server.
$ cat line_len.pl
#!/usr/bin/perl -w
#
# prints out a file sorted by longest lines
#
# $Id: line_len.pl 1739 2008-07-05 13:44:31Z mbr $
#
use strict;
my %url = ();
my %len_stats = ();
my $mlen = 0;
my $mnum = 0;
open F, "< $ARGV[0]" or die $!;
while (<F>) {
my $len = length $_;
$url{$len} = $_;
$len_stats{$len}++;
$mlen = $len if $mlen < $len;
$mnum = $len_stats{$len}
if $mnum < $len_stats{$len};
}
close F;
$mlen = length $mlen;
$mnum = length $mnum;
for my $len (sort {$b <=> $a} keys %url) {
printf "[len: %${mlen}d, tot: %${mnum}d] %s",
$len, $len_stats{$len}, $url{$len};
}
exit 0;
To illustrate how it works, below is the output of the line_len.pl script used against
itself. Note that at the top of the output the more interesting code appears whereas
the most uninteresting code (such as blank lines and lines that contain closing "}"
characters) are summarized away at the bottom:
$ ./line_len.pl line_len.pl
[len: 51, tot: 1] # $Id: line_len.pl 1739 2008-07-05 13:44:31Z mbr $
[len: 50, tot: 1] printf "[len: %${mlen}d, tot: %${mnum}d] %s",
[len: 48, tot: 1] $len, $len_stats{$len}, $url{$len};
[len: 44, tot: 1] # prints out a file sorted by longest lines
[len: 43, tot: 1] for my $len (sort {$b <=> $a} keys %url) {
[len: 37, tot: 1] if $mnum < $len_stats{$len};
[len: 34, tot: 1] $mlen = $len if $mlen < $len;
[len: 32, tot: 1] open F, "< $ARGV[0]" or die $!;
[len: 29, tot: 1] $mnum = $len_stats{$len}
[len: 25, tot: 1] my $len = length $_;
[len: 24, tot: 1] $len_stats{$len}++;
[len: 22, tot: 2] $mnum = length $mnum;
[len: 21, tot: 1] $url{$len} = $_;
[len: 20, tot: 2] my %len_stats = ();
[len: 19, tot: 1] #!/usr/bin/perl -w
[len: 14, tot: 3] while (<F>) {
[len: 12, tot: 1] use strict;
[len: 9, tot: 1] close F;
[len: 8, tot: 1] exit 0;
[len: 2, tot: 5] }
[len: 1, tot: 6]
Now, let's execute the line_len.pl script against the trac_access_log file and look
at one of the longest web requests. (The line_len.pl script was able to reduce
the 12,000,000 web requests in my Trac logs to a total of 610 interesting lines.) This
particular request is 888 characters long, but there were some other similar suspicious
requests that had over 4,000 characters that are not displayed for brevity:
[len: 888, tot: 1] 195.250.160.37 - - [02/Mar/2008:00:30:17 \ -0500] "GET /trac/fwsnort/anydiff?new_path=%2Ffwsnort%2Ftags%2Ffwsnort\ -1.0.3%2Fsnort_rules%2Fweb-cgi.rules&old_path=%2Ffwsnort%2Ftags%2F\ fwsnort-1.0.3%2Fsnort_rules%2Fweb-cgi.rules&new_rev=http%3A%2F%2Ff\ 1234.info%2Fnew5%2Findex.html%0Ahttp%3A%2F%2Fa1234.info%2Fnew4%2F\ map.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew2%2Findex.html%0Ahttp%3A%2F\ %2Fs1234.info%2Fnew9%2Findex.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew6%2F\ map.html%0A&old_rev=http%3A%2F%2Ff1234.info%2Fnew5%2Findex.html%0Ahttp\ %3A%2F%2Fa1234.info%2Fnew4%2Fmap.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew2\ %2Findex.html%0Ahttp%3A%2F%2Fs1234.info%2Fnew9%2Findex.html%0Ahttp%3A\ %2F%2Ff1234.info%2Fnew6%2Fmap.html%0A HTTP/1.1" 200 3683 "-" \ "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; \ Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR \ 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)"My guess is that the above request is a bot that is trying to do one two things: 1) force Trac to accept the content in the request (which contains a bunch of links to pages like "http://f1234.info/new/index.html" - note that I altered the domain so as to not legitimize the original content) and display it for other Trac users or to search engines, or 2) force Trac itself to generate web requests to the provided links (perhaps as a way to increase hit or referrer counts from domains - like mine - that are not affiliated with the spammer). Either way, the strategy is flawed because the request is against the Trac "anydiff" interface which doesn't accept user content other than svn revision numbers, and (at least in Trac-0.10.4) such requests do not cause Trac to issue any external DNS or web requests - I verified this with tcpdump on my Trac server after generating similar requests against it.
Still, in all of my Trac web logs, the most suspicious web requests are against the "anydiff" interface, and specifically against the "web-cgi.rules" file bundled within the fwsnort project. But, the requests never come from the same IP address, the "anydiff" spam attempts never hit any other link besides the web-cgi.rules page, and they started with regularity in March, 2008. This makes a stronger case for the activity coming from bot that is unable to infer that its activities are not actually working (no surprise there). Finally, I left the original IP address 195.250.160.37 of the web request above intact so that you can look for it in your own web logs. Although 195.250.160.37 is not listed in the Spamhaus DNSBL service, a rudimentary Google search indicates that 195.250.160.37 has been noticed before by other sites as a comment spammer.
Next month in NYC the final Hackers On Planet Earth (HOPE)
conference will take place from July 18th through the 20th. I will be giving a
talk there entitled
"Port Knocking and Single Packet Authorization: Practical Deployments", and here is
the abstract:
Port Knocking and
its big brother, Single Packet Authorization (SPA), can provide a robust additional layer
of protection for services such as SSH, but there are many competing Port Knocking and SPA
implementations. This talk will present practical usages of fwknop in Port Knocking and SPA
modes, and discuss what works and what doesn't from a protocol perspective. Integration
points for both iptables and ipfw firewalls on Linux and FreeBSD systems will be highlighted,
and client-side support on Windows will be demonstrated. Finally, advanced functionality
such as inbound NAT support for authenticated connections, sending SPA packets over the
Tor anonymity network, and covert channel usages will be discussed. With SPA deployed,
anyone scanning for a service with Nmap cannot even tell that it is listening; let alone
target it with an exploit (zero-day or not).
Gootrude-0.2 is available for
download. This release changes the
default search engine from Google to Yahoo because it appears that Yahoo's Terms of Service do not
prohibit automated queries (although I'm not a lawyer). Gootrude retains the ability to query Google
at your own risk if you so desire. As long as you use Gootrude as designed to collect search engine
results only once per day with a limited number of search terms (say, 20 or less), it is unlikely to
provoke a negative reaction from search engines. Over time, support for many additional search engines
will be added to Gootrude so that search results can be trended from all sorts of different data
sources.
UPDATE 06/16/08: 1) After making Slashdot
today and reading the comments, I feel I should clarify a few things regarding this post.
First, it was pointed out (quite rightly) that
Gootrude trends the number of documents in
Google's index that contain each search term, whereas Google Trends tracks the search volume
associated with a search term. These are not the same thing. However, I would wager that
they are related, though certainly not in some nice 1:1 correspondence. That is, if a
relatively unique word - say, "myspace" - is mentioned within 100,000 web sites, then the
number of times this word is used as a search term in Google will be higher than if "myspace"
were mentioned in only 10 web sites just because it is exposed to many more people who would
then be interested in all things "myspace". I would also wager that the reverse is true.
So, if "myspace" is an extremely popular search term, then it will probably also appear in
many more web sites around the Internet. Gootrude attempts to trend the number of hits a
given search term returns in Google, and by the above reasoning, there is a loose correspondence
between this and the number of times the term is searched for in Google. Perhaps this is
not useful, but only time will tell after Gootrude is used by a lot of people. If you will
allow the above interpretation, then the remainder of the post below makes sense.
As you can see, at the beginning of the graph around July 1st Google steadily shows
about 28,000 results for "cipherdyne", but towards the end of July this dips to well below 20,000 only to
rebound in August to about 30,000. Then, beginning around March 1st, 2008 the results shoot
up to over 100,000 briefly and then back to around 70,000 in May. How does one interpret this
data? It seems unlikely that these fluctuations can be entirely explained by "actual"
day-by-day changes in how external sites reference the term "cipherdyne" - there must be some
index updating component that is internal to Google at work here, and we'll see a better example
of this below.
The most obvious feature of the gpgdir graph above is the large spike to around 60,000 results
around May 1st. It turns out that an article
was posted to linux.com on the 24th of April, so given that
"gpgdir" is not a common word, the spike seems nicely correlated with the posting of the article
as it got bounced around the Internet and blogosphere. A more interesting feature perhaps is
the sharp cyclic oscillation between July and December 2007. During this time, search results for
gpgdir bounced from 1,000 to around 10,000 and back again several times, and the transition
each time was fast - making the jump to 10,000 over the course of two days and then stabilizing
for about 10 days or so and then back down to 1,000. It is almost as though Google was trying to
establish the proper order of magnitude for "gpgdir" search results during this time via a sort
of step function.
Again, we see a dramatic spike in search results - from around 5,000 to well over 50,000 and
settling down to about 10,000 around the beginning of June. Although there has been some
activity related to SPA in the Ubuntu forums
and also in the Gentoo forums, if this caused Google to report the search results as over 50,000 why did
this number return so precipitously back to around 10,000? The links have not gone away, but
they were probably mentioned on other referencing sites and then moved to less important pages
over time on those sites. Perhaps Google is trying to find the appropriate steady state for
its search results, and there are many factors that Google takes into account that are not
available to the public.
The following releases of the Cipherdyne security projects are now available:
psad-2.1.3, fwknop-1.9.5,
and gpgdir-1.9.1. The motivation for a group release
stems from the need to make similar changes to the fwknop and gpgdir projects to
update to version 2.11 of the
Class::MethodMaker
perl module from CPAN - a dependency of GnuPG::Interface. This version fixes a
build error under recent versions of perl (such as perl-5.10.0) which are distributed
on systems like Fedora 9.
Also, for both fwknop and gpgdir, thanks to a suggestion made by
Jean-Denis Girard on the fwknop mailing list, the
default locale has been set to "C" via the LC_ALL environmental variable so that
GnuPG output can be properly interpreted even on systems where a different locale is
used. The locale can be manually set or not used at all with two new command line
arguments --locale <str> and --no-locale respectively. On another note,
Kevin Hilton has written
an excellent how-to for fwknop on Ubuntu systems.
After two months of development, the 1.9.4 release of fwknop
is available for download. This release introduces
new functionality that has implications for hardening both fwknop SPA communications
and the follow-on connections that client programs make (such as SSH). Specifically,
the 1.9.4 release adds two port randomization options to 1) send the SPA packet over
a random port between 10,000 and 65,535 (requires updating the default PCAP_FILTER
variable on the fwknopd server side), and 2) select a random port that is used as
the destination port in a NAT operation on the fwknopd server system. These two
options can be used individually or together, and are enabled with two new command
line options, --rand-port and --NAT-rand-port respectively, to the fwknop client
(see examples below).
WordPress has a loyal following in the blogosphere and
is used to power thousands of websites and blogs, and several books have been written about it.
A compelling measure of WordPress popularity is the number of
downloadable themes that can provide an easily deployed
and consistent look and feel to WordPress sites. So, when I was looking to update the presentation
of the cipherdyne.org site, it seemed a natural step to consider a WordPress theme.
The March issue of
Hakin9 Magazine contains some noteworthy articles
and interviews. First is an article written by Ryan Maple entitled "Best Practices for
Secure Shell" which discusses various security measures that an administrator can use
to heighten the security of SSH. These measures include (among others) forcing the usage
of SSH protocol version 2, restricting the address (via the ListenAddress variable) that
SSHD binds to from the default of 0.0.0.0, using tcpwrappers, and configuring SSHD to
listen on a port other than tcp/22. Covering Single
Packet Authorization would have been difficult to include in the same article, but a
reference is made to another online article
Knock, Knock, Knockin' on EnGarde's Door (with FWKNOP) that does discuss protecting
SSH with SPA. Next, is Matt Jonkman's fifth part in his series "Writing IPS Rules".
This article covers the byte_test keyword in the Snort rules language and how to use it
to write signatures against length encoded protocols. In particular, Matt illustrates
using byte_test to look for specific byte values at particular offsets derived from data
within the DHCP protocol. The offsets themselves are determined by the data on the wire
and therefore cannot simply be hard coded within a signature beforehand. For those who are
interested in the latest IDS signatures from Matt, he has updated the online home of the
Bleeding Edge Snort ruleset to Emerging Threats.
Finally, both Marcus Ranum and
Richard Bejtlich were interviewed for the March issue. I would like to see Hakin9
do a more in-depth interview of Bejtlich though since he always has an
insightful perspective on computer security - particularly as demonstrated in his book
The Tao of Network Security Monitoring.
Most port knocking or Single Packet Authorization
implementations offer the ability to passively authenticate clients for access only
to a locally running server (such as SSHD). That is, the daemon that monitors a
firewall log or that sniffs the wire for port knock sequences or SPA
packets can only reconfigure a local firewall to allow the client to access a local
socket. This is usually accomplished by allowing the client to connect to the
server port by putting an ACCEPT rule in the INPUT chain for
iptables firewalls, or adding a pass rule for ipfw firewalls
for the client source IP address. For local servers, this works well enough, but
suppose that you ultimately want to access an SSH daemon that is running on an
internal system? If the SPA software is deployed on a Linux gateway that is
protecting a non-routable internal network and has a routable external IP address,
it is inconvenient to first have to login to the gateway and then login to the
internal system.
The UK's Unix & Open Systems User Group has re-printed
an article I wrote originally for the December, 2007 security issue of
USENIX ;login: Magazine
The article is entitled
"IDS signature matching with iptables, psad, and fwsnort"
and concentrates on how to use the iptables infrastructure in the Linux kernel as a source of
intrusion detection data. That is, iptables offers many features (such as application layer
string matching) that allow a significant fraction of Snort rules to be converted into
iptables rules, and fwsnort automates the conversion process. The
end result is an iptables policy that is looking for evidence of malicious traffic. Also
covered in the article is the concept of log analysis with an emphasis on passive OS
fingerprinting. The completeness of the iptables logging format - which even includes
the options portion of the TCP header when the --log-tcp-options argument is given on the
iptables command line when building a LOG rule - allows psad to run
the same algorithm that p0f uses to
passively fingerprint remote operating systems.
At the SOURCE Boston conference in Boston
last week I gave talk entitled "Advanced Linux Firewalls"
(slides). The conference
attendance was good considering that this is the first year the conference was
offered, and I look forward to next year.
I managed to see a few talks, and two that stood out from the crowd were
Roger Dingledine's talk "How To Make
Tor Play Well With The Rest Of The Internet",
and Andrew Jaquith's talk
"Not Dead But Twitching: Anti-Virus Succumbs to the
Scourge of Modern Malware". Roger highlighted several technology research
and development areas for the Tor project,
including the ability to use UDP instead of TCP for Tor virtual circuits.
This is of particular interest to me, since it would mean that
SPA packets could be routed over the Tor network without having to resort to
the establishment of full TCP connections (which breaks the "single packet" part
of "SPA"). Andrew gave some interesting perspectives on malware trends, including
the fact that malware over time is becoming more targeted while at the same time
exhibiting high variability. The end result is that malware authors are able to
attack the weakest link in the creation of signatures for malware detection - the
people that reverse engineer malware. Because human resources are scarce and slow
when it comes to reverse engineering (there is no fully automated mechanism for
this yet), malware authors are able to essentially perpetrate a DoS against
vendors that offer malware detection.
The SANS Internet Storm Center has an interesting
diary entry that discusses a set
of distributed brute force attempts against SSHD originating from a specific Class C subnet
58.147.10.0/24. According to the diary entry, a random selection of IP addresses
within this subnet (which is owned by an organization in Thailand) are probing for
listening SSH daemons on the open Internet, but each IP address is limiting the number of
probes to two. Presumably this is to fly beneath the radar of any threshold-based
detection mechanisms that might automatically block the attempts to communicate with SSHD.
Over the years the editor I have become the most familiar with is
vim. It provides features that allow the software
development process to go smoothly, such as split screens, syntax highlighting,
integration with cscope tags, function
folding, and more. By applying configuration
directives to your ~/.vimrc file, you can instruct vim to perform some
nice preprocessing functions against files. This blog post illustrates a
~/.vimrc tweak (originally from Wouter Hanegraaff) that allows vim to leverage
GnuPG to decrypt a previously encrypted file,
allow edits to be made, and then re-encrypted before it is written back to disk.
Here is the section of the .vimrc file to allow such transparent encryption and
editing: