Michael Rash, Security Researcher

fwknop    [Summary View]

« Previous

Software Release - fwknop-2.0.1

fwknop-2.0.1 released The fwknop-2.0.1 release is fwknop is available for download. This is mainly a bug fix release before the new HMAC-SHA256 work is ready for general consumption. A couple of the fixes are important for PF users on OpenBSD and for people who use the same encryption key within multiple access stanzas.

Here is the complete fwknop-2.0.1 ChangeLog:

  • [server] Bug fix where the same encryption key used for two stanzas in the access.conf file would result in access requests that matched the second stanza to always be treated as a replay attack. This has been fixed for the fwknop-2.0.1 release, and was reported by Andy Rowland. Now the fwknopd server computes the SHA256 digest of raw incoming payload data before decryption, and compares this against all previous hashes. Previous to this commit, fwknopd would add a new hash to the replay digest list right after the first access.conf stanza match, so when SPA packet data matched the second access.conf stanza a matching replay digest would already be there.
  • [server] Updated PCAP_LOOP_SLEEP default to 1/10th of a second (in microseconds). This was supposed to be the default anyway, but C Anthony Risinger reported a bug where fwknopd was consuming more resources than necessary, and the cause was PCAP_LOOP_SLEEP set by default to 1/100th of a second - this has been fixed.
  • [libfko] Added SPA message validation calls to fko decoding routines to help ensure that SPA messages conform to expected values.
  • Bug fix for PF firewalls: updated the PF anchor check to not rely on listing the PF policy - fwknopd now uses 'pfctl -s Anchor' instead.
  • [test suite] Added parsing of valgrind output to produce a listing of functions that have been flagged - this assists in the development process to ensure that fwknop is not leaking memory.
  • [test suite] Bug fix on Mac OS X systems to account for libfko.dylib path instead of This fixes the existence check for libfko.
  • [test suite] Added tests for --nat-local mode.
  • [client] Fixed several minor memory leaks caught by valgrind.
  • [libfko] Minor gcc warning fix: fko_decode.c:43:17: warning: variable ‘edata_size’ set but not used [-Wunused-but-set-variable].
  • Updated fwknopd init script for Debian systems (contributed by Franck Joncourt).
The complete fwknop-2.0.1 ChangeLog can be found here via the fwknop gitweb interface.

On SPA Cross-Packet Ciphertext Entropy

fwknop SPA packet entropy With fwknop now re-written in C for the 2.0 release, I thought it would be a good idea to take a look at how close encrypted SPA packet data comes to having high levels of entropy - as understood to be a measure of randomness - from one packet to the next. If fwknop is properly using encryption, and the ciphers themselves are also well-implemented (fwknop can use either Rijndael or GPG), then we would expect there to be no obvious relationship between SPA packets even for repeated access requests to the same service. If there are any such relationships in the encrypted data across multiple SPA packets, then an adversary might be able to infer things about the underlying plaintext - precisely what strong encryption is supposed to make difficult. This blog post covers SPA packet entropy for AES (Rijndael) CBC and ECB encryption modes, and leaves GPG to another post.

Although this post has some similarities with an older blog entry "Visualizing SPA Packet Randomness", a more rigorous and automated way of measuring cross-packet SPA entropy will be presented. In addition, we'll take a look at what happens when (normally) random salt values for AES encrypted SPA packets are artificially forced to be constant. This helps to highlight some real differences in AES electronic codebook (ECB) and cipher block chaining (CBC) encryption modes.

First, the next release of fwknop will most likely offer the ability to select different AES encryption modes (such as cipher feedback (CFB) mode and output feedback (OFB) mode), and a dedicated "crypto_update" branch has been created for this work. The default AES encryption mode used by fwknop is cipher block chaining (CBC) mode as defined here. Within the crypto_update branch there is a new script "" that is designed to execute the fwknop client multiple times, collect the encrypted SPA packet data, use the ent program to measure the entropy in slices for each byte position across the SPA data set, and then plot the results with gnuplot. What does this accomplish? It allows us to easily see for any given byte position within a collection of SPA packets whether there is a relation from one to the next. If there is such a relation, then the cipher used to encrypt the data was not very good at achieving high levels of entropy in the ciphertext across multiple packets.

As a motivating example from Wikipedia, AES in ECB mode encrypts identical plaintext blocks into identical ciphertext blocks, and this results in patterns in plaintext data being preserved to some extent in the ciphertext. So, an adversary can make good guesses about the underlying plaintext just by looking at the ciphertext! Wikipedia does a nice job of illustrating this with the following two images of the Linux kernel mascot "Tux" - before and after AES encryption in ECB mode:

plaintext Tux       AES ECB encryption ->                     AES ECB encrypted Tux
Encryption Fail.

Now, let's take a look at SPA packet entropy with the script. For reference, fwknop builds SPA packets according to the following data format before encryption:

[random data: 16 bytes]:[username]:[timestamp]:[version]:[message type]:[access request]:[digest]

So, if a user wants repeated access to the same service protected behind fwknopd on some system, then several fields above will be identical across the corresponding SPA packets before they are encrypted. The username, version, message type, and access request fields will likely be the same. If fwknop has made proper use of encryption, then the fact that these fields are the same across multiple SPA packets should not matter. After encryption, an observer should not be able to tell anything about the underlying plaintext (other than perhaps size since AES is a block cipher). Let's verify this for 1,000 SPA packets encrypted with the default CBC mode - they are all encrypted with the same key 'fwknoptest' by the script: $ ./ -f -r -c 1000 --base64-decode
[+] Running fwknop client via the following command:

LD_LIBRARY_PATH=../../lib/.libs ../../client/.libs/fwknop -A tcp/22 -a -D --get-key local_spa.key -B -b -v --test -M cbc

[+] Read in 1000 SPA packets...
[+] Min entropy: 7.75 at byte: 54
[+] Max entropy: 7.86 at byte: 115
[+] Creating entropy.gif gnuplot graph...
This produces the gnuplot graph below. Perfectly random data would produce 8 bits of entropy per byte, and the min/max values of 7.75 and 7.86 along with the fairly uniform distribution of similar values across all of the SPA byte positions implies that there is little relation from one SPA packet to the next - good. SPA entropy for CBC mode As an aside, here is what ent reports against the local /dev/urandom entropy source on my Linux system, and it is the "Entropy =" line that parses for each SPA byte slice: $ dd if=/dev/urandom count=1000 |ent
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.128497 s, 4.0 MB/s
Entropy = 7.999625 bits per byte.

Optimum compression would reduce the size
of this 512000 byte file by 0 percent.

Chi square distribution for 512000 samples is 265.77, and randomly
would exceed this value 50.00 percent of the times.

Arithmetic mean value of data bytes is 127.5076 (127.5 = random).
Monte Carlo value for Pi is 3.138715386 (error 0.09 percent).
Serial correlation coefficient is -0.001293 (totally uncorrelated = 0.0).
Now, let's switch to ECB mode and see what happens (just run the script with '-e ecb'): SPA entropy for ECB mode Well, that still looks pretty good. Revisiting the ECB encrypted image of Tux above for a moment - the reason that the Tux outline can be seen in the encrypted version is that in the JPG image file there must be identical blocks in multiple locations to represent the solid black regions. These blocks are all encrypted in the same way by AES in ECB mode, so the outline persists. But, this is one instance of ECB encryption against a file that has multiple identical blocks. For the encrypted SPA packets, we're dealing with 1,000 separate instances of encrypted data (all with the same key). Across this data set there are certainly lots of identical plaintext blocks (all of the SPA packets request access for source IP to destination port tcp/22 for example), but the encrypted data still shows a high level of entropy. This source of entropy is provided by the random salt values that are used to generate the initialization vector and final encryption key for each encrypted SPA packet. As proof, if we apply the following patch to force the salt to zero for all SPA packets (of course, one would not want to use this patch in practice): $ git diff lib/cipher_funcs.c
diff --git a/lib/cipher_funcs.c b/lib/cipher_funcs.c
index 0a0ce3b..32c8bd6 100644
--- a/lib/cipher_funcs.c
+++ b/lib/cipher_funcs.c
@@ -153,6 +153,8 @@ rij_salt_and_iv(RIJNDAEL_context *ctx, const char *pass, const unsigned char *da
         get_random_data(ctx->salt, 8);

+    memset(ctx->salt, 0x00, 8);
     /* Now generate the key and initialization vector.
      * (again it is the perl Crypt::CBC way, with a touch of
      * fwknop).
Here is what reports after recompiling fwknop with the patch above: SPA entropy for ECB mode zero salt Now we can easily see where there are identical blocks across the SPA packet data set. The first eight bytes contains the salt, so these are all zero (note that fwknop strips the usual "Salted__" prefix before transmitting an SPA packet on the wire). The next 16 bytes are the random bytes that fwknop includes in every SPA packet so these bytes have high entropy. Next up are the username and timestamp - the later changes with each second, so there is some entropy there since it takes a few seconds to create the 1,000 SPA packet data set. Then the entropy goes back to zero with the next fields and there isn't any decent entropy until the final message digest.

As a final contrasting case, let's leave the patch applied to force the salt to zero, but now switch back to CBC mode: SPA entropy for CBC mode zero salt In CBC mode, the random data included by the fwknop client now results in decent entropy even though the salt is zero. This is because every ciphertext block in CBC mode depends on all previous plaintext blocks, so randomness in one plaintext block implies that every subsequent encrypted block will look different from one SPA packet to the next. This graphically shows that CBC mode is a better choice for strong security. Now, if the pseudo random number generator on the local operating system is poorly implemented, this will negatively impact ciphertext entropy regardless of the encryption mode, but still CBC mode is a better alternative than ECB mode.

Although is geared towards measuring SPA packet entropy, this technique could certainly be generalized to arbitrary collections of ciphertext. If you know of such an implementation, please email me.

Software Release - fwknop-2.0

fwknop-2.0 released After a long development cycle, fwknop-2.0 has been released. This is the first production release of the fully re-written C version of fwknop, and is the culmination of an effort to provide Single Packet Authorization to multiple open source firewalls, embedded systems, mobile devices, and more. On the "server" side, supported firewalls now include iptables on Linux, ipfw on FreeBSD and Mac OS X, and pf on OpenBSD. The fwknop client is known to run on all of these platforms, and also functions on Windows systems running under Cygwin. There is also an Android client, and a good start on a iPhone client as well. On a personal note, I wish to thank Damien Stuart for a heroic effort to port most of the original perl code over to C. Also, several other people have made significant contributions including Jonathan Bennet, Max Kastanas, Sebastien Jeanquier, Ozmart, and others. If there are any issues, please get in touch with me directly or send an email to the fwknop mailing list.

Update 01/03: Both libfko library that powers much of fwknop operations and the fwknop client can be compiled as native Windows executables. In addition, there are perl and python bindings to libfko as well.

Update 01/07: Damien Stuart has built RPM files for fwknop on RHEL5, RHEL6, Fedora 15, 16, and 17 and for other architectures the Fedora koji build system can produce.

Software Release - fwknop-2.0rc5

fwknop-2.0rc5 released The 2.0rc5 candidate release of fwknop is available for download. There may be a few tweaks to the code before the official 2.0 release is made, but this is pretty close as-is. Significant development work has gone into fwknop since the 2.0rc4 release, and adds some major new functionality as well as fixing a few bugs. Here is a summary of the changes:

iPhone fwknop client: Max Kastanas has contributed an iPhone port of the fwknop client. He had already contributed on Android client, so the iPhone was the next natural step! We're looking for a maintainer of the iPhone code so that eventually it can be made available through the App Store. If you have iPhone development experience and are interested in taking this on, please contact me.

PF firewall support on OpenBSD: For quite a while now fwknop has brought Single Packet Authorization support to iptables firewalls on Linux and ipfw firewalls on FreeBSD and Mac OS X systems. The 2.0rc5 release now introduces support for the PF firewall on OpenBSD systems. By interfacing with the pfctl command, fwknop creates new rules to access a protected service (such as SSHD) to a PF anchor. Rules are deleted out of a running anchor with pfctl -a <anchor> -f - with the expired rule(s) removed. There is support in the fwknop test suite (see the test/ directory in the fwknop-2.0rc5 sources) to validate fwknop operations on OpenBSD systems, and if there are any issues please let me know.

Expiring SPA keys: With large SPA deployments where many different encryption keys - either Rijndael or GPG keys - are used to service lots of external users, key management can become somewhat of a burden. This feature allows an expiration date to be set in the access.conf file on a per-key basis. Any SPA packet received for an expired key is ignored by fwknopd. This feature was suggested by ozmart from the fwknop mailing list.

FORCE_NAT mode: For iptables firewalls, a new FORCE_NAT mode has been implemented that works as follows: for any valid SPA packet, force the requested connection to be NAT'd through to the specified (usually internal) IP and port value. This is useful if there are multiple internal systems running a service such as SSHD, and you want to give transparent access to only one internal system for each stanza in the access.conf file. This way, multiple external users can each directly access only one internal system per SPA key.

lsof launcher: The fwknop lsof launcher (extras/fwknop-launcher/ is a lightweight daemon that allows the user to not have to manually run the fwknop client when attempting to gain access to a service that is protected by via fwknopd. This is accomplished by checking the output of lsof to look for pending connections in the SYN_SENT state, which (usually) indicate that a remote firewall is blocking the attempted connection. At this point, the launcher executes the fwknop client with the --get-key arg (so the user must place the key in the local filesystem) to generate an SPA packet for the attempted connection. The remote fwknopd daemon will reconfigure the firewall to allow temporary access, and this usually happens fast enough that the original connection attempt will then succeed as TCP retries to establish the connection. The idea for this was originally for a pcap-based connection watcher by Sebastien Jeanquier.

Several other changes and small fixes have been made as well. The fwknop test suite supports running all tests through the excellent valgrind project, and this enabled several memory handling issues to be found and corrected.

fwknop is released under the GPL version 2, and the complete fwknop-2.0rc5 ChangeLog can be found here via the fwknop gitweb interface. Single Packet Authorization Proxy

Vasilis Mavroudis has developed a web proxy called that allows anyone to generate an fwknop SPA packet on their behalf with just a web browser. Although fwknop client portability has improved quite a bit in anticipation of the upcoming fwknop-2.0 release, it is a nice addition to the SPA world to not need the fwknop client installed at all. There are probably several platforms where the native client might not function but can run a web browser.

Using the proxy requires that the user provide the SPA key over SSL to, but this is a necessary step in exchange for not having to install the fwknop client. As of this writing, SPA via gpg keys is not yet supported, but there are plans to support this in the future. All requests to generate an SPA packet are protected by a captcha.

Behind the scenes, executes the fwknop client on behalf of users, and Vasilis informed me that he's using the latest client code (written in C) instead of the older perl client. This is good since all recent development is done on the C version of fwknop in order to make it as small and lightweight as possible.

The service is free, and will hopefully be open-sourced at some point as well. If there are any issues, please either email me or open a ticket on the fwknop github interface. Here is a screenshot of the current site: SPA proxy

fwknop in the OpenWrt and Pentoo Linux Distributions

fwknop in OpenWrt The C version fwknop has now made it into the OpenWrt Linux distribution for embedded devices. Jonathan Bennett made this possible by contributing a Makefile for OpenWrt, and it was picked up the OpenWrt maintainers. It is good to see progress made towards the integration of Single Packet Authorization into operating systems that are designed to function as secure gateway devices between multiple networks.

So far, fwknop is available in the OpenWrt trunk packages feed, but will eventually become available via the opkg package manager too. Fortunately, OpenWrt makes everything available via git: $ git clone git:// openwrt_packages.git
Initialized empty Git repository in /home/mbr/src/openwrt_packages.git/.git/
remote: Counting objects: 56118, done.
remote: Compressing objects: 100% (21342/21342), done.
remote: Total 56118 (delta 29694), reused 54875 (delta 29054)
Receiving objects: 100% (56118/56118), 11.85 MiB | 2.57 MiB/s, done.
Resolving deltas: 100% (29694/29694), done.
$ cd openwrt_packages.git
$ git ls-files |grep fwknop
$ git log net/fwknop/Makefile
commit 89475e5d6136833fa3b59c3d47c4f2be02718c7a
Author: florian <florian@3c298f89-4303-0410-b956-a3cf2f4a3e73>
Date: Wed Aug 17 10:13:20 2011 +0000

[package] add fwknopd

Signed-off-by; Jonathan Bennett <[email removed]>

git-svn-id: svn:// 3c298f89-4303-0410-b956-a3cf2f4a3e73

In other news, both the perl and C versions of fwknop are also available in the Pentoo Linux distribution thanks to ozmart and the Pentoo maintainers. Pentoo is a live-cd distribution that is focused on security and derived from Gentoo. ozmart wrote a description of the use case for fwknop on Pentoo from a pentration testing perspective:

"...This is a useful script when combined with iptables and sshd. Configuration can accommodate pgp and replay attack checks. It allows the box to appear silent when running daemons if your box is deployed in say, a hostile environment.

It can also allow commands to be run without actually having to log into the box, say if you wanted to trigger something interesting from a remote location..."

fwknop in BackTrack 5

fwknop has made it into BackTrack Linux distribution. Well, it didn't make it into the default image, but it is in the BackTrack software repository so installing it is a snap with apt-get or aptitude. Below is a screenshot of the BackTrack 5 VM installed under VMware Player running on Ubuntu, and running a search shows both the fwknop-client and fwknop-server packages that are maintained by Franck Joncourt for the Debian distribution. fwknop in BackTrack 5
Update 06/24/2011: Thanks to Sebastien Jeanquier for calling this to my attention and helping to support the inclusion of fwknop in BackTrack Linux.

Single Packet Authorization on Android

fwknop on Android Kicking off 2011 with a bang is easy this year - Max Kastanas has ported the fwknop client to the Android mobile operating sytem. This brings Single Packet Authorization to Google's smart phones, and was accomplished using Damien Stuart's libfko implementation. All of the Android code can be found in the Cipherdyne Trac respository. A couple of screenshots of using the SPA app from an Android phone appear below - after the SPA packet is sent and verified passively by the fwknop daemon on the remote system, the fwknop Android app automatically launches Connectbot to access SSHD:

fwknop Android app fwknop Android app

A new SSH password guessing botnet: dd_ssh

dd_ssh botnet Every few months it seems that there is a new variation of a password guessing botnet. This time around, as reported by the SANS ISC and also by, a botnet called "dd_ssh" has been built up around a vulnerability in phpMyAdmin. (Here is the vulnerability announcement from the Debian security list.) Once a vulnerable webserver has been compromised, the botnet starts scanning random IP addresses for SSH servers and attempts to brute force passwords for those systems that accept an SSH connection. Although software like Fail2ban and DenyHosts can offer detection and mitigation by applying thresholds to the number of failed login attempts in the SSH log and communicate offending source IP's to a set of interested users, a fundamental problem is that the targeted SSH daemons are reachable in the first place from arbitrary source IP's. A more powerful solution is to hide your SSH daemon behind a default-drop firewall policy and only grant access through Single Packet Authorization. Any botnet that is scanning for SSH servers will not be able to guess any passwords or even exploit a zero-day vulnerability in an SSH daemon that is protected in this way.

Further, SSH brute forcing botnets have shown some innovative techniques where password guessing attempts are spread out over a long period of time and failed passwords are distributed across the botnet. The end result is that each botnet node knows the population of passwords that other nodes have already tried on each targeted SSH daemon, and so new passwords can be tried while still slipping beneath common detection thresholds. The same password never has to be tried twice botnet-wide. It is not clear yet whether the dd_ssh botnet uses a similar password distribution strategy, but either way I intend to stop such guessing games at the earliest stage possible and block all SSH connections by default.

Pure C Implementation of Single Packet Authorization

Pure C Implementation of Single Packet Authorization For the past several months, the pure-C implementation of Single Packet Authorization with fwknop has been in development, and the first release candidate of fwknop-2.0.0 is ready for download. Damien Stuart is the primary developer of this code, and I wish to thank him for all of his efforts in making this release possible. The original perl version of fwknop will continue to receive bug fixes as they are found, but the primary development effort is now on the lightweight and portable C code. The Trac interface for the perl code has moved to fwknop-perl. Any release that is at fwknop-2.0.0 or above will be the C implementation, and any release in the 1.9 series or before are in perl (and will remain available here). The perl and C versions are compatible with each other, so SPA packets created by the fwknop perl client can be properly decrypted and decoded by the new fwknopd C server, and similarly SPA packets built with the fwknop C client can be interpreted by the fwknopd perl server.

The heart of the new C code is the libfko library, which is used by both the fwknop client and the fwknopd server for SPA packet encryption/decryption and interpretation. The libfko library is portable to several platforms including Linux and Windows, and the standard GNU autoconf project is used for building fwknop from sources.

Given the compact nature of the C code, the fwknop-2.0.0rc1 release candidate is already known to function on OpenWRT, and Damien Stuart (the primary developer of the fwknop C code) has built packages for the OpenWRT Kamikaze and Backfire releases. This brings SPA functionality to embedded systems, and has been tested on a Linksys WRT54GS V2.0 router. The OpenWRT packages are available here.

There will be a few release candidates of fwknop-2.0.0 to ensure proper operation on a variety of platforms, and the emphasis will be on stability and ironing out any bugs. Also, fwknop-2.0.0 will initially support iptables firewalls on the servers side, with ipfw support coming soon after. There is one nice feature in fwknop-2.0.0rc1 that deserves a mention - the fwknop client now supports the notion of a ~/.fwknoprc file that allows a set of stanzas to be defined for remote SPA servers and customize the command line arguments used to access each one. Here is an example stanza: # Example for a destination server of to open access to
# SSH for an IP that is resolved externally
ACCESS tcp/22
ALLOW_IP resolve
Now, with the fwknop client, we reference this stanza with the -n command line argument like so: $ fwknop -n myssh -vvv
Resolved external IP (via as:
Enter encryption password:

FKO Field Values:

Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String:,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3

Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy


Final Packed/Encrypted/Encoded Data:


Generating SPA packet:
protocol: udp
port: 50023
send_spa_packet: bytes sent: 182
On the fwknopd server side, we see the following printed to the terminal in --foreground mode as the SPA packet is received and an iptables ACCEPT rule is added for SSH access and then removed after 30 seconds. Using Digest Cache: '/usr/var/run/fwknop/digest.cache' (entry count = 93)
PCAP filter is: udp port 50023
Starting fwknopd main event loop.

SPA Packet from IP: received.
SPA Packet: '87wQ3s171if9x8628P6k1oBrQEJEmS5c/+ECed5cfDmu8hGhC5esUCEVcI8IH5TdjWva44efebX+cfXUVPBU9c/6tJpOLwC8k/8sN87SgYkBK+m64aD4FnnndaeqsrIcO/oHcpXHk9S/8rqqypcumNtlXVZD/isS3Q729LMTa+2uI6TsGX9OQ3'

SPA Decode (res=0):
SPA Field Values:
Random Value: 8308719569638051
Username: mbr
Timestamp: 1280030806
FKO Version: 1.9.12
Message Type: 1
Message String:,tcp/22
Nat Access: <NULL>
Server Auth: <NULL>
Client Timeout: 0
Digest Type: 3
Encoded Data: 8308719569638051:bWJy:1280030806:1.9.12:1:OTYuMjQ0LjE3MS4yNTMsdGNwLzIy

Added Rule to FWKNOP_INPUT for, tcp/22 expires at 1280030886
RES=0, CMD_BUF: /sbin/iptables -t filter -L FWKNOP_INPUT --line-numbers -n 2>&1
RULES LIST: Chain FWKNOP_INPUT (1 references)
num target prot opt source destination
1 ACCEPT tcp -- tcp dpt:22 /* _exp_1280030886 */

Removed rule 1 from FWKNOP_INPUT with expire time of 1280030886.
« Previous