cipherdyne.org

Michael Rash, Security Researcher



gootrude    [Summary View]

Software Release - gootrude-0.2

Gootrude queries Yahoo Gootrude-0.2 is available for download. This release changes the default search engine from Google to Yahoo because it appears that Yahoo's Terms of Service do not prohibit automated queries (although I'm not a lawyer). Gootrude retains the ability to query Google at your own risk if you so desire. As long as you use Gootrude as designed to collect search engine results only once per day with a limited number of search terms (say, 20 or less), it is unlikely to provoke a negative reaction from search engines. Over time, support for many additional search engines will be added to Gootrude so that search results can be trended from all sorts of different data sources.

Also, this post should really have announced Gootrude as a "search results trender" instead of a "search trender". The goal of the project is made clear in the README, and here is the complete ChangeLog, for the Gootrude-0.2 release:

  • Added support for querying Yahoo for search results and made this the default because Google's TOS does not technially allow automated queries. However, Gootrude continues to support querying Google for search results, and responsible use of Gootrude once per day with a limited set of search terms should most likely not be a cause for concern for Google. A new syntax of the searchterms.conf file allows search terms to express which search engine should be queried on an individual basis according to the following syntax (note the middle element): [Linux "highspeed firewalls"] [yahoo:count] [Linux_highspeed_firewalls.dat]
  • Changed the default USER_AGENT variable to "Gootrude <version>".
  • Added a percentage-based offset for the plot minimum and maximum ranges on the y-axis (with a default of 10% controlled by two new variables MIN_PERCENT_DIFF and MAX_PERCENT_DIFF). This value can also be set by the search terms themselves via an additional config element as follows (for a 20% offset instead for example): [yrange:20%/20%]
  • Increased SEARCH_TERM_DELAY to two seconds to be nicer to search engines.

Trending Low-Volume Google Searches - Introducing Gootrude

Introducing Gootrude UPDATE 06/16/08: After making Slashdot today and reading the comments, I feel I should clarify a few things regarding this post. First, it was pointed out (quite rightly) that Gootrude trends the number of documents in Google's index that contain each search term, whereas Google Trends tracks the search volume associated with a search term. These are not the same thing. However, I would wager that they are related, though certainly not in some nice 1:1 correspondence. That is, if a relatively unique word - say, "myspace" - is mentioned within 100,000 web sites, then the number of times this word is used as a search term in Google will be higher than if "myspace" were mentioned in only 10 web sites just because it is exposed to many more people who would then be interested in all things "myspace". I would also wager that the reverse is true. So, if "myspace" is an extremely popular search term, then it will probably also appear in many more web sites around the Internet. Gootrude attempts to trend the number of hits a given search term returns in Google, and by the above reasoning, there is a loose correspondence between this and the number of times the term is searched for in Google. Perhaps this is not useful, but only time will tell after Gootrude is used by a lot of people. If you will allow the above interpretation, then the remainder of the post below makes sense.

Now for the original post:

The Google Trends project allows you to input search terms like "Myspace", "2008 election", or "Linux", and see how Google tracks how popular these search terms are over time. The resulting graphs can be quite interesting - spikes in search volume can sometimes be correlated against particular news articles and world events, and the Google Trends interface points these out.

This is a handy tool, but there are many search terms that Google Trends does not display any results for. Such terms (such as "Linux Firewalls" - with the quotes) have insufficient search volumes to display graphs according to the error message that Google Trends generates. Fair enough. I suppose that Google sets an internal threshold on search volume, and this threshold could be set for reasons that range anywhere from Google Trends is still experimental to Google not wanting to provide data on how it builds its massive search index for emerging search terms. Either way, I would like a way to see search term trends that Google doesn't currently make available to me.

Although I'm an open source developer and author, search terms related to my projects are not popular enough yet to be displayed by Google Trends. So, I had to roll my own trending mechanism, and this blog post announces the release of a new open source tool Gootrude (see the quick start, source code, and download links) that I wrote to do just this. The basic strategy is to take a collection of search terms defined by the user, automatically query Google for the number of results associated with each of these search terms (this is displayed by Google when doing a web search), and graph these numbers over time with Gnuplot. At this point let me state up front that Gootrude only makes use of data that Google freely provides to everyone with normal web searches, and is meant to be run once per day (so as to not be a pest in terms of the numbers of queries it makes). As an example, if you type in the word "security" into the Google web interface, it will return a string like "Results 1 - 10 of about 1,010,000,000". The "1,010,000,000" number is collected by Gootrude and stored in a file along with the current time.

For the past year, I have sent a set of search terms through Google once per day with Gootrude and the results are displayed below. Visible within the data returned from Google are strange oscillations that vary quite a bit more than I would have expected, and also evidence for what happens when a large site (like linux.com) posts an article about a Cipherdyne project.

First, below is the graph of the fairly unique word "cipherdyne" since late June 2007. The filled-in red curve is the absolute number of search results (taken each day around 1am), and the green line is the 10-day moving average. Gootrude plot of cipherdyne search term As you can see, at the beginning of the graph around July 1st Google steadily shows about 28,000 results for "cipherdyne", but towards the end of July this dips to well below 20,000 only to rebound in August to about 30,000. Then, beginning around March 1st, 2008 the results shoot up to over 100,000 briefly and then back to around 70,000 in May. How does one interpret this data? It seems unlikely that these fluctuations can be entirely explained by "actual" day-by-day changes in how external sites reference the term "cipherdyne" - there must be some index updating component that is internal to Google at work here, and we'll see a better example of this below.

Now, here is the graph of the search term "gpgdir": Gootrude plot of gpgdir search term The most obvious feature of the gpgdir graph above is the large spike to around 60,000 results around May 1st. It turns out that an article was posted to linux.com on the 24th of April, so given that "gpgdir" is not a common word, the spike seems nicely correlated with the posting of the article as it got bounced around the Internet and blogosphere. A more interesting feature perhaps is the sharp cyclic oscillation between July and December 2007. During this time, search results for gpgdir bounced from 1,000 to around 10,000 and back again several times, and the transition each time was fast - making the jump to 10,000 over the course of two days and then stabilizing for about 10 days or so and then back down to 1,000. It is almost as though Google was trying to establish the proper order of magnitude for "gpgdir" search results during this time via a sort of step function.

Finally here is the graph of "single packet authorization": Gootrude plot of single packet authorization Again, we see a dramatic spike in search results - from around 5,000 to well over 50,000 and settling down to about 10,000 around the beginning of June. Although there has been some activity related to SPA in the Ubuntu forums and also in the Gentoo forums, if this caused Google to report the search results as over 50,000 why did this number return so precipitously back to around 10,000? The links have not gone away, but they were probably mentioned on other referencing sites and then moved to less important pages over time on those sites. Perhaps Google is trying to find the appropriate steady state for its search results, and there are many factors that Google takes into account that are not available to the public.

There are lots of unanswered questions this sort of data brings to mind:

  • All of the data for the above graphs was collected from a single Linux system. How different would the results be if several systems in different geographic locations collected the data and the average for each data point was used instead?
  • Each data point was collected around 1am every morning. If the data collection time were, say, 1pm, would the results have been significantly different?
  • What is the "optimal" time scale for the moving average? Given that Google's own Trends interface seems to show search results on the macro level, would a much longer moving average than 10 days - perhaps on the order of several weeks - be a more accurate reflection of search popularity?
One thing is clear - getting search results that are meaningful is much easier with unusual search terms. With the posting of this blog entry, the term "Gootrude" should evolve nicely within Google results, and the graph of these results will be updated daily on the main Gootrude page so you can see this evolution as it unfolds.

In closing, I would like to mention that Gootrude is just getting started, so there are lots of enhancements that need to be made. Some of the most important features to develop are:

  • Integration with the Google Charts API.
  • Development of an online web portal for Gootrude so that users don't have to have their own infrastructure to run Gootrude.
  • Ability to import search data from different Gootrude collection systems.
  • Add support for data collected from additional search engines.
If you are an open source developer and would like to contribute, see the TODO file for an updated list of development tasks, or send me an email (mbr[at]cipherdyne[dot]org). Also, if you have any ideas or feedback on why some of the graphs above look the way they do in the context of how Google builds its index, please email me.

Finally, here are a few additional graphs of search terms over the past year: Gootrude plot of fwsnort Gootrude plot of michael rash Gootrude plot of Linux Firewalls Attack Detection Gootrude plot of single packet authentication Gootrude plot of iptables attack visualization