Nutch: Getting my Feet Wet

My motivation for learning Nutch is twofold. The first is that we are using Nutch for a number of our more recent crawls, so I figured its something I should know about. The second is that Nutch uses Hadoop Map-Reduce, so I figured I would get some Map-Reduce programming tips by looking at Nutch sources.

This post describes my attempt to crawl this blog using Nutch and index it. It also describes a very simple plugin to filter URLs by pattern at index time.

Command line Usage

Nutch can be used as a Search Appliance that bundles crawling, indexing and search. Such bundling can be very useful if you just want to use Nutch with whatever it has (and it has plenty for most people), but I was more interested in trying to write some plugins for it, so my approach is to crawl once, then reuse the downloaded pages as an input data to test my plugins.

Nutch's "crawl" subcommand is actually a combination of other subcommands, as detailed on this page. Before I ran a crawl, however, I needed to set up two things.

1. Create Seed List.
2. Customize Nutch Configuration.

Create Seed List

This is a flat file consisting of top level URL(s) for the site being crawled. In my case, I just have one URL as shown below.

1	http://sujitpal.blogspot.com/

Customize Nutch Configuration

Nutch's main configuration file is in $NUTCH_HOME/conf/nutch-defaults.xml, where $NUTCH_HOME is the directory where you installed Nutch. In my case it is /opt/nutch-1.0. The recommended approach is to override the entries that you need in conf/nutch-site.xml, which I did. Here is what my nutch-site.xml file looks like:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>http.agent.name</name> <value>testnutch</value> </property> <property> <name>http.robots.agents</name> <value>testnutch,*</value> </property> <property> <name>http.agent.description</name> <value>Nutch testing</value> </property> <property> <name>http.agent.url</name> <value>http://sujitpal.blogspot.com</value> </property> <property> <name>http.agent.email</name> <value>me@myprovider.net</value> </property> <property> <name>plugin.includes</name> <value> myplugins|protocol-http|urlfilter-regex|parse-(text|html\ |js)|index-(basic|anchor)|query-(basic|site|url)|\ response-(json|xml)|summary-basic|scoring-opic\ |urlnormalizer-(pass|regex|basic) </value> </property> </configuration>

Here is the crawl command to do an initial crawl, all the way to the index, in one fell swoop. I figured I would do the subcommands separately when I do the crawl the next time round. You may have to experiment with the depth a bit - my depth of 4 was based on the depth of the navigation path to a post using the left navigation bar.

1 2 3 4

sujit@sirocco:~$ CRAWL_DIR=/home/sujit/tmp sujit@sirocco:~$ cd /opt/nutch-1.0 sujit@sirocco:/opt/nutch-1.0$ bin/nutch crawl $CRAWL_DIR/seeds.txt \ -dir $CRAWL_DIR/data -depth 4 2>&1 | tee $CRAWL_DIR/crawl.log

Once the crawl is done, $CRAWL_DIR/data looks like this:

The crawldb contains information about the various URLs that were crawled, the linkdb stores incoming links for each url, segments is where the raw page contents are stored, indexes are where the initial index is created, and index is where the contents of indexes are merged into the final index.

You can quickly see how the crawl did by looking at its statistics. Here the db_fetched is the number of URLs that were fetched. The db_unfetched indicates how many could not be fetched for some reason (more on that later), and db_gone means it could not even find it (ie HTTP 404).

1 2 3 4 5 6 7 8 9 10 11 12 13

sujit@sirocco:/opt/nutch-1.0$ bin/nutch readdb \ $CRAWL_DIR/data/crawldb -stats CrawlDb statistics start: /home/sujit/tmp/data/crawldb Statistics for CrawlDb: /home/sujit/tmp/data/crawldb TOTAL urls: 446 retry 0: 446 min score: 0.0 avg score: 0.006143498 max score: 1.159 status 1 (db_unfetched): 115 status 2 (db_fetched): 222 status 3 (db_gone): 109 CrawlDb statistics: done

You can also get some detailed information by dumping out the crawldb and processing it. For example, I wanted to see what URLs were retrieved in the crawl, with a view to suppressing the non-blog post pages. Here is what I did.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

# dump out the crawldb to a structured text format sujit@sirocco:/opt/nutch-1.0$ bin/nutch readdb \ $CRAWL_DIR/data/crawldb -dump $CRAWL_DIR/reports_crawl # see how many URLs were crawled sujit@sirocco:~/tmp/reports_crawl$ cat part-00000 | \ cut -f1 -d"\t" | \ ## \t is a ctrl-V ctrl-I grep -E "^http://" | wc -l 446 # see how many URLs there will be if we remove the ones we don't want sujit@sirocco:~/tmp/reports_crawl$ cat part-00000 | \ cut -f1 -d"\t" | \ ## \t is a ctrl-V ctrl-I grep -E "^http://" | \ grep -vE "archive|label|feeds" | \ grep ".html" | wc -l 146

Similar to the crawldb, one could dump out the linkdb as well. The linkdb contains information about incoming links for each URL. This is used for link analysis. I didn't have a use (not yet, at least) for the linkdb results. The command to dump the linkdb is:

1 2	sujit@sirocco:/opt/nutch-1.0$ bin/nutch readlinkdb \ $CRAWL_DIR/data/linkdb -dump $CRAWL_DIR/reports_link

To run the indexing process separate from the crawl (perhaps to regenerate the index after changing or adding a plugin, you need to run the index, dedup and merge commands. The index subcommand writes to the indexes subdirectory, dedups it there, and the merge copies the merged version into the index subdirectory.

1 2 3 4 5 6 7 8

sujit@sirocco:/opt/nutch-1.0$ bin/nutch index \ $CRAWL_DIR/data/indexes $CRAWL_DIR/data/crawldb \ $CRAWL_DIR/data/linkdb $CRAWL_DIR/data/segments/* sujit@sirocco:/opt/nutch-1.0$ bin/nutch dedup \ $CRAWL_DIR/data/indexes sujit@sirocco:/opt/nutch-1.0$ bin/nutch merge \ -workingdir $CRAWL_DIR/data/work $CRAWL_DIR/data/index \ $CRAWL_DIR/data/indexes

A Simple Plugin

Nutch's power comes from its plugin based architecture. Its core is quite small, but user-written plugin code can be plugged in to its various extension points. As this page says:

Since everybody can write a plugin, hopefully in future there will be a large set of plugins to choose from. At that point Nutch administrators will each be able to assemble their own search engine based on her/his particular needs needs by installing the plugins he or she is interested in... Each developer is focused on his/her own context. The core developers are able to write code for the nutch core engine and provide a described interface - a plug. A plugin developer is able to focus on the functionality of their specific plugin without worrying about how the system as a whole works. They only need to know what kind of data the plug and the plugin exchange. Since both sides are encapsulated nobody needs to take care of the integration of the other side...

To get familiar with how to write such a plugin, I decided on a really simple one that throws away pages that are not my blog posts - these include the top page, all the archive pages, and the comment feeds. I had initially planned to make it a URL filter, but that would have meant that this page would never be crawled, and therefore I would miss the post pages that are accessible from these pages. So the plugin I describe will filter out pages with these specific URL patterns at the indexing phase, ie, it will be an IndexFilter.

I followed this step-by-step guide to writing a Nutch plugin. It is written for version 0.9, but it is accurate (as far as I can see) for version 1.0 as well. For development, I found this page very helpful for getting Nutch setup with all its plugins within Eclipse - a starting point for writing my own.

The only extra JAR files I had to add are the jai_core.jar and jai_codec.jar that were listed in the Nutch 1.0 distribution's README.txt.

To hold my plugins, I created a plugin source directory called myplugins. My plugin contains three files, an Ant build.xml file, a plugin.xml file to describe the plugin to Nutch, and the Java code for the plugin. The directory structure is the one recommended by the step-by-step guide.

The build.xml just delegates to the plugin's build-plugin.xml, so its basically a one-liner, as shown below:

1 2 3 4

<?xml version="1.0" encoding="UTF-8"?> <project name="myplugins" default="jar"> <import file="../build-plugin.xml"/> </project>

The code for the InvalidUrlIndexFilter.java is shown below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

// Source: src/plugin/myplugins/src/java/com/mycompany/nutch/indexing/InvalidUrlIndexFilter.java package com.mycompany.nutch.indexing; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.Text; import org.apache.log4j.Logger; import org.apache.nutch.crawl.CrawlDatum; import org.apache.nutch.crawl.Inlinks; import org.apache.nutch.indexer.IndexingException; import org.apache.nutch.indexer.IndexingFilter; import org.apache.nutch.indexer.NutchDocument; import org.apache.nutch.parse.Parse; /** * This indexing filter removes "invalid" urls that have been crawled * (out of necessity, since they lead to valid pages), but need to be * removed from the index. The invalid urls contain the string * "archive" (for archive pages which contain full text and links to * individual blog pages), "label" (tag based search result page with * full text of blogs labelled with the tag, and links to the individual * blog pages), and "feeds" (for RSS/Atom feeds, which we don't care * about, since they are duplicates of our blog pages). We also don't * care about the urls that are not suffixed with a .html extension. * @author Sujit Pal * @version $Revision$ */ public class InvalidUrlIndexFilter implements IndexingFilter { private static final Logger LOGGER = Logger.getLogger(InvalidUrlIndexFilter.class); private Configuration conf; public void addIndexBackendOptions(Configuration conf) { // NOOP return; } public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) throws IndexingException { if (url == null) { return null; } if (url.find("archive") > -1 || url.find("label") > -1 || url.find("feeds") > -1) { // filter out if url contains "archive", "label" or "feeds" LOGGER.debug("Skipping URL: " + new String(url.getBytes())); return null; } if (url.find(".html") == -1) { // filter out if url does not have a .html extension LOGGER.debug("Skipping URL: " + new String(url.getBytes())); return null; } // otherwise, return the document return doc; } public Configuration getConf() { return conf; } public void setConf(Configuration conf) { this.conf = conf; } }

The plugin.xml file that describes the filter to Nutch looks like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

<?xml version="1.0" encoding="UTF-8"?> <plugin id="myplugins" name="My test plugins for Nutch" version="0.0.1" provider-name="mycompany.com"> <runtime> <library name="myplugins.jar"> <export name="*"/> </library> </runtime> <extension id="com.mycompany.nutch.indexing.InvalidUrlIndexFilter" name="Invalid URL Filter" point="org.apache.nutch.indexer.IndexingFilter"> <implementation id="myplugins-invalidurlfilter" class="com.mycompany.nutch.indexing.InvalidUrlIndexFilter"/> </extension> </plugin>

To compile this, we execute ant at the myplugins directory, then we go up a level and execute ant again. This builds the myplugins.jar, and copies it, and the plugin.xml to the build/plugins/myplugins directory.

I use a different nutch installation to run on than the one I developed on, so I created a directory plugins/myplugins under my /opt/nutch-1.0 (the install I run on), and copied these two files in there.

The one other change I did was to turn on DEBUG level logging for my package in Nutch's conf/log4j.properties.

1 2	# Logging for development log4j.logger.com.mycompany.nutch=DEBUG

To make Nutch recognise the new plugin.xml, the next step is to add this to the plugin.includes in the nutch_site.xml. My nutch-site.xml file shown above already contains this information, essentially I've added the myplugins string to the regular expression for the value tag.

To run it, I run the index, dedup and merge commands, and I notice from the log files that URLs are indeed getting skipped. I had done a quick test on the index that was generated before the plugin, and there were 166 records in there - the index generated after the plugin has only 124. I also had a query (title:april) that was returning 4 archive pages for April, which no longer returns any results, so my filter is doing its job.

This was a fairly simple plugin, but now that I have this working, I have some more plugin ideas that I want to implement, which I will probably talk about next week if I get it done.