Experiments with Prompt Compression

I recently came across Prompt Compression (in the context of Prompt Engineering on Large Language Models) on this short course on Prompt Compression and Query Optimization from DeepLearning.AI. Essentially it involves compressing the prompt text using a trained model to drop non-essential tokens. The resulting prompt is shorter (and in cases of the original context being longer than the LLM's context limit, not truncated) but retains the original semantic meaning. Because it is short, the LLM can process it faster and cheaper, and in some cases get around the Lost In the Middle problems observed with long contexts.

The course demonstrated Prompt Compression using the LLMLingua library (paper) from Microsoft. I had heard about LLMLingua previously from my ex-colleague Raahul Dutta, who blogged about it on his Edition 26: LLMLingua - A Zip Technique for Prompt post, but at the time I thought maybe it was more in the realm of research. Seeing it mentioned in the DeepLearning.AI course made it feel more mainstream, so I tried it out a single query from my domain using their Quick Start example, compressing the prompt with the small llmlingua-2-bert-base-multilingual-cased-meetingbank model, and using Anthropic's Claude-v2 on AWS Bedrock as the LLM.

Compressing the prompt for the single query gave me a better answer than without compression, at least going by inspecting the answer produced by the LLM before and after compression. Encouraged by these results, I decided to evaluate the technique using a set of around 50 queries I had lying around (along with a vector search index) from a previous project. This post describes the evaluation process and the results I obtained from it.

My baseline was a naive RAG pipeline, with the context retrieved by vector matching the query against the corpus, and then incorporated into a prompt that looks like this. The index is an OpenSearch index containing vectors of document chunks, vectorization was done using the all-MiniLM-L6-v2 pre-trained SentenceTransformers encoder, and the LLM is Claude-2 (on AWS Bedrock as mentioned previously).

1 2 3 4 5 6 7 8 9

Human: You are a medical expert tasked with answering questions expressed as short phrases. Given the following CONTEXT, answer the QUESTION. CONTEXT: {context} QUESTION: {question} Assistant:

While the structure of the prompt is pretty standard, LLMLingua explicitly requires the prompt to be composed of an instruction (the System prompt beginning with Human:), the demonstration (the {context}) and the question (the actual quary to the RAG pipeline). The LLMLingua Compressor's compress function expects these to be passed separately as parameters. Presumably, it compresses the demonstration with respect to the instruction and the question, i.e. context tokens that are non-essential given the instruction and question are dropped during the compression process.

The baseline for the experiment uses the context as retrieved from the vector store without compression, and we evaluate the effects of prompt compression using the two models listed in LLMLingua's Quick Start -- llmlingua-2-bert-base-multilingual-cased-meetingbank (small model) and llmlingua-2-bert-base-multilingual-cased-meetingbank (large model). The three pipelines -- baseline, compression using small model, and compression using large model -- are run against my 50 query dataset. The examples imply that the compressed prompt can be provided as-is to the LLM, but I found that (at least with the small model), the resulting compressed prompt generates answers that does not always capture all of the question's nuance. So I ended up substituting only the {context} part of the prompt with the generated compressed prompt in my experiments.

Our evaluation metric is Answer Relevance as defined by the RAGAS project. It is a measure of how relevant the generated answer is given the question. To calculate this, we prompt the LLM to generate a number of (in our case, upto 10) questions from the generated answer. We then compute the cosine similarity of the vector of each generated question with the vector of the actual question. The average of these cosine similarities is the Answer Relevance. Question Generation from the answer is done by prompting Claude-2 and vectorization of the original and generated questions are done using the same SentenceTransformer encoder we used for retrieval.

Contrary to what I saw in my first example, the results were mixed when run against the 50 queries. Prompt Compression does result in faster response times, but it degraded the Answer Relevance scores more times than improve it. This is true for both the small and large compression models. Here are plots of the difference of the Answer Relevance score for the compressed prompt against the baseline uncompressed prompt for each compression model. The vertical red line separates the cases where compression is hurting answer relevance (left side) versus improving answer relevance (right side). In general, it seems like compression helps when the input prompt is longer, which intuitively makes sense. But there doesn't seem to be a simple way to know up front if prompt compression is going to help or hurt.

I used the following parameters to instantiate LLMLingua's PromptCompressor object and to call its compress_prompt function. These are the same parameters that were shown in the Quick Start. It is possible I may have gotten different / better results if I had experimented a bit with the parameters.

1 2 3 4 5 6 7 8 9

from llmlingua import PromptCompressor compressor = PromptCompressor(model_name=model_name, use_llmlingua2=True) compressed = compressor.compress_prompt(contexts, instruction=instruction, question=query, target_token=500, condition_compare=True, condition_in_question="after", rank_method="longllmlingua", use_sentence_level_filter=False, context_budget="+100", dynamic_context_compression_ratio=0.4, reorder_context="sort") compressed_context = compressed["compressed_prompt"]

A few observations about the compressed context. The number of context documents changes before and after compression. In my case, all input contexts had 10 chunks, and the output would vary between 3-5 chunks, which probably leads to the elimination of Lost in the Middle side-effects as claimed in LLMLingua's documentation. Also, the resulting context chunks are shorter and seems to be a string of keywords rather than coherent sentences, basically unintelligible to human readers, but intelligible to the LLM.

Overall, Prompt Compression seems like an interesting and very powerful technique which can result in savings in time and money if used judiciously. Their paper shows very impressive results on some standard benchmark datasets with supervised learning style metrics using a variety of compression ratios. I used Answer Relevance because it can be computed without needing domain experts to grade additional answers. But it is likely that I am missing some important optimization, so I am curious if any of you have tried it, and if your results are different from mine. If so, would appreciate any pointers to things you think I might be missing.

Finetuning RAGAS Metrics using DSPy

Last month, I decided to sign-up for the Google AI Hackathon, where Google provided access to their Gemini Large Language Model (LLM) and tasked participants with building a creative application on top of it. I have worked with Anthropic's Claude and OpenAI's GPT-3 at work previously, and I was curious to see how Gemini stacked up against them. I was joined in that effort by David Campbell and Mayank Bhaskar, my non-work colleagues from the TWIML (This Week In Machine Learning) Slack. Winners for the Google AI Hackathon were declared last Thursday, and whilte our project sadly did not win anything, the gallery provides examples of some very cool applications of LLMs (and Gemini in particular) for both business and personal tasks.

Our project was to automate the evaluation of RAG (Retrieval Augmented Generation) pipelines using LLMs. I have written previously about the potential of LLMs to evaluate search pipelines, but the scope of this effort is broader in that it attempts to evaluate all aspects of the RAG pipeline, not just search. We were inspired by the RAGAS project, which defines 8 metrics that cover various aspects of the RAG pipeline. Another inspiration for our project was the ARES paper, which shows that fine-tuning the LLM judges on synthetically generated outputs improves evaluation confidence.

Here is a short (3 minutes) video description of our project on Youtube. This was part of our submission for the hackathon. We provide some more information about our project in our blog post below.

We re-implemented the RAGAS metrics using LangChain Expression Language (LCEL) and applied them to (question, answer, context and ground truth) tuples from the AmnestyQA dataset to generate the scores for these metrics. My original reason for doing this, rather than using the using what RAGAS provided directly, was because I couldn't make them work properly with Claude. This was because Claude cannot read and write JSON as well as GPT-3 (it works better with XML), and RAGAS was developed using GPT-3. All the RAGAS metrics are prompt-based and transferrable across LLMs with minimal change, and the code is quite well written. I wasn't sure if I would encounter similar issues with Gemini, so it seemed easier to just re-implement the metrics from the ground up for Gemini using LCEL than try to figure out how to make RAGAS work with Gemini. However, as we will see shortly, it ended up being a good decision.

Next we re-implemented the metrics with DSPy. DSPy is a framework for optimizing LLM prompts. Unlike RAGAS, where we tell the LLM how to compute the metrics, with DSPy the general approach is to have very generic prompts and show the LLM what to do using few shot examples. The distinction is reminiscent of doing prediction using Rules Engines versus using Machine Learning. Extending the analogy a bit further, DSPy provides its BootstrapFewShotWithRandomSearch optimizer that allows you to search through its "hyperparameter space" of few shot examples, to find the best subset of examples to optimize the prompt with, with respect to some score metric you are optimizing for. In our case, we built the score metric to minimize the difference between the the score reported by the LCEL version of the metric and the score reporteed by the DSPy version. The result of this procedure are a set of prompts to generate the 8 RAG evaluation metrics that are optimized for the given domain.

To validate this claim, we generated histograms of scores for each metric using the LCEL and DSPy prompts, and compared how bimodal, or how tightly clustered around 0 and 1, they were. The intuition is that the more confident the LLM is about the evaluation, the more it will tend to deliver a confident judgment clustered around 0 or 1. In practice, we do see this happening in case of the DSPy prompts for all but 2 of the metrics, although the differences are not very large. This may be because we the AmnestyQA dataset is very small, only 20 questions.

To address the size of AmnestyQA dataset, Dave used the LLM to generate some more (question, context, answer, ground_truth) tuples given a question and answer pair from AmnestyQA and a Wikipedia retriever endpoint. The plan was for us to use this larger dataset for optimizing the DSPy prompts. However, rather than doing this completely unsupervised, we wanted to have a way for humans to validate and score the LCEL scores from these additional questions. We would then use these validated scores as the basis for optimizing the DSPy prompts for computing the various metrics.

This would require a web based tool that would allow humans to examine the output of each step of the LCEL metric score process. For example, the Faithfulness metric has two steps, the first is to extract facts from the answer, and the second is to provide a binary judgment of whether the context contains the fact. The score is computed by adding up the individual binary scores. The tool would allow us to view and update what facts were extracted in the first stage, and the binary output for each of the fact-context pairs. This is where implementing the RAGAS metrics on our own helped us, we refactored the code so the intermediate results were also available to the caller. Once the tool was in place, we would use it to validate our generated tuples and attempt to re-optimise the DSPy prompts. Mayank and Dave had started on this , but unfortunately we ran out of time before we could complete this step.

Another thing we noticed is that calculation of most of the metrics involves one or more subtasks to make some kind of binary (true / false) decision about a pair of strings. This is something that a smaller model, such as a T5 or a Sentence Transformer, could do quite easily, more predictably, faster, and at lower cost. As before, we could use extract the intermediate outputs from the LCEL metrics to create training data to do this. We could use DSPy and its BootstrapFindTune optimizer to fine-tune these smaller models, or fine-tune Sentence Transformers or BERT models for binary classification and hook them up into the evaluation pipeline.

Anyway, that was our project. Obviously, there is quite a bit of work remaining to make it into a viable product for LLM based evaluation using the strategy we laid out. But we believe we have demonstrated that this can be viable, that given sufficient training data (about 50-100 examples for the optimized prompt, and maybe 300-500 each for the binary classifiers), it should be possible to build metrics that are tailored to one's domain and that can deliver evaluation judgments with greater confidence than those built using simple prompt engineering. In case you are interested in exploring further, you can find our code and preliminary results at sujitpal/llm-rag-eval on GitHub.

Sentence Similarity using Word2Vec and Word Movers Distance

Sometime back, I read about the Word Mover's Distance (WMD) in the paper From Word Embeddings to Document Distances by Kusner, Sun, Kolkin and Weinberger. The WMD is a distance function that measures the distance between two texts as the cumulative sum of minimum distance each word in one text must move in vector space to the closest word in the other text. In the paper, the authors provide some examples where WMD is calculated against a Word2Vec vector space. Since Word2Vec word embeddings preserve aspects of the word's context, its a good way to capture semantic meaning (or difference in meaning) when calculating WMD.

The paper reminded me of a similar (in intent) algorithm that I had implemented earlier and written about in my post Computing Semantic Similarity for Short Sentences. There, we captured the semantic meaning using an external semantic network (Wordnet).

Since the problems were so similar, I figured that it might be interesting to compute the WMD for the sentence pairs in this paper and see how they match up with intuition. I already had lying around a dump of the GoogleNews vectors (pretrained vectors over about 100B words of Google News) from a previous project. The paper described results over a dataset of just 16 short sentence pairs, so I decided to do this interactively on Spark using a Databricks notebook. We use Databricks at work and its ideal for this kind of quick and dirty ad-hoc work.

First we load up our 16 sentence pairs. The input is 3 columns - sentence#1, sentence#2 and the original score, tab separated. Since we don't care about the original score, we discard it and convert the input to a pair.

Since we want to compare words across sentences in the same pair, it makes sense to have these words in the same worker when they are compared, so we add an index key to each sentence pair. The output of this cell is an RDD that looks like ((sentence1: String, sentence2: String), index: Long).

1 2 3 4 5 6 7 8 9 10

import org.apache.spark.storage.StorageLevel val sentencePairs = sc.textFile("sentence_pairs.txt") .map(line => { val Array(s1, s2, _) = line.split('\t') (s1, s2) }) .zipWithIndex .persist(StorageLevel.MEMORY_AND_DISK) sentencePairs.count()

WMD between two sentences (or between any two blobs of text) is computed as the sum of the distances between closest pairs of words in the texts. The words are pre-processed to remove stop words, so the next cell pulls in a list of English stopwords which I convert to a Set and broadcast to the Worker boxes.

1 2	val stopwords = sc.textFile("stopwords.txt").collect.toSet val bStopwords = sc.broadcast(stopwords)

We now split up both sentences into words (removing punctuation and splitting on whitespace), removing stopwords from each, then flatMap-ing them to the format (index: Long, (word1: String, word2: String)). This gives us a list of 71 word pairs.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

def getWordPairs(id: Long, s1: String, s2: String, stopwords: Set[String]): List[(Long, (String, String))] = { val w1s = s1.toLowerCase .replaceAll("\\p{Punct}", "") .split(" ") .filter(w => !stopwords.contains(w)) val w2s = s2.toLowerCase .replaceAll("\\p{Punct}", "") .split(" ") .filter(w => !stopwords.contains(w)) val wpairs = for (w1 <- w1s; w2 <- w2s) yield (id, (w1, w2)) wpairs.toList } val wordPairs = sentencePairs.flatMap(ssi => getWordPairs(ssi._2, ssi._1._1, ssi._1._2, bStopwords.value)) wordPairs.count()

Next we ingest the Word2Vec vectors. I've used Gensim's Word2Vec module to convert the the original Word2Vec binary format to TSV. The format of this dataset is (word: String, comma-separated list of 300 vector elements).

1 2 3 4 5

val w2vs = sc.textFile("GoogleNews-vectors-negative300.tsv") .map(line => { val Array(word, vector) = line.split('\t') (word, vector) })

Next, we join the wordPairs against the w2vs RDD on the RHS and the LHS words to get the 300 dimensional word2vec vector for the RHS and LHS word respectively. We do a lot of moving things around so I have used case matching instead of the less intuitive underscore syntax to represent tuple elements and subelements. Note that we need to hang on to the left word because we want to find the word that is closest to each left word.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

import breeze.linalg._ def dist(lvec: String, rvec: String): Double = { val lv = DenseVector(lvec.split(',').map(_.toDouble)) val rv = DenseVector(rvec.split(',').map(_.toDouble)) math.sqrt(sum((lv - rv) :* (lv - rv))) } val wordVectors = wordPairs.map({case (idx, (lword, rword)) => (rword, (idx, lword))}) .join(w2vs) // (rword, ((idx, lword), rvec)) .map({case (rword, ((idx, lword), rvec)) => (lword, (idx, rvec))}) .join(w2vs) // (lword, ((idx, rvec), lvec)) .map({case (lword, ((idx, rvec), lvec)) => ((idx, lword), (lvec, rvec))}) .map({case ((idx, lword), (lvec, rvec)) => ((idx, lword), List(dist(lvec, rvec)))}) .persist(StorageLevel.MEMORY_AND_DISK)

I used Euclidean Distance in Word2Vec space for distance between words. I also tried using Cosine Distance (1 - Cosine Similarity) with similar results. We then sum all the shortest distances across all LHS words to get the WMD for the sentence pair.

1 2 3 4

val bestWMDs = wordVectors.reduceByKey((a, b) => a ++ b) .mapValues(dists => dists.sortWith(_ < _).head) // dist to closest word .map({case ((idx, lword), wmd) => (idx, wmd)}) .reduceByKey((a, b) => a + b) // sum all wmds for sent

Finally, we join these WMD scores back into the original dataset using the pair index that we originally generated using zipWithIndex.

1 2 3 4 5 6 7 8 9 10

val sqlContext = new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ case class SentencePair(s1: String, s2: String, wmd: Double) val results = sentencePairs.map(_.swap) .join(bestWMDs) .map({case (id, ((s1, s2), wmd)) => SentencePair(s1, s2, wmd)}) val resultsDF = sqlContext.createDataFrame(results) .orderBy($"s1".asc, $"wmd".asc) display(resultsDF)

The results are shown below. The sentences are sorted by the LHS sentence first, then by WMD (lowest WMD first so we can easily see the closest sentence pairs first and compare them to pairs that are not as close).

LHS Sentence	RHS Sentence	WMD
A glass of cider.	A full cup of apple juice.	2.2169259719396095
Canis familiaris are animals.	Dogs are common pets.	1.859694788966317
Dogs are animals.	They are common pets.	1.4537090848972198
I have a hammer.	Take some nails.	1.1578027104196844
I have a hammer.	Take some apples.	1.3028564676146912
I have a pen.	Where is ink?	1.020277185488236
I have a pen.	Where do you live?	1.3924941078355293
I like that bachelor.	I like that unmarried man.	1.176742725809037
It is a dog.	That must be your dog.	0
It is a dog.	It is a pig.	1.04864558369858
It is a dog.	It is a log.	1.3798001799052624
John is very nice.	Is John very nice?	0
Red alcoholic drink.	Fresh orange juice.	3.1161814560971166
Red alcoholic drink.	A bottle of wine.	3.386809492524872
Red alcoholic drink.	Fresh apple juice.	3.505168296314785
Red alcoholic drink.	An English dictionary.	4.106139922327307

As you can see, the scoring seems correct. For example, it finds that a "glass of cider" and a "cup of apple juice" are quite similar, even though there are no shared words (except for stopwords). Similarly "I have a hammer" is more similar to "Take some nails" than "Take some apples". The only intuitively incorrect result in this set is that "Red alcoholic drink" is more similar to "Fresh orange juice" than a "A bottle of wine". However, "A bottle of wine" is more similar to "Red Alcoholic drink" than "Fresh apple juice" and "An English dictionary" respectively. So overall, it seems to work on my limited dataset.

In my case, I already have two sentences and I just have to find the distance between them. In cases where you have to find the closest sentence, the complexity of the algorithm is O(p³ log p). One suggestion is to prune the number of possible RHS sentences by thresholding on the centroid distance (WCD) or relaxed WMD (see the paper for details) between the two sentences, and only running the full WMD on the pruned set of sentence pairs.

Topic Modeling with Mahout on Amazon EMR

Introduction

The motivation for this work was a desire to understand the structure of a corpus in a manner different from what I am used to. Central to all our applications is a knowledge graph derived from our medical taxonomy. So any document corpus can easily be defined as a small set (50-100) of high level concepts, merely by rolling up document concepts into their parents until an adequate degree of granularity is achieved. I wanted to see if standard topic modeling techniques would yield comparable results. If so, perhaps the output of such a process could be used as feedback for concept creation.

This post describes Topic Modeling a smallish corpus (2,285 documents) from our document collection, using Apache Mahout's Latent Dirichlet Allocation (LDA) algorithm, and running it on Amazon Elastic Map Reduce (EMR) platform. Mahout provides the LDA implementation, as well as utilities for IO. The code I wrote work at the two ends of the pipeline, first to download and parse data for Mahout to consume, and then to produce a report of top terms in each topic category.

Even though Mahout (I used version 0.8) provided most of the functionality for this work, the experience was hardly straightforward. The official documentation is outdated, and I had to repeatedly refer to discussions on the Mahout Mailing lists to find solutions for problems I faced along the way. I found only one blog post based on Mahout version 0.5 that I used as a starting point. Of course, all's well that ends well, and I was ultimately able to get the top terms for each topic and the topic composition of each document.

Theory

The math behind LDA is quite formidable as you can see from its Wikipedia page, but here is a somewhat high-level view, selectively gleaned from this paper by Steyvers and Griffiths.

Topic Models are based upon the idea that documents are mixtures of topics, where a topic is a probability distribution over words. To make a new document, one chooses a distribution over topics. Then for each word in the document, one chooses a topic at random and draws a word from the topic.

In order to answer the (IMO more interesting) question of what topics make up a collection of documents, you invert this process. Each Topic Modeling algorithm does it differently. LDA provides an approximate iterative method to sample values sequentially, proceeding until sample values converge to the target distribution.

Preparing the data

The documents for this work come from our Content Management System, and this section describes the extraction code. Its included for completeness. Your setup is likely very different, so it may be of limited use to you. In any case, our CMS is loosely coupled to our web front end via a publisher, which serializes documents in JSON format onto a network filesystem. Content can be pulled off a REST API off this filesystem most efficiently if you know the "file ID". I use Solr to get a list of these file IDs, and download it to my local filesystem for further processing.

Processing consists of parsing out the text content of the files (each content type can define its own JSON format), then using NLTK to remove HTML tags, stopwords, numeric tokens and punctuation. The text versions of the JSON files are written out to another directory for feeding into the Mahout pipeline.

Code is in Python, its shown below. Hostnames and such have been changed to protect the innocent.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125

import json import nltk import os import os.path import string import urllib import urllib2 SOLR_SERVER = "http://solrserver.mycompany.com:8983/solr/select" CONTENT_SERVER = "http://contentserver.mycompany.com/view" JSON_OUTPUTDIR = "/path/to/data/hlcms_jsons" TEXT_OUTPUTDIR = "/path/to/data/hlcms_text" FILENAMES_FILE = "/tmp/hlcms_filenames.txt" STOPWORDS = nltk.corpus.stopwords.words("english") PUNCTUATIONS = {c:"" for c in string.punctuation} def textify(s): text = nltk.clean_html(s) sentences = nltk.sent_tokenize(text) words = [] for sentence in sentences: sent = sentence.encode("utf-8", 'ascii') sent = "".join([PUNCTUATIONS[c] if PUNCTUATIONS.has_key(c) else c for c in sent]) ws = nltk.word_tokenize(sent) for w in ws: if w in STOPWORDS: continue if w.replace(",", "").replace(".", "").isdigit(): continue words.append(w.lower()) return " ".join(words) # build list of all file parameter values from solr params = urllib.urlencode({ "q" : "sourcename:hlcms", "start" : "0", "rows" : "0", "fl" : "contenttype,cmsurl", "wt" : "json" }) conn = urllib.urlopen(SOLR_SERVER, params) rsp = json.load(conn) numfound = rsp["response"]["numFound"] print "# of CMS articles to download: ", numfound filenames = open(FILENAMES_FILE, 'wb') npages = int(numfound/10) + 1 for pg in range(0, npages): if pg % 100 == 0: print "Downloading HLCMS page #: %d" % (pg) params = urllib.urlencode({ "q" : "sourcename:hlcms", "start" : str(pg * 10), "rows" : "10", "fl" : "contenttype,cmsurl", "wt" : "json" }) conn = urllib.urlopen(SOLR_SERVER, params) rsp = json.load(conn) for doc in rsp["response"]["docs"]: try: contenttype = doc["contenttype"] cmsurl = doc["cmsurl"] filenames.write("%s-%s\n" % (contenttype, cmsurl)) except KeyError: continue filenames.close() # for each file parameter, build URL and extract data into local dir filenames2 = open(FILENAMES_FILE, 'rb') for filename in filenames2: fn = filename.strip() ofn = os.path.join(JSON_OUTPUTDIR, fn + ".json") print "Downloading file: ", fn try: output = open(ofn, 'wb') response = urllib2.urlopen(CONTENT_SERVER + "?file=" + fn + "&raw=true") output.write(response.read()) output.close() except IOError: continue filenames2.close() print "All files downloaded" # build parser for each content type to extract title and body for file in os.listdir(JSON_OUTPUTDIR): print "Parsing file: %s" % (file) fin = open(os.path.join(JSON_OUTPUTDIR, file), 'rb') ofn = os.path.join(TEXT_OUTPUTDIR, os.path.basename(file[0:file.rindex(".json")]) + ".txt") fout = open(ofn, 'wb') try: doc_json = json.load(fin) # parsing out title and body based on content type # since different content types can have own format if file.startswith("ctype1-"): for fval in ["title", "bm_intro", "bm_seo_body"]: fout.write("%s\n" % (textify(doc_json[fval]))) elif file.startswith("ctype2-"): for fval in ["body"]: fout.write("%s\n" % (textify(doc_json[fval]))) elif file.startswith("ctype3-"): for fval in ["title", "body"]: fout.write("%s\n" % (textify(doc_json[fval]))) elif file.startswith("ctype4-"): fout.write("%s\n" % (textify(doc_json["recipeDeck"]))) fout.write("%s\n" % (textify(". ".join([x.values()[0] for x in doc_json["directions"]])))) elif file.startswith("ctype5-"): for fval in ["title", "body"]: fout.write("%s\n" % (textify(doc_json[fval]))) else: continue except ValueError as e: print "ERROR!", e continue fout.close() fin.close() # filter out files with 0 bytes and remove them from text output directory for file in os.listdir(TEXT_OUTPUTDIR): fname = os.path.join(TEXT_OUTPUTDIR, file) size = os.path.getsize(fname) if size == 0: print "Deleting zero byte file:", os.path.basename(fname) os.remove(fname)

Converting Text Files to Sequence File

The end product of the step above is a directory of text files. Punctuations, stopwords and number tokens have been stripped (because they are of limited value as topic terms) and all characters have been lowercased (not strictly necessary, because the vectorization step takes care of that). So each file is essentially now a bag of words.

Our pipeline is Hadoop based, and Hadoop likes small number of large files, so this step converts the directory of 2,258 text files into a single large sequence file, where each row represents a single file. I run the mahout seqdirectory subcommand locally to do this, then copy the output to Amazon EMR using s3cmd (available on Ubuntu via apt-get and on Mac OS via macports).

1 2 3 4 5 6

sujit@localhost:data$ $MAHOUT_HOME/bin/mahout seqdirectory \ --input /path/to/data/hlcms_text \ --output /path/to/data/hlcms_seq \ -c UTF-8 sujit@localhost:data$ s3cmd put /path/to/data/hlcms_seq \ s3://mybucket/cmstopics/

Vectorizing the Input

The next step is to create a term-document matrix out of the sequence files. Once again, we can do this locally with the Mahout seq2sparse subcommand. I choose to do this on Amazon EMR - the only change is to specify the name of the class that corresponds to the seq2sparse subcommand (you can find this information in $MAHOUT_HOME/conf/driver.classes.default.props). You also need to copy over the Mahout job JAR to S3.

JAR location: s3n://mybucket/cmstopics/mahout-core-0.8-job.jar
JAR arguments:
org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles \
-i s3n://mybucket/cmstopics/hlcms_seq \
-o s3n://mybucket/cmstopics/hlcms_vec \
-wt tf

With Amazon's Hadoop Distribution (ie choosing Amazon Distribution for the Hadoop Version prompt in the AWS EMR console) results in this error.

1	Error running child : java.lang.NoSuchFieldError: LUCENE_43

This is very likely caused by the Amazon distribution gratitously including old Lucene JARS (older than the Lucene 4.3 the Mahout 0.8 job JAR includes) within it. At runtime Lucene classes from the Amazon JARs are being loaded, which don't know anything about LUCENE_43 because they do not (yet) exist for it. My solution was to try the MapR M7 distribution (at least partly based on the reason that Ted Dunning works for MapR and he is a committer for Mahout :-)). However, MapR (all distributions) require m1.large instances at minimum, so its a bit more expensive.

This step creates an output directory hlcms_vec that looks like this. Of these, the only ones of interest to this pipeline are the tf-vectors folder and the dictionary.file-0 file.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

hlcms_vec/ +-- df-count | +-- _SUCCESS | +-- part-r-00000 +-- dictionary.file-0 +-- frequency.file-0 +-- tf-vectors | +-- _SUCCESS | +-- part-r-00000 +-- tokenized-documents | +-- _SUCCESS | +-- part-m-00000 +-- wordcount +-- _SUCCESS +-- part-r-00000

Converting Keys to IntWritables

This step is not documented in the official documentation. The blog post does not mention it either, but thats probably because Mahout 0.5's lda subcommand was deprecated in favor of the cvb subcommand. The tf-vectors file contains (Text, VectorWritable) tuples, but cvb expects to read (IntWritable, VectorWritable). The rowid subcommand does this conversion. Interestingly, the rowid job is contained in mahout-examples-0.8-job.jar and not in the main job JAR. Attempting to run it on Amazon EMR on either Amazon or MapR distributions produces errors to the effect that it can only be run locally.

1 2 3 4 5 6 7 8 9

# running under MapR distribution java.io.IOException: \ Could not resolve any CLDB hostnames for cluster: mybucket:7222 # running under Amazon distribution java.lang.IllegalArgumentException: \ This file system object (hdfs://10.255.35.8:9000) does not support \ access to the request path 's3n://mybucket/cmstopics/cvb-vectors/docIndex'\ You possibly called FileSystem.get(conf) when you should have called \ FileSystem.get(uri, conf) to obtain a file system supporting your path.

So I ended up pulling down tf-vectors locally, converting to tf-vectors-cvb and then uploading back to S3.

1 2 3 4 5 6 7 8

sujit@localhost:data$ s3cmd get --recursive \ s3://mybucket/cmstopics/hlcms_vec/tf-vectors/ \ hlcms_vec/tf-vectors sujit@localhost:data$ $MAHOUT_HOME/bin/mahout rowid \ -i /path/to/data/hlcms_vec/tf-vectors \ -o /path/to/data/hlcms_vec/tf-vectors-cvb sujit@localhost:data$ s3cmd put tf-vectors-cvb \ s3://mybucket/cmstopics/hlcms_vec/

After this subcommand is run, there is an additional folder tf-vectors-cvb in the hlcms_vec folder. The tf-vectors-cvb folder contains 2 files, matrix and docindex. Our pipeline only cares about the data in the matrix file.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

hlcms_vec +-- df-count | +-- _SUCCESS | +-- part-r-00000 +-- dictionary.file-0 +-- frequency.file-0 +-- tf-vectors | +-- _SUCCESS | +-- part-r-00000 +-- tf-vectors-cvb | +-- docindex | +-- matrix +-- tokenized-documents | +-- _SUCCESS | +-- part-m-00000 +-- wordcount +-- _SUCCESS +-- part-r-00000

Run LDA on Modified term-vector input

Finally, we are ready to run LDA on our corpus. The Mahout lda subcommand has been deprecated and replaced with the cvb subcommand, which uses the Collapsed Variational Bayes (CVB) algorithm to do LDA. We run LDA with 50 topics (-k) for 30 iterations (-x) on Amazon EMR using a MapR distribution, with the following parameters.

JAR location: s3n://mybucket/cmstopics/mahout-core-0.8-job.jar
JAR arguments:
org.apache.mahout.clustering.lda.cvb.CVB0Driver \
-i s3n://mybucket/cmstopics/hlcms_vec/tf-vectors-cvb/matrix \
-dict s3n://mybucket/cmstopics/hlcms_vec/dictionary.file-0 \
-o s3n://mybucket/cmstopics/hlcms_lda/topicterm \
-dt s3n://mybucket/cmstopics/hlcms_lda/doctopic \
-k 50 \
-ow \
-x 30 \
-a 1 \
-e 1

Number of things to keep in mind here. For one, -nt (number of terms) should not be specified if -dict is specified, since it can be inferred from -dict (or your job may fail). Also don't specify -mt (model directory) since otherwise the job will fail if it can't find one.

The output of the job is two folders, doctopic and topicterm. Both contain sequence files with (IntWritable,VectorWritable) tuples. Each row of doctopic represents a document and the VectorWritable is a list of p(topic|doc) for a topic. Each row of topicterm represents a topic and the VectorWritable is a list of p(term|topic) values for each term.

1 2 3 4 5 6 7 8 9

hlcms_lda +-- doctopic | +-- _SUCCESS | +-- part-m-00000 +-- topicterm +-- _SUCCESS +-- part-m-00001 +-- ... +-- part-m-00009

Dump results into CSV

The official documentation says to use Mahout's ldatopics subcommand, but according to StackOverflow page, ldatopics is deprecated and you should use the vectordump subcommand instead.

The vectordump subcommand merges the information from the dictionary file and one of doctopic or topicterm and it writes out a CSV file representing a matrix of p(topic|doc) or p(term|topic) respectively. I wasn't sure how to dump out into a local filesystem on Amazon EMR, so I just copied the files locally using s3cmd and ran vectordump on them.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

sujit@localhost:data$ s3cmd get --recursive \ s3://mybucket/cmstopics/hlcms_lda hlcms_lda sujit@localhost:data$ $MAHOUT_HOME/bin/mahout vectordump \ -i /path/to/data/hlcms_lda/topicterm \ -d /path/to/data/hlcms_vec/dictionary.file-0 \ -dt sequencefile \ -c csv \ -p true \ -o ./p_term_topic.txt -sort /path/to/data/hlcms_lda/topicterm \ -vs 10 sujit@localhost:data$ $MAHOUT_HOME/bin/mahout vectordump \ -i /path/to/data/hlcms_lda/doctopic \ -d /path/to/data/hlcms_vec/dictionary.file-0 \ -dt sequencefile \ -c csv \ -p true \ -o ./p_topic_doc.txt -sort /path/to/data/hlcms_lda/doctopic \ -vs 10

The p_term_topic.txt contains the p(term|topic) for each of the 50 topics, one topic per row. The p_topic_doc.txt contains the p(topic|doc) values for each document, one document per row.

Create Reports

We can create some interesting reports out of the data computed above. One such would be to find the top 10 words for each topic cluster. Here is the code for this report:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

import operator import string terms = {} f = open("/path/to/data/p_term_topic.txt", 'rb') ln = 0 for line in f: if len(line.strip()) == 0: continue if ln == 0: # make {id,term} dictionary for use later tn = 0 for term in line.strip().split(","): terms[tn] = term tn += 1 else: # parse out topic and probability, then build map of term to score # finally sort by score and print top 10 terms for each topic. topic, probs = line.strip().split("\t") termProbs = {} pn = 0 for prob in probs.split(","): termProbs[terms[pn]] = float(prob) pn += 1 toptermProbs = sorted(termProbs.iteritems(), key=operator.itemgetter(1), reverse=True) print "Topic: %s" % (topic) print "\n".join([(" "*3 + x[0]) for x in toptermProbs[0:10]]) ln += 1 f.close()

And the results are shown (after some editing to make them easier to read) below:

Topic: 0	Topic: 1	Topic: 2	Topic: 3	Topic: 4
droids applaud technique explosions sufferers born delight succeed compliant warming	responds stools technique explosions applaud proposal stern centers warming succeed	responds applaud droids explosions proposal born delight sexually upsidedown hemophilia	elisa responds sufferers born delight sexually fully hemophilia fury upsidedown	technique sufferers stools droids explosions knees amount stabilized centers stern
Topic: 5	Topic: 6	Topic: 7	Topic: 8	Topic: 9
group's technique stools applaud born amount stern vascular vectors knees	technique droids stools authored interchangeably stern households vectors bleed muchneeded	sufferers technique responds explosions applaud born compliant stabilized recording punch	droids explosions responds technique born upsidedown hypogastric compliant flinn bleed	group's responds applaud explosions technique born vectors delight punch fully
Topic: 10	Topic: 11	Topic: 12	Topic: 13	Topic: 14
group's responds sufferers explosions droids authored proposal centers thick flinn	applaud droids sufferers technique responds stools born vectors delight succeed	explosions applaud stools stern born upsidedown delight fury recording hypogastric	sufferers applaud interchangeably muchneeded households stabilized sexually ninety succeed flinn	technique stools responds droids interchangeably centers muchneeded thick upsidedown punch
Topic: 15	Topic: 16	Topic: 17	Topic: 18	Topic: 19
group's responds sufferers technique stools explosions flinn hemophilia delight centers	responds applaud technique vectors knees stern stabilized vascular sexually recording	responds stools sufferers vectors centers ninety warming households muchneeded interchangeably	technique sufferers explosions proposal born hemophilia centers delight fury compliant	group's sufferers applaud droids stools born centers punch compliant delight
Topic: 20	Topic: 21	Topic: 22	Topic: 23	Topic: 24
technique responds sufferers applaud droids stools interchangeably amount born ninety	responds applaud sufferers droids born delight sexually flinn vascular thick	applaud explosions droids born delight upsidedown interchangeably amount compliant punch	technique explosions vectors fury stern vascular households untreatable hemophilia stabilized	technique droids applaud sufferers stools stern amount interchangeably households centers
Topic: 25	Topic: 26	Topic: 27	Topic: 28	Topic: 29
stools sufferers responds born knees amount vectors flinn untreatable upsidedown	stools explosions proposal authored droids vectors knees fury amount succeed	stools proposal responds applaud born knees amount vascular untreatable hypogastric	applaud technique explosions sufferers droids responds stabilized centers punch muchneeded	responds stools droids explosions interchangeably stern households ninety upsidedown amount
Topic: 30	Topic: 31	Topic: 32	Topic: 33	Topic: 34
responds explosions applaud sufferers stools droids centers compliant vectors thick	stools explosions droids technique vectors centers muchneeded thick flinn stabilized	responds technique droids stools explosions born interchangeably households fury hypogastric	applaud explosions droids technique compliant punch centers warming hemophilia fully	droids technique vectors stern interchangeably fury households muchneeded amount knees
Topic: 35	Topic: 36	Topic: 37	Topic: 38	Topic: 39
sufferers technique responds authored centers vectors interchangeably punch fully warming	technique stools responds droids authored stern fury ninety bleed compliant	elisa sufferers group's technique droids interchangeably centers vectors punch thick	stools proposal technique sexually upsidedown stabilized thick punch muchneeded compliant	interchangeably stabilized vectors centers punch compliant ninety delight hemophilia droids
Topic: 40	Topic: 41	Topic: 42	Topic: 43	Topic: 44
stools applaud responds sufferers authored born flinn interchangeably hypogastric fury	group's responds sufferers applaud authored centers fury bleed hypogastric stern	responds stools technique sufferers applaud vectors amount knees untreatable upsidedown	elisa technique explosions responds stools proposal stern succeed born warming	stools applaud authored interchangeably stern born ninety muchneeded households warming
Topic: 45	Topic: 46	Topic: 47	Topic: 48	Topic: 49
responds droids sufferers interchangeably fury vectors households ninety muchneeded stern	group's droids stools explosions applaud authored proposal sufferers interchangeably stabilized	sufferers group's explosions applaud responds droids technique stools interchangeably fury	amount stern knees flinn compliant sexually thick bleed upsidedown punch	technique droids applaud sufferers explosions born amount knees centers succeed

Another interesting report would be to see the composition of topics within the corpus. We calculate the "topic" of a document as the topic with the highest p(topic|doc) value for that document. We then display the number of documents across various topics as a histogram. Here is the code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

import numpy as np import pylab as pl f = open("/path/to/data/p_topic_doc.txt", 'rb') xvals = range(0, 50) tcounts = np.zeros((50)) for line in f: line = line.strip() if len(line) == 0 or line.startswith("#"): continue docid, probs = line.split("\t") plist = [float(p) for p in probs.split(",")] topic = plist.index(max(plist)) tcounts[topic] += 1 f.close() yvals = list(tcounts) print xvals print yvals fig = pl.figure() ax = pl.subplot(111) ax.bar(xvals, yvals) pl.ylabel("#-Documents") pl.xlabel("Topics") pl.show()

and here is the resulting histogram. As you can see, the distribution appears fairly uniform with a few popular topics. We could try to correlate these topics with the popular words in the topic to figure out what our corpus is all about.

Yet another application could be to think of LDA as a feature reduction strategy, converting the problem down to only 50 features (the number of topics) represented by the p(topic|doc) values..

Conclusion

Topic Modeling can be a powerful tool and provides interesting insights into your data. Mahout is one of the few packages that can do Topic Modeling at scale. However, using it was daunting because of poor/outdated documentation. Mahout hasn't yet reached the 1.0 release milestone, and there is already some work being done within the Mahout community to improve documentation, so hopefully it will all be ironed out by that time.