Apparently, I have learned absolutely nothing all semester, because what seemed like a very straightforward project proved to be completely beyond my abilities.

The overarching goal is to generate data for the visualization I’m making for Lisa Strausfeld and Christian Marc Schmidt’s Mainstreaming Information class. The following are some slides explaining the gist of the project, provisionally called Vocabulap (vocabulary + overlap; not a handsome coinage):

My specific goals for the A2Z midterm were as follows (with subsequent comments in all caps):

For A2Z midterm
===============
Prep
—-
* Remove all blank lines
DONE
* Remove all extra spaces
DONE
* Break all lines – DONE
* Rename all to number consecutively: A01, A02, . . . A10 (for old books); B01, B02, . . . B10 (for new books)
DONE

Compare major sets
——————
* Extract the text from between the body tags in each file. Dump it out as a new file with the extension body.txt in the folder ../body.
THIS IS HARDER THAN IT LOOKS (FOR ME, AT LEAST). EASIER TO JUST CUT THEM OFF BY HAND.
* Concatenate all the files in each set.
DID THIS FROM THE COMMAND LINE, USING CAT
* Make a list of unique words in each concatenated set, with the number of times the word appears.
CAN GET THE UNIQUE WORDS, BUT NOT THE COUNT.
* Strip out all words beginning with numerals.
DONE BY HAND
* Create the following lists:
– Words shared by both major sets, with frequency counts
– Words unique to set 1, with frequency counts
– words unique to set 2, with frequency counts
I APPARENTLY CANNOT DO ANY OF THIS.

Find unique words in each book
——————————
For each book:
* Concatenate all the files in that major set *except* the file for that book.
* Make a list of the unique words, with frequency counts, in
– the current book
– the set of all books except the current one
* Make three lists:
– Words shared by all books in the major set, with frequency counts
– Words that appear only in the current book, with frequency counts
– Words that appear only outside the current book, with frequency counts

Return lines surrounding specific words
—————————————
For each word in a given list:
* Get the line numbers on which it appears.
For each appearance,
* Print the line above
* Print the line with the word, replacing it with itself wrapped in span tags to apply color
* Print the line below

The most essential piece of code that I could not get working is the comparison doodad. It almost worked for, like, five seconds, but it was generating a huge file of every unique word times however many words were in the document, or something like that. When I tried to fix it, it completely stopped working. The offending code is as follows:

[java]
/* 1. Takes in a file name from the command line.
2. Makes a string array out of the hard-coded comparison file.
3. Imports the contents of the file whose name was passed in.
4. For each line of the input file (i.e., each word), changes it to
lowercase and checks to see if it’s contained in the comparison file.
5. If it’s not in the comparison file, checks to see if it’s in a hashset of
unique words.
6. If the word’s not in the hashset, add it.
7. Print the contents of the hashset.
*/

import java.util.ArrayList;
import java.util.HashSet;
import com.decontextualize.a2z.TextFilter;

public class CompareUnique extends TextFilter
{
public static void main(String[] args)
{
new CompareUnique().run();
} // end main

private String filename = “body/unique/allB_uci.txt”;
private HashSet uniqueWords = new HashSet();
private HashSet lowercaseWords = new HashSet();

// make a String array out of the contents of the comparison file
String[] checkAgainst = new TextFilter().collectLines(fromFile(filename));

public void eachLine(String word)
{
String wordLower = word.toLowerCase();
for (int i = 0; i < checkAgainst.length; i++)
{
if (checkAgainst[i] != null && checkAgainst[i].contains(wordLower))
{} // end if
else if (checkAgainst != null)
{
if (lowercaseWords != null && lowercaseWords.contains(wordLower))
{ } // end if
else if (lowercaseWords != null)
{
uniqueWords.add(wordLower);
lowercaseWords.add(wordLower);
} // end else
} // end else
} // end for
} // end eachLine

public void end()
{
for (String reallyunique: uniqueWords) {
println(reallyunique);
} // end for
} // end end

} // end class
[/java]

I know, it seems very simple, but you have no idea how long it took me to get this far.

So, basically, for the midterm I’ve got bupkis—just a big pile of text files, and a list of unique words for each.