I just created a new project for geo-coding addresses: http://github.com/jgsmith/ruby-fabulator-geo.

Combined with the ability to walk through the TEI documents housed on a Radiant site and execute XPath queries against them, we can extract address information and get the latitude and longitude from Yahoo! Maps (only works for US and Canadian addresses — a limitation in the Yahoo! Maps API).

If we encode an address using:

then we can access the information as child nodes of $a:

The string equivalent of $a will be the original address.

I just checked in some changes that show ‘with’ working, at least to some degree. More use cases will help flesh it out.

The ‘with’ keyword in expressions is used to add information to nodes without changing which nodes are being returned by an expression. This is useful for annotating data while passing the data on to another function for further processing. In the context of concordances, this means we can annotate the words and then pass the list of words on to a function that combines the lists into a larger list. This lets us break the concordance process up into smaller steps that can work on a particular line or page of a manuscript. We can attach information to each word as to which line or page it was found on and then retain that information as we combine concordances of pages into concordances of manuscripts.

This lets us do the following:

That should (hopefully) build the concordance data for a particular manuscript (given as the URL in $manuscript). Note that the c:concordance function only compiles a list of word frequencies for the given text. There’s not a lot of other magic going on.

I’m making good progress on the concordance front.  I can now do the following:

c:concordance(“some text”)/some/count

This will convert the string to a concordance object (implicitly compiling the concordance) and give the the frequency/count of the word “some”.

My goal now is to get the “with ./*/foo := bar” fragment working with an internal Ruby representation of a concordance.  This will allow me to annotate the words in a concordance.  The internal object already preserves annotations when combining concordances.

Once I have a good serialization format for the concordance, I will be able to persist the concordance in some form — perhaps RDF, but not necessarily so.  That combined with annotations of where a word appears will let me do searches of words:

c:concordance($doc)/foo/line

to give me the lines on which a word appears.

c:concordance($doc)/*[f:matches(node-name(.), “^f”]/line

to give me the lines on which a word appears beginning with the letter “f”.

At that point, the challenge will be to optimize these idioms so they don’t take forever to run.

At a basic level, a concordance is an annotated word frequency chart.  Presentations and corpus might differ across projects, but any project with a concordance manages tables of words tied to frequencies and other information.

My current work is to set up an environment where we can say something like the following.

f:sum( for $line in $doc/* return c:concordance($line) with ./*/line = $line )

That might look a bit daunting, but let’s unpack it.

The outer function is stating that we want to accumulate concordances to build an overall concordance.  The sum of two concordances is a single concordance with combined counts and accumulated annotations (e.g., combined references to lines).

The ‘for $line in $doc/*’ is a way of going through every line in a document (in this case, a transcription in the Donne project).  This will depend on how concorded documents are managed and how fine grained you want your location tracking.  The ‘return …’ part just says that we want to build a list of things based on what we’re looping over (in this case, the ‘$line’).

The ‘c:concordance($line)’ builds a concordance object with word frequencies for the given ‘$line’.  We add a reference to the line with the ‘with ./*/line = $line’ part, which runs the expression against the concordance, adding a reference to the line to every word in the concordance.  The children of a concordance are the words in the concordance, so we end up adding the reference to each word.  The reference gets carried through the f:sum resulting in one large concordance with word frequencies and annotations of every line containing a particular word.

I’m working on some functions that can be useful for a concordance.  Right now, that’s a function to give me the frequency of each word in a text.

I’ve successfully defined it as follows:

function_decl 'count-words', %{ f:histogram( f:split( f:normalize-space( f:keep( f:lower-case($1), ('lower') ) ), " " )/* ) }, { 'f' => FAB_NS }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

function_decl 'count-words', %{   f:histogram(     f:split(       f:normalize-space(         f:keep(           f:lower-case($1),           ('lower')         )       ),       " "     )/*   ) }, {   'f' => FAB_NS }

This allows functions to be written in terms of previously defined functions, using all of the expressiveness of the Fabulator expression language.

For example, if I defined the XML prefix ‘c’ to correspond to the namespace in which the ‘count-words’ function is defined, then I could run the following expression:

let $i := c:count-words("This allows functions to be written in terms of previously defined functions, using all of the expressiveness of the Fabulator expression language.")

1	let $i := c:count-words("This allows functions to be written in terms of previously defined functions, using all of the expressiveness of the Fabulator expression language.")

And then ‘$i/of’ would equal 3 because the word ‘of’ appears three times.

I created a github repository for the XML extensions to Fabulator.  This is the first step towards a set of general purpose functions for managing TEI documents.  With this and recent changes to the core Fabulator Radiant extension, we can browse TEI documents in the CMS and extract information from within the document.

These two functions move us much closer to being able to extract geographic information from documents for use in the Digital Concord project.

The Donne project has a special markup language for their transcriptions.  I’ve created a Radiant filter that can transform it into HTML for viewing.  This afternoon, I was able to change the parser in the filter from producing HTML to producing a data structure that I can manipulate in the Fabulator engine.

Continue Reading Donne transcriptions are getting closer

The Fabulator is a way to build interactive applications within Radiant.  I’m using it as the framework for building several DH projects this semester that work with transcriptions of textual artifacts.  I just pushed a set of changes that allow the fabulator applications to traverse the page hierarchy and access page content (for example, ‘radiant::/@title’ gives the title of the home page).  This means that we can manage TEI documents as pages in the CMS and still process them to extract information for a database, all within the CMS.