Image via Wikipedia

I don’t have any on-line documentation of the language and extension API yet, but I should in a month or two. June at the latest. I’m redesigning and rebuilding the TAMU College of Liberal Arts Digital Humanities Program website. The new site will go live around June.

The current core engine has three complementary systems: the XML framework for defining the parts of an application, the XML actions that can be run, and an expression language for manipulating the data tree in the application. The framework sets up the overall application, view, and transition contexts for the actions. The core actions manage control flow and data transformation. The expression language can be used to perform calculations, similar to XPath. The system is designed to act on application data similarly to how XSLT does when manipulating XML documents.

In addition to the core engine, I’m splitting the RDF manipulation out into its own extension and starting a workflow management extension. By summer, we should have the core engine and the two extensions fairly polished.

The beauty of this system is that I can add capabilities that any project can use, and add them in a way that lets me focus on the capability instead of all of the tasks that I would have to worry about in a general, standalone application that provided the same capability. For example, I’m planning on developing an extension to manage all of the data calculations needed to support a faceted browser. If I do it right, all I’ll have to worry about is providing the calculations. The core engine will be able to tie everything together easily. In addition, all the calculations should be generic enough that I can use them for something other than a faceted browser.

This engine allows you to build an interactive, database-backed application without having to do any of the repetitive basic work that traditionally goes into such an application. No worrying about filtering and constraining data from the browser (it’s trivial to specify the filters and constraints and not worry about how it’s done), no getting the right view in the right place in the code, no worrying about people jumping into the middle of a process, no worrying about what to do with data between requests when you need to collect more data before doing something, no worrying about managing a relational database schema. Everything just works with descriptions of what you want.

The next steps will be to look at several projects using this system and see where I’m repeating myself in those projects. Then, following the DRY principle, reduce the redundancy in the descriptions. For example, if I have the same RDF pattern over and over again for a particular object type (a person, for instance), then I should have somewhere to describe that pattern so I can just say that I want all of the people and not worry about what the pattern looks like. At the same time, I should be able to say that I want all of the people as well as some additional information without jumping through too many hoops or worrying about how the data is stored.

Academics seem to have an allergic reaction to projects like Wikipedia and Project Gutenberg, but they are missing the point of these projects and an opportunity as educators.

Before the Gutenberg Press, knowledge of the Bible was mediated by the Church. Your understanding of what God wanted was based on your faith in the Church and its priests. No one could afford to own their own illuminated edition of the Bible, much less have a need to read.

After the Press, the Bible was affordable for a lot of people. The Press lead to the Reformation because people could trust their own reading instead of having to put their faith in the Church. Printed books weren't as beautiful as the earlier illuminated editions on parchment, but the information content was the same and costs much less. Printing didn't preserve the form of the earlier work, but transformed it in a way that lead to wider and more flexible use. Today, books can be carried in a back pocket and read anywhere.

Electronic books are going to have the same impact once they figure out how they should work. The various gadgets like Amazon's Kimble will eventually go away. None of the current snake oil does for printed books what printed books did for hand copied editions. The places you can use an e-book are fewer than those in which you can use a printed book. Current electronic devices represent regression in reading.

However, electronic books do what print books can't do: they let you own a library at an affordable price. But electronic books aren't yet a revolution. They just put in electronic form what is in print.

The Press also opened up another realm to people with fewer means: authorship. By making books cheaper to print, new books could be printed with less initial investment. The Press lead to an explosion in writing.

The Internet has produced a similar explosion through blogs and websites. The barrier to entry today is the lowest it has ever been, regardless of the audience size. Anyone can be an author, and anything published can be read.

The Internet is today's library, and devices like the iPhone are the means of accessing that library.

The Internet also represents the Press to todays Church of scholarship. No longer does information require faith in the mediation of scholars. We can write and read our own encyclopedia directly.

Instead of fighting this revolution, scholars should be teaching critical thinking skills that let us sort through the information we find and decide for ourselves what is good and consistent with everything else we've seen and learned.

This seems to break down into a fairly simple set of responsibilities: entry, storage, retrieval, interpretation, and presentation. If I can define simple, open, standard APIs to interface between each stage, then each stage can be fairly independent and reusable by itself.

Artifact and annotation have slightly different needs, so we need to break those apart into artifact collection and knowledge base. There might some special cases, but I'm trying to push as much as possible any "special handling" into the back end.

Artifacts have a central document (image, text, video, etc.) that can be considered a canonical source. Some meta-data will be associated with it, such as the size of the document, the type, the collection it belongs to, perhaps some ACLs and a URL for public access. This meta-data is the minimal set needed to manage the artifact itself. Any information such as who created it, where it was published, etc., would be in an associated knowledge base.

Knowledge bases are collections of facts. These can be written as RDF triples (predicate, subject, object) with the subject being the resource described, the predicate indicating an aspect of the subject, and the object being the value associated with that aspect. If we want to add a little power to the knowledge base, we can add an inference engine, such as Prolog, and a set of rules.

An inference engine lets us ask questions such as "who are the descendants of A?" or "which papers lead to P through a citation chain?" or even "Are A and B related?" without having to write a lot of programming code.

The resources in the knowledge base will be exposed through a REST interface that will allow editing (with the proper permissions) as well as reading. This allows the greatest flexibility in using applications from web browsers to other non-end-user systems, such as other research projects.

So that knowledge bases and inference engines are available, but aren't open to abuse (inferencing can require a lot of computation), I'm planning on offering the results of running a particular query as an RSS or Atom feed. This is similar to the interface for listing all of the resources of a particular type in a model, but would be read-only. The actual query would be defined through some administrative interface that would associate a public URL with the query—similar to how Yahoo! Pipes exposes its pipes. This has the added advantage of disassociating the resource identifier from the resource definition for the query, making client applications easier to maintain.

If licensing works out (which we expect it to at the moment), I should have a code release in a month. It won't have ACLs, inferencing, or other "fancy" features, but it should be a proof-of-concept for the basic ideas.

I believe the way to do this is to introduce the idea of the project edition. This mirrors how reference works are produced in print media. Libraries and journals already have developed a system for working with such editions.

After an edition is printed, the publisher will sometimes produce an errata listing all of the errors corrected. Some of these corrections will be made for the next printing. Editions are usually produced when the changes are so large that the work is substantially different and shouldn't serve as a substitute for the earlier version.

Editions help researchers communicate on which version of a work they are building their own work. This helps other researchers verify their work and possibly reproduce it. If the foundation is constantly shifting, then reproducibility can't be reliably verified.

Trying to build an edition of a web application shows that such an application consists of two parts: the data on which the application is built, and the interface that allows people to explore that data.

To preserve reproducibility, we need to keep changes to a minimum and publish what they are. For the interface, this means that new features should be saved for new editions. New data can be added to the project without breaking reproducibility if the interface has a way to restrict its data at the user's discretion to what was available at an earlier date. We also need to publish when data has been added or changed.

Another property of printed material is that when a new edition comes out or the author moves to another institution, the older editions already in a library's collection remain. Once a work has been made available for others to build on, it remains available.

For digital works, continued availability requires a permanent home for a particular edition. This requires commitment from both the institution and the author. Institutions need to provide the infrastructure on which an edition could run, consisting of the server platforms and the URL namespaces, and authors need to grant the institution a license to run the edition in perpetuity with permission to make sufficient modifications to keep it running (just as Beowulf requires translation before the story is accessible to most English speakers today).

Such a license does not restrict future work of the author. Future editions of the application and data can run anywhere the author chooses. By granting such a license though, the author ensures that they will not be the reason that their work won't be found if referenced by another project.

In summary:

• Digital humanities projects need to follow a versioning model that produces discrete editions that can be referenced.
• These projects need to publish a list of any changes that are made to the published editions of the interface and data.
• Institutions need to make the infrastructure commitment that allows for long-term maintenance of these projects.
• Authors need to make the licensing commitment that allows for long-term maintenance of these projects.

This is definitely a pure computer science problem, but it has digital humanities applications.

A web search shows some work in this direction, but usually having people manually map elements between the two document sets to generate the XSLT.

Another thing that would come from this is a way to rank XML vocabularies based on their expressive range. If we have two sets of documents (A and B) based on two different XML vocabularies, then if an XSLT exists that maps A -> B, but no XSLT exists that maps B -> A, then the vocabulary for A could be seen as having a larger expressive range than that used for B. That would let us have a more solid foundation for saying that TEI is more expressive than Docbook (which I believe it is, but don't have good data to base that belief on at the moment).

I can manually create XSLTs to go from TEI to Docbook to HTML because I believe there's a loss of information from one format to the next (ignoring the pushing of that information into CSS at the final HTML stage) and because Docbook is a publishing vocabulary and HTML is, with CSS, a de facto typesetting vocabulary. The information isn't so much lost as transformed from semantic to presentation, with the person reading the resulting document adding back the semantic information based on the presentation. The semantic information though is removed from a readily computer-understood form: it's gone from a context-free to a context-dependent form.

At one point, someone posted some information on a new language for describing the game world. I don't remember too many details now (this was around 1989), but looking back, it could have been the first inklings of LPC and LPMuds.

Through the years, I've kept an avid interest in MUDs. I've played Everquest and World of Warcraft, but those don't have quite the richness that can be achieved in a MUD. Their graphics can be beautiful (WoW sometimes looks like a painting on my monitor), but there's a limit to how immersed I can get in the world.

I've also played Darker Realms, one of the early LPMuds (founded around 1991). It sometimes shows its age, but it's an interesting game historically, having been played by quite a few people who are now prominent in the Internet community.

One of the problems with a lot of MUDs though is that the world isn't as original as some people would like to think it is. Many are based on very familiar settings: Star Wars, Wheel of Time, Pern, etc. Worlds that are based on the creative work of other people not involved in the MUD. I wanted to avoid repeating the same mistake.

I have enjoyed playing World of Warcraft and will continue to play it as time allows, but I also want to create. I think MUDs could learn some from WoW. The game has a lot of hack-n-slash activity (almost everything involves killing something), but there's also a rich story line running through the game that comes out in the quests.

I am working on building a world then that emphasizes role playing and questing over killing things. There will be some killing, I'm sure, but I'm more interested in solving problems than just brute-forcing my way through. I'm also more interested in story.

The Middle of Nowhere is going to be a MUD that is built around the world I've been working on for my thesis. You probably won't find the characters from my thesis in the game, but you'll find the spirit of the world. The Muses will be there as well as the Cardinalities. The city center and the wild areas.

I'll be posting more on the game website.

Part of this equation is Google Gears which allows offline interaction with web applications. When I'm in the store and want to know if I have a particular book in my library, or what that wine was that I enjoyed the previous week but can't quite remember exactly which one it was, I want to have access to the information I need so I can make a well-informed decision. I want to be able to update my catalog at the point of purchase instead of trying to remember to do so later after I get home. Maybe I'm in another city, or traveling. The closer in time I can make all the tasks that go together, the more likely I am to do all of them. Managing data syncing and allowing client-side storage of data enables this.

The only thing we need to convince Apple to do is provide Google Gears in their iPhone installation of Safari. This would allow the iPhone to be like a small tablet computer that can store limited amounts of information, even if the network is unavailable.

Before then, though, we have an opportunity to play with the format for print publishing, even if we use public domain works. The iPhone is about the right size to feel like a small book and for the text to be a column of print—comfortable to the eyes and hands. This might be the opportunity to finally bring text to an electronic medium and not remove all of the comfortable aspects of reading dead-tree print.

var capture_file = function(file_element, url, callback){ var ls = google.gears.factory.create("beta.localserver", "1.0"); var rs = ls.openStore('uploads') || ls.createStore('uploads'); rs.enabled = true; rs.captureFile(file_element, url); new Ajax.Request(url, { method: 'get', onSuccess: function(transport) { callback(transport.responseText); } }); };

Given a file input element (DOM reference), the url at which to store the file in the local file store, and a callback to call with the content of the file, it will retrieve the content of the file from the local disk.

Those familiar with the GEP will recognize the basis of this project. However, this project is not a competitor with the GEP but a complement. The GEP is studying within the context of a university course the evolution of speculative fiction. While the GEP does characterize the stories it reads, it does so with particular questions designed with genre evolution in mind. More important than data collection, the GEP provides a good environment for students studying objective research methods in literary criticism. The GEP does not provide a lot of data that can be used outside the project. SF.OokOok does not try to answer any particular question but instead tries to collect as much data as possible that might be helpful in other research projects.

One of the problems with open data collection is the accuracy and trustworthiness of the data. SF.OokOok will try to correct some of this by borrowing some of the calibration ideas from CPR. In general, the more an individual agrees with others, the more weight their answers are given.

Finally, the question of what consititutes science fiction and fantasy is passed off to another project that is already trying to answer that question: the ISFDB. Only stories in the ISFDB will be surveyed. Until the new web API is put into production, we will periodically import the ISFDB data into our own database. Once the new web API is in production, we will consider feeding bibliography updates back to the ISFDB so people can survey fiction that might not yet be in the database.

A few companies (notably, Google) have found ways to make application development take advantage of parallelism without introducing huge complications for the application developer. Google's search engine and Apple's Automator are both examples of applications that can take advantage of parallelism under the covers. The trick is that both are designed around a functional programming paradigm, allowing easy analysis of the application to automatically identify operations that can be run in parallel.

The article correctly expects new languages to be domain specific. Parallelism is much easier to achieve if you only need to worry about the aspects of the application specific to the problem and ignore all of the necessary cruft that allows someone (or another application) to interact with the application. The I/O might not always be parallelizable, but the core aspects that make the application unique many times can be.

We need a new language and platform for web applications. This platform should make it easy for someone to write an application that can be spread across machines and take advantage of parallelism wherever possible. This should be one of the hundred-year languages that Paul Graham writes about.

The last two nights, I've woken with farely vivid dreams. Dreams aren't useful in their raw state. If you actually transcribe a dream, it won't make much sense because dream logic isn't sufficiently realistic. But dreams can provide interesting settings and plot pointers. That's what these two dreams have done.

I'm not spending too much time putting the second or third novels together right now. I'm half way through the first and want to finish it before I begin the second. I want to have enough preparation though by time I finish the first that I can begin the second without too much of a gap. Otherwise, I will get lazy and it will be difficult to pick up the pen again.

Now that I'm half way, I can consider podcasting a few chapters knowing that the final book won't change so much that the podcasts will be of a completely different creature than what might get published. I'll aim at getting a chapter every week or so, at least for the first few chapters. If I find a publisher and they are comfortable with a complete podcast (similar to what Cory Doctorow does), then I will put the complete book up on this site.

Academic progress seems to depend on a person's ability to contribute to society as measured by attributable work such as articles and monographs in areas that are interesting to society. These attributions depend on copyrights and patents, the keys to establishing and protecting intellectual property ownership. The various print journals and publishing houses act as a peer-reviewed intermediary. The academy establishes social interest in a project by the project's ability to attract financial support.

The Open Source Community has its roots in the academy, but has lived for a while in the wild among amateurs in the classic sense. Many people participate in open source because they enjoy doing so, though some participate because of their employment. The OSC is built on a stereotype of the academy: information is free and everyone is able to work on problems they find interesting. Just as professors enjoy working in a University or College with other smart people in their field, OSC members are attracted to projects with smart people. OSC members also value the openness with which they can develop and discuss projects.

The Open Source Community has not abandoned the notion of attribution and social relevancy. The openness of a project's development creates a trajectory along which the project travels. This trajectory is a measure of the talent behind the project and is apparent to most who are interested in the project. Misappropriating a particular release of a project captures only a single point on that trajectory. Because any particular release of a project does not bring with it any of the talent behind the project, `stealing' from the project is worth much less. Attribution in the OSC is much more than just the name beside the copyright or on the patent.

An Open Source project is socially relevant if it is widely used and has a strong community. There are no financial requirements for a project to be successful. Sourceforge.net hosts Open Source projects at no cost to the project not because any particular project is worth the cost, but because the OSC itself is socially relevant. People contribute their time to a project not necessarily because they get paid but because they enjoy the project and, if they make significant contributions, they can become a well known talent in the OSC.

The problem, then, is how to balance the requirements of the academy against the need to create an environment that is inviting to the open source community. By openly involving the open source community, DH can access a wide variety of talent which will be involved in various projects not because they are being paid to help, but because they love the project. At the same time, DH can maintain the attribution required for academic progress.

One academic field of study that is leading the way with technology is Physics with the development of the world wide web at CERN to aid in sharing documents to the electronic pre-print server (http://xxx.lanl.gov/). The Physics community can do this because it is small enough that its members know each other. Reputation is built much as it is in the Open Source Community: by personal experiences between members of the community. Formal peer review plays a secondary role. By reviewing the output of a physicist, another can see the pattern and tell if something new fits that pattern.

If any company represents the commercial potential of Digital Humanities, it is Google. They have been able to attract some of the top talent in the industry by providing a work environment that resembles the Open Source Community in many aspects. Some of the more interesting projects for DH have come from employees' `play time.' Google has brought a lot of smart people together under one roof, much as a University or College might do.

By looking to other academic fields and the Open Source Community, Digital Humanities can create a new environment encouraging rapid evolution of ideas without sacrificing the need for attribution, reputation, or social relevance.

Traditional programming languages divide a program into two parts: comments and code. The comments describe the goal of the code while the code describes how those goals are met. Bugs arise when the two don't agree. Most programmers would rather do one or the other, but not both. Doing both seems a waste of time because too much information is being duplicated in a way that can introduce mistakes. One of the primary goals of the Gestinanna project was to collapse the comments and code into a single descriptive document that would both describe the program and be the program.

What we need is a method of writing a program (the goal of the solution) that is both human and computer readable. The most widely understood form that meets this requirement is XML. By writing a program using an XML vocabulary, we create a document that can easily be transformed into something readable by people (e.g., HTML) and something that can easily be compiled into a procedure that acomplishes the goal (e.g., a Perl module). The resulting XML vocabulary will need to borrow heavily from functional programming in the way it approaches the problem of describing solutions. Functional programming thinks in terms of data transformation instead of sequential instructions. All computing is simply transforming data. Examples of functional languages include LISP, Haskell, and XSLT.

All applications basically take input data, apply some transformations, and provide output data. Some are fancier in how they do this, but everything can be broken down into that basic form. Desktop applications, web applications, servers. It doesn't matter what role the application plays, it still takes input, transforms it in some way, and provides output. Some applications even change the transformations based on past input. All applications are state machines.

Since all applications are state machines, we can define the root XML element for our programs: <state-machine/>. Of course, state machines have states. Our next element is then <state/>. We'll want to give it an identifier so we can refer to it, so we will add an id attribute. We should be able to add an id to a <state-machine/> as well so we can reference it from other <state-machine/>s.We want to be able to add data transformations to the state machine. Otherwise, the state machine doesn't really do anything. We can do this with a <transform/> element. Sometimes, we need to apply a transformation to an empty value to initialize a program. We need to specify when certain transformations should happen, so we will add the <init/> and <clean-up/> elements to act as special <transform/>s.

<state-machine xmlns:wf="urn:for-work-flows" xmlns:authz="urn:for-authorization"> <init match="/requests" select="wf:find('/example/requests', 'PENDING' | 'APPROVED' | 'DENIED', authz:actor())"> <value name="id" select="./id"/> <value name="state" select="./state"/> <value name="list" select="wf:context-params(.)/list/name"/> <value name="last_update" select="./last_update"/> </init> </state-machine>

This will initialize the state machine with a list of work flows based on the /example/requests work flow definition that are in the PENDING, APPROVED, or DENIED state and owned by the person logged in to the website. It will transform the list of work flows into a list of associations, one association for each work flow. Each association will have four key/value pairs: the id of the work flow, its state, the name of the requested list (e.g., an e-mail list), and the date the work flow was last updated. This list will be available as /requests in any subsequent processing (e.g., a Template Toolkit view might access the list as requests[]).

In an implementation, the contents of the <init/> can be lazily evaluated. For example, in Perl, the <init/> can be stored as an anonymous subroutine that is called when the value of /requests is actually needed. Because of this, we can think of the <init/> as being applied when we use a value that matches /requests. This leads to an expectation that arbitrary XPath-like expressions can be matched against. The <init/> thus provides a default value for any data element that matches the match attribute.

Continuing this line of thinking, the <init/> and similar container elements don't really contain scripts that are run, but sets of rules that are applied to various bits of data at the appropriate time. The <init/> and similar elements act as scopes for the data transformation rules. If we are applying data transformation rules instead of steps in a script, then we can make use of the parallelism that functional programming allows. The rules are independent of each other and have no side effects. We can compress the rules by looking for common phrases and computing them once when needed.

Next time, we'll look at how states in the state machine can be specified and how flow through the state machine can be managed.