I’ve added a Project Wonderful ad to the sidebar. I’m not doing this to make any kind of significant money. Most sites with my traffic might get a penny a day in advertising if they’re lucky. I’m doing an experiment to see how Project Wonderful works, both as a publisher and as an advertiser. Advertising will come later. I have a few projects I’m working on that I’ll advertise as they mature.

There are two main reasons I’m trying Project Wonderful: funds are usable, and the system is more community oriented than other advertising networks that I’ve looked at. 

I can spend any money in my account on advertising, so anything I earn as a publisher will contribute to any advertising I do. I have between $3 and $4 locked away at Google because I ran adwords a few years ago but never earned enough to get a payout. Now that they require $100 minimum before handing any money over, there’s not much point in running adwords. I don’t get enough traffic to get to $100 before they bump it up again.

Project Wonderful came out of the webcomics world. The list of publishers strikes me as fringe, which suites me rather well. Science fiction. Gay. Liberal. Those aren’t descriptions of the Project Wonderful community, but I think anyone who identifies with those labels will find themselves among friendly company in that community, as I do. Liberal might not be fringe for many, but I grew up in Texas and spent quite a few years at Texas A&M University. Liberal is a fighting word at TAMU, at least among many undergraduates. By the way, Aggiecon 43 is coming up in a few weeks. Check it out if you’re in the area. By adding a Project Wonderful ad box to my site, I can expose you, my reader, to the Project Wonderful community. I’ll keep an eye on the ads and make sure they aren’t too far out there. They should be safe for work and not use Flash or animation. I don’t want them to distract you too much.

Thinking about projects, I’ll mention a couple that I’m working on that aren’t digital humanities: a novel and a tool to help self publishers. There’s also Second Contract, a mud based on the world in my first novel, but I’m putting that under digital humanities for now.

I’ll add a progress meter to the sidebar in a couple weeks to track my next novel. I still need to come up with a working title. I’m shooting for a July 1st finish for drafting the novel. So far, it’s turning into a thriller with some supernatural-like elements. Think vampire or zombie meets natural disaster. I’ll post a few times on my thought processes as I go along.

My second project is a website to help people typeset their book manuscripts. I’ve already written a post here about how to use LaTeX to create the interior of your print-on-demand book. Most people aren’t going to want to learn LaTeX to do their own typesetting. It’s almost worth the $100 it might cost to have someone else take your Word document and return a well-formatted PDF document. I’m going to try and take a lot of the mystery out of producing well-formatted PDFs by yourself without requiring that you learn LaTeX or the UNIX command line. I’ll post on this also as I get it ready for testing.

If you haven’t been reading A Newbie’s Guide to Publishing, you should. As with any blog, read it with a critical mind, but Konrath does address a lot of good points about publishing and the effect that e-books are having on the industry.

I’ve been reading The Innovator’s Dilemma recently. I’m about two-thirds of the way through, but I’m feeling resonance with literature as well as the academic world of digital humanities. For this blog post, I want to address how I see it playing in the publishing world. If you haven’t yet, read “At Home With the Sixes,” a post on Konrath’s blog. He covers some of the same ideas, but in a humorous story.

The basic premise in The Innovator’s Dilemma is that as companies mature and reacher for the higher end market, they find the lower end market not worth the trouble. If a company wants to grow at 5% per year and they have annual revenue of $1 million, they only need to find $50,000 in new business. But a company that is bringing in several billion dollars a year needs to find multiple millions of dollars in new business to sustain the same growth. The small stuff doesn’t interest them when they can focus on the larger, more lucrative markets. The problem is that this is entirely rational. To do otherwise would be considered crazy. You always want to maximize your income.

What happens is that smaller business which don’t need to bring in as much revenue to show robust growth will start addressing the markets that are too small for the larger companies. These markets usually have low margins, but they also don’t have as much overhead as the higher end markets.

Think about the difference between a cheap car and a Lexus. You don’t expect much service when you buy the cheap car. You just want wheels to get to work and to the store. But when you buy a Lexus, you expect the dealer to care about you as an individual. You want quality treatment. You pay for it. The margin on a Lexus is much higher than on the cheaper car because the dealer has to make enough money to cover the extra services it provides in connection with the purchase.

For this reason, the dealer won’t try to sell you the cheap car, because there’s not much money to be made. Instead, they want you to buy the expensive car. Even though the services they provide cost them money, they make up for it in in their profit margin.

Some else who is just starting out in the car business can afford to pass up selling the Lexus and focus on the cheap cars because they don’t need to make a lot of profit to show growth. The large dealer is happy to let them have the business because it lets them focus on the higher margin luxury models.

The problem is that the cheap cars will get better over time and eventually be as good as the luxury cars, but because the business built around them has become used to a lower profit margin, they can sell the equivalent car for less. As soon as buyers can get the same product in several different ways, they tend to go with the cheaper one. As soon as this happens, the larger luxury dealer loses his business, even if he had record profits just the year before.

In The Innovator’s Dilemma, this story is told over and over again in the disk drive industry, earth mover industry, and several other unrelated industries to show that it’s a general pattern not isolated to any. The same can happen in publishing.

Consider publishing. An editor has to look through a pile of manuscripts hoping that one will come through that might be publishable. If it does, then there are contracts to negotiate, editing to be done, typesetting, cover design, marketing. There’s a lot there that costs a lot of money. A publishing company that is willing to do all of this for an author is the Lexus dealer of publishing, and they focus on books that they feel will provide them with enough of a profit margin to pay for all of the services. They don’t want the books that might be publishable or the ones that will require some work to make into a bestseller. If they had their way, they would only do bestsellers.

A publisher doesn’t negotiate which services they will provide. They don’t ask the author if they know a cover artist who can provide the artwork, or if they can manage the editing, or the layout. They assume all of the publication management and then negotiate the final price, which is usually a large percentage of the value of the work. They assume that they provide about 80%-90% of the value of a book. Next time you’re in a book store, look at a book and see if that 80% of the value is really why you’re buying the book, or if you’re buying it for what the author provided.

Part of this is because of the processes that are in place in a publishing house. Individual editors might know there’s a problem, but they still have a process they have to go through. That’s the hardest thing to change. If you look at what some of the publishers are doing now with their spin off self-publishing imprints, you can see them trying to change this, but it’s going to be a while before they find the right model.

Someone who wants to focus only on writing might want this full service model, just as someone might want a Lexus. In many circles, it’s not the quality of the book that marks someone as a professional, but the fact that they went through a publisher and gave up most of their claim on the material, so they might want to go through a publisher so they can gain status in a community. I’ll cover this angle a bit more when I discuss The Innovator’s Dilemma in the context of academics. Going through a publisher is not bad or evil. Consider it a luxury.

Publishers are perfectly happy to have low-margin books be self-published. As publishers grow, and they have through the mergers of the last few decades, they have to attract more and more business to show the same percentage growth year over year. This means they need more bestsellers, fewer mid-listers, and far fewer experiments. The small publishing houses tend to pick up the lower profit-margin books, but they too are slowly going up market.

The main room for self-publishing is at the bottom of the market where the publishing businesses aren’t really interested in working. This doesn’t mean that a book is bad because it is self-published. This just means that the book hasn’t been produced as a luxury good. Instead of the author providing 10-20% of the value of the book, the author is providing all of the value. Instead of getting mass-produced product from the local megamart, you’re getting handcrafted works from artisans.

The reason mass production works is because buyers can expect a consistent quality and a reasonable price. The problem is that though a cheaper method of production might not start with the same consistency in quality, it will eventually meet it. When that happens, the cheaper method will win.

Self-published books might not have the consistency that big publishing houses can provide, but they will eventually catch up as the community figures out how to identify good, well-written self-published work. Once that happens and buyers can find consistent quality self-published work, the larger businesses will fail. No one will see it coming before it happens, and it will happen almost over night.

Image via Wikipedia

I began the month intending to write 50,000 words. I got a bit past 5,000 and then got sidetracked by another project. I shouldn’t be surprised. This is how it’s been in the past.

From the title, you might guess that the distraction was a game, and you’d be right. But I didn’t get sidetracked playing a game. Instead, I’ve been sidetracked creating a game. It’s one I’ve been working on now and again over the last few years, but I’m diving back in using my research day, evenings, and weekends to get enough stuff together that I can show it off at THATCamp Games near the end of January.

I might not be getting the 50,000 words done, but I am doing a lot of creating. At least I’m keeping within the spirit of the month.

I’m using the MUD as a way to explore interactive fiction. I’m hoping to teach a course eventually in which the students will spend the semester exploring how games such as MUDs work by playing the game and creating an area. Graphical games are a bit complex in all the wrong ways to get everything done in a single semester. They also don’t offer as rich an experience as text-based games.

If you’ve ever played World of Warcraft, you’ll notice how constrained you are when interacting with the world. The design hides it well because almost everything you can do with a mouse and a click will result in something happening, but everything is constrained. At every point in which you interact with the world, everything is scripted. You might have a few choices, but they are very few.

A game that uses text as its medium of interaction can (but doesn’t have to) offer a much wider range of possibilities. While the number of commands in any game is finite, they can be sufficiently complex and regular that it’s harder to hit against the boundaries.

The first system I encountered in building a game was LPC and the world of LPMuds. These tend to consist of a driver that manages the system resources (the player connections, coordinating execution of actions, managing inventory), a mudlib that provides the basic building blocks for the game (basic rooms, objects, NPCs), and the game itself. The LPC driver is event driven in the same sense as NodeJS.

I’m building the game using the Dead Souls mudlib because it provides a way for you to create new things in the game without having to do any programming and also allows you to see all of the game logic at the appropriate level of abstraction. I’m incorporating some ideas from the Discworld mudlib (e.g., skill trees, continuous media) and making some other changes that I hope will make for a more interesting game.

Feel free to follow the game development. I hope to be open for a small number of alpha testers at the end of January. If you’d like to try it out, drop me an email at jgsmith at gmail.spam.com (taking out the spam).

Image by ? Crystal Writer ? via Flickr

Every November, I mean to buckle down and write 50,000 words. Every November, something comes up that keeps me from doing it. Last year, I taught an introductory course to creative writing at Texas A&M University. The year before, I probably got too busy with work. This year, I’m going to make it happen! I don’t have any trips planned except for Thanksgiving. I don’t have any activities after work or on the weekend that take up a large amount of time. Nothing is standing in my way.

For those who haven’t heard of NaNoWriMo, hop over to the website and take a look. I’m raising money to help the nonprofit that runs it build communities in classrooms, coffee shops, libraries, and living rooms all over the world and help the inspiration flow for me and thousands of my fellow novelists. More importantly, your contribution will help The Office of Letters and Light build a more engaged and inspiring world.

For the rest of this post, I want to explore why NaNoWriMo works and touch a bit on what it could mean for digital humanities. Today is my research day, after all, so I need to tie this in with my work somehow.

For my creative writing course last year, I assigned Self-Editing for Fiction Writers: How to Edit Yourself into Print as the textbook. It’s cheap, practical, and available in a variety of formats. It’s a book that the students should find useful throughout their careers. The last two chapters are “Sophistication” and “Voice,” or, as I titled the weeks that we covered them, “Editing without writing” and “Writing without editing.” These two are the culmination of everything else a person learns in a creating writing course. Another book that is helpful in trying to understand why voice comes from turning off editing is Blink: The Power of Thinking Without Thinking.

The idea is that there are two processes going on in our head: the creation and rejection of possibilities. What isn’t suggested by our subconcious can’t be rejected, and what isn’t rejected is what we end up doing. Experts have exercised both of these enough (usually over a 10,000 hour period) that the part of their brain that does the creating has learned what is likely not to be rejected. This is why experts shouldn’t think too much about decisions they make in their area of expertise. It’s why we have to not think too much about what we’re writing if we want to bring out our own voice.

Our internal editor is what makes blogging difficult for someone who hasn’t blogged before, especially if they are used to publishing only finely honed text. It’s the source of writer’s block, of the blank page that remains blank because the right word isn’t popping into mind.

But turning off that part of our brain that says, “No,” is the hardest thing to do. It’s the part of our brain that provides judgement. It’s what keeps us from jumping out the window when a something in the back of our head wants to feel the air rush by, and our stomach churns at the thought. Our survival depends on it.

A good way to turn off the editor is to force a crisis. Judgement takes time, and when that time isn’t available, judgement can’t happen. The subconscious has to take over. It’s easy to watch a sports game and decide that a player made a bad decision, but the mistake might be in thinking that a decision was made at all.

A baseball player can’t know where the pitcher has thrown the ball until after he’s swung the bat. It’s all of the practice that has made his subconscious learn how to play the game. It takes a little under half a second for the ball to go from the pitcher to the home plate. It takes about 200ms for the brain to realize what it’s seeing, so by time the player realizes the pitch has been thrown (much less where the ball actually is), it’s almost half way to the plate. There’s just enough time to swing the bat.

The trick with NaNoWriMo is that people who are going to school or have jobs or other things keeping them busy in life don’t have time to write fifty thousand words if they spend time choosing each word carefully. By throwing the writer into crises and forcing the fingers to move on they keyboard (or across the paper), the editor has to be turned off.

A good typist can type over fifty words per minute. That’s a thousand minutes to type fifty thousand words. That’s sixteen hours and forty minutes, or about thirty three minutes each day for thirty days. Given that my bus ride is about twenty minutes each way, that seems perfectly doable. Of course, that depends on just typing and not doing any editing, which is the whole point of NaNoWriMo.

Digital Humanities and “More Hack, Less Yack”

What does this have to do with digital humanities? “More hack, less yack” has been floating around for a few years now. The humanities are good at contemplating and talking about what other people produce. Literary theorists study literature created by other people. Historians study events created by other people. This tendency to produce secondary sources instead of primary sources is baked into the profession.

Secondary sources are still useful. As a writer, I find Freud and Foucault useful, but only to the extent that they help me write better fiction. Reading what historians have discovered is useful because it helps us learn from past mistakes and successes. But sometimes, a pipe is just a pipe. Sometimes, it takes some practice doing a thing to understand how a thing is done or why certain decisions are made.

Thus, “more hack and less yack.”

I’ve been at a THATCamp before where we were talking about the need for standards and had someone raise the question of why we should try to have standards at all. A question that comes from a lack of experience dealing with the lack of standards for certain areas of computing in the humanities. Any attempt to explain our experience was met with resistance. Even the idea that we were using English as a standard of communication within the room was met with derision.

The need for “more hack and less yack.”

We can watch a baseball game all day and think we understand what’s going on, but we can’t become experts in the game until we understand what’s going on inside each player’s mind. Any criticism we make, that a pitcher should have thrown differently, or a batter swung differently, comes from a different reality than the ones the players are in. All of the studying done in neuroscience and psychology are just attempts to put us in that place by proxy, by deconstructing everything into the basic ways in which our brains process information, resulting in the fact that there just isn’t enough time left in a pitch for a player to contemplate what they want to do once their biology has been accounted for.

This is probably why so many sports announcers are former players. They’ve done a lot of hacking. Now they can do some yacking.

How to Increase DH Hacking

I’d like to try an experiment some time. We have workshops in creative writing programs in which students write material and pass it around to the other students and professor for comment. Why not have a digital humanities workshop sometime in which people try to produce original digital work and pass it around for comment from other participants? Perhaps something after the Clarion model.

The nice thing about this is that it would force the participants to spend time building instead of constantly talking about what they want someone to build. Just as Clarion expects people to know how to write paragraphs and use a typewriter or word processor, participants in a DH version would need to know something about the technologies they might want to use to compose their project. The discussions would focus on higher-level criticism of the work. Perhaps the narrative structure or the scholarly discourse enabled by the design.

The week of DHSI isn’t quite long enough to have the same effect that a Clarion-style workshop would have, though I’m not knocking it in any way. Six weeks of intense creation allows the participant to go through several iterations of a project. It’s almost a capstone experience after several years building up skills.

The problem is that we don’t really have anything computational that is as easy to use as words and sentences when building a narrative. While the writer has to produce sensible sentences, there can be problems that don’t keep someone from reading what they’ve written. Programming requires enough correctness that someone can run the program and see what it’s doing. Since the number of bugs per line of code is fairly constant across languages, we need very high level languages that let people create projects with little enough programming that there are few bugs if the program reads well.

Until then, perhaps we can do a NaNoWriMo style event. Take a month and produce a project regardless of bugs. Just write code. Put things together. Create some pages and interfaces. See what happens.

At the end of November, I’ll have a novel that I won’t want anyone to read because it will be absolutely awful, but I’ll have a novel. At the end of the DH equivalent, we might have a program that hardly works or a site that is so ugly it’s obvious we can’t do web design, but we’ll have a project. We’ll have something we can edit and slowly modify until it communicates what we want it to communicate.

Last week, we explored the Poisson distribution as a possible distribution of sentence lengths. If you look at the figure for Hunter Crackdown, the Poisson seems reasonable, but it breaks down when looking at other works. In this post, I’d like to go back and try to derive a distribution that has the same qualitative features as the distributions we saw for each of the works. Then, I want to discuss a bit what we might want to do next.

Sentence Lengths for Twenty-One Words and Five Sentences

Suppose we have a number of sentences, labeled by index i, each containing a total of n_i words chosen from a dictionary of d words with possible duplication of words. That is, sentence i has n_i slots for words and those slots are filled by selecting words from the dictionary without requiring that a word be selected at most once from the dictionary for each sentence.

We can also apply a few more constraints on n_i. It must be greater than zero. Sentences must have at least one word. We’ll worry later about varying the number of sentences in a work. Additionally, the sum of all of the sentence lengths must equal the number of words in the text.

How many ways can we combine words to form sentences if we have n words in a work and s sentences? The visual we want to work with is easy if we represent words as circles (o) and sentence boundaries as corsses (×):

oo×oooo×ooo×ooooooo×ooooo

This says that we have five sentences with lengths 2, 4, 3, 7, and 5. That’s 21 circles and 4 crosses. There are one fewer crosses than sentences because we’re indicating the boundary between sentences. We could put crosses at the beginning and end of the sequence (resulting in one more cross than sentences), but because they would be invariant, we don’t need to consider them.

Now, we have n+s-1 positions into which we place n circles and s-1 crosses. If we place the crosses first, then we can fill in the rest of the positions with circles, so we just need to count how many different ways we can place the crosses.

Note that regardless of how we place the crosses, the average number of words per sentence won’t change. We’re not changing the number of words or the number of sentences.

What we want to do now is see how likely a particular distribution of sentence lengths might be. This will be a bit tricker. In the following sections, we will assume that the fifth sentence has all of the remaining words. Feel free to skip down to the Discussion section if you don’t want to see all of the ways we count words and sentences.

Four Sentences of One Length

Let’s consider the case where we have all but one sentence be one word long. The remaining sentence will have all of the other words. In circles and crosses, this might look like the following:

o×o×o×o×ooooooooooooooooo

This is the same number of words and sentences as before, but now we have four sentences with one word and one sentence with seventeen words. How many ways can we order these sentences, assuming we only distinguish sentences by their sentence length?

We know we can’t split the long sentence, so the only option is to place the long sentence between the one-word sentences. This changes the problem to the following: we have five slots (sentences) filled with four circles (the one word sentences) and one cross (the seventeen word sentence). This is just choosing one out of five, of which there are five different ways. If we’re keeping count of how many ways we can get one-word sentences, then we now have twenty (four sentences in five permutations). We also have five ways of having a seventeen word sentence.

The same approach will give us five options of arranging four sentences of equal length with one sentence having the remainder of the words. If we have four sentences of two words each, we have a fifth sentence with thirteen words. Three and nine. Four and five. Five and one.

Three Sentences of One Length Plus One Sentence of Another Length

Now let’s consider the case where we have three sentences of one word each, one sentence of two words, and one sentence with sixteen words. Still twenty one words and five sentences. If we place the two-word sentence first, we have five different choices. Once we place it, we have four choices left for the sixteen-word sentence. This gives us twenty different ways of arranging five sentences that consist of one two-word sentence, one sixteen-word sentence, and three one-word sentences.

Again, if we have three sentences with a particular length, another sentence with some other length, and a fifth sentence with the remaining words, we have the following combinations (e.g., 1, 2, and 16 indicate three sentences of length one, one sentence of length two, and one sentence of length sixteen):

Combinations such as (3,3,9) aren’t allowed because that’s four sentences with length three and one of length nine, but we’ve already counted that. Likewise, (2,6,9) and (2,9,6) are the same: three sentences of length two, one of length six, and one of length nine. We don’t want to double count them. We’ve already accounted for swapping positions.

We also have three sets that aren’t in the above table: three sentences with one length and the remaining words split evenly between the remaining two sentences: (1,9), (3,6), and (5,3)

Now we can add some more counts to our table. For each of the rows in the above table, we have twenty sets of sentences, resulting in sixty one-word sentences, twenty two-word sentences, and twenty sixteen-word sentences for the first triple. 

If we count up everything, we get 11,025 words in 2,625 sentences, which is 4.2 words per sentence.

Two Sentences of One Length and Two Sentences of Another Length

Now, we consider two sentences of length a and two sentences of length b. The fifth sentence must be of length 21-2a-2b. We also must have a not be the same as b. This gives us five-choose-two choices for the first sentence length and three-choose-one for the fifth sentence for thirty different possibilities. I’ll leave the generation of the table of possible sentence lengths as an exercise. The resulting counts of sentence lengths increase to the following:

Again, we can check our work and find that there are 22,995 words spread across 5,475 sentences for an average sentence length of 4.2.

Two Sentences of One Length, One Sentence Each of Two Other Lengths

Next, we consider two sentences of length a, one sentence of length b, and one sentence of length c. The fifth sentence must be of length 21-2a-b-c. We also must have a, b, and c all different. This gives us five-choose-two choices for the first sentence length and three-choose-one for the third sentence, and two-choose-one for the fourth sentence for sixty different possibilities. I’ll leave the generation of the table of possible sentence lengths as another exercise. The resulting counts of sentence lengths increase to the following:

Now we have 78,435 words in 18,675 sentences for 4.2 words per sentence on average.

Four Sentences All of Different Lengths

Finally, we have the situation in which every single sentence has a different length. Each combination of lengths will have one hundred twenty combinations: the first sentence has five possibilities, the second has four possibilities, etc. The list is short:

The resulting final counts for each sentence length:

If we count the total words (96,075) and divide by the total number of sentences (22,875), we get 4.2, which is what we expect (21 words / 5 sentences).

Discussion

We can plot the total number of sentences for each length as well as the number of words contained in sentences with each length. The plot of the number of words contained in sentences of particular lengths looks qualitatively like the distributions we saw last week when we counted the sentence lengths in a number of texts. The problem for now is that I’m not able to connect the two: one is counting the number of words assigned to sentences of particular lengths while the other is counting the number of sentences with a particular length.

Number of sentences with the given word length

Number of words contained in sentences with the given length

I’m not too worried about the counts for three, ten, and eleven word sentences. There are several “edge effects” that come about because of the short length of the text. The result of the small length is that we don’t have as much freedom for certain sentence lengths. As the length of the text increases, these effects should be less visible.

The important thing in this analysis is that we could have replaced the terms “word” and “sentence” with “sentence” and “paragraph”, respectively, and come to the same results (i.e., a text with twenty one sentences divided into five paragraphs). Likewise with “section” and “chapter” (a text with twenty one sections divided into five chapters), or “paragraph” and “chapter” (a text with twenty one paragraphs divided into five chapters).

One way to interpret this is that textual statistics may be invariant with respect to scale. Regardless of the scale at which we examine the text, we expect to see similar statistics: be it the word/sentence, the sentence/paragraph, or the paragraph/chapter.

This is a bit naïve because we may need to account for the information content. Not all words go together in a sentence of a given length. Longer sentences can carry more information, but sentences that are too long may not be readable. This tendency to a middle ground for sentence length combined with the counting we did today may result in a model that gives us something that looks like the results we were getting last week.

I had hoped that we would be a lot further along in this exploration by now, but counting can be difficult to get right. I’m still not confident that I haven’t made any mistakes, but everything seems fairly consistent so far.

I am happy though that there is reason to hope that text statistics might be self-similar over a range of scales. This is important for some of the other analysis I want to do eventually.

Open Questions

There are a few questions that come out of the combinatorics that need to be addressed before I’m comfortable.

The number of combinations we ended up counting have 22,875 sentences, but we should see 24,225. Essentially, take the words in the sentence and force the first word to be in the first sentence. Then select four other words from the remaining twenty to be the first words of sentences. The result is twenty choose four, or 4,845 sets of five sentences for 24,225 sentences total. I may need to redo the analysis with this in mind. It may simplify things quite a bit. I tend to take the long route on problems like this anyway.

The other question is how to generalize this to an arbitrary number of sentences for a text with an arbitrary number of words. I need the generalization if I want to test it against any texts.

Thursdays are my research days. I have a couple things cooking away that I’m not quite ready to write about yet, but I want to take a little time today to explore something that I plan on doing a lot more once my cooking is done.

I’m interested in studying narrative as a dynamic system. That is, there are several variables at play that determine the direction of a narrative. There are plot dynamics, character dynamics, and thematics that an author plays with to construct the story. They all interact in complex ways. A particular plot might require certain type of characters. A particular character might not fit certain types of plots. Some plots and characters don’t illustrate well certain themes. The author has to select the right plots, characters, and themes (and write well) for the reader to enjoy the story. 

Usually, when we want to study a dynamic system, we look at the equations that govern its motion, energy, or some other measurable quantity. But most of the interesting things that come out of the humanities don’t have such equations. We can’t make precise predictions about how a story will be written, or a piece of music will be composed, or what the stock market will do. That doesn’t mean that they are random or non-deterministic. We know that there are patterns. When we listen to a piece of music or read a book, we develop expectations about what will happen next. We have innate predictive capabilities that provide our expectations.

When we don’t know what the underlying determining system might be, we have a box of tools that we can use to find out about the system. We can develop models that can help us know what might happen next, or know how many independent variables should be in the model, or what kind of distribution of behaviors we might expect.

A classic study in this area is the 1984 paper by Robert Shaw, The Dripping Faucet as a Model Chaotic System, in which the entire dynamic of the dripping faucet is shown to be recoverable by measuring the time between drips.

How does this help us with narrative? If we know what human written narratives look like, then we know what kinds of computational algorithms can produce narratives that look like they are written by people.

With the dripping faucet, for example, we know that there are three, and only three, independent variables that work together to govern the time between drips. So we know that whatever relationships determine the behavior of the drips, they must only use three independent variables. We can rule out any systems of equations that have four independent variables or two independent variables.

With this in mind, I’m embarking on a quest to gather statistics on fiction narrative. What characteristics should computer generated text have if we want to trick people into thinking it was written by a person, even if poorly written or not containing new ideas?

The first, easiest, and probably most obvious thing to do is look at the distribution of sentence lengths. We don’t expect sentences to have less than one word in them or more words than there are in the text. We don’t expect a uniform distribution of lengths because we tend to use different sentence lengths for different effects. Short, punchy sentences for quick action. Long, languid, meandering sentences for thoughts that need some time to percolate.

An example of a poisson distribution.

This sounds a lot like a poisson distribution. The only parameter needed to draw the distribution is the mean, µ. The equation is:

f(k, µ) = µ^ke^-µ / k!

This equation is for a probability distribution, not the actual number of sentences in the text with the given number of words, so we will need to scale it so the maximum matches up with whatever maximum we find for a text.

The next step is to start looking at sentence lengths and see if we get anything that looks like this.

Hacker Crackdown

The first text we can look at is Hacker Crackdown, by Bruce Sterling. The text is available from Project Gutenberg. It has about 101,716 words, 6201 sentences, and averages 16.40316 words per sentence. The maximum sentence length is 213 words. These numbers are approximate because we use Perl to discard the header and license sections from the Project Gutenberg file and then run it through the Lingua::EN::Sentence Perl module to extract the sentences.

Distribution of sentence lengths for Hacker Crackdown

The resulting plot does resemble a poisson distribution, but it’s a bit tighter than what we might expect. We obviously need to scale the k parameter (the horizontal axis). The problem we run into is that k is supposed to be an integer. If we scale it, we lose that fundamental requirement for the interpretation of the poisson distribution.

This gives us a distribution with two parameters and a factor to transform the numbers from a probability to the number of sentences with those words:

f(k, µ, A, B) = Aµ^Bke^-µ / (Bk)!

I don’t know what A or B should be for the text. I need to spend a bit more time with Mathematica to figure that out. But the plot is qualitatively in the realm of the poisson distribution. I also need to look at other distributions that share some of the same characteristics.

More Texts

Let’s look at some more texts just in case Bruce Sterling is a fluke. The following graphs show the distributions of sentence lengths.

Distribution of sentence lengths for Catharine Furze

Distribution of sentence lengths for The Son of Tarzan

Distribution of sentence lengths for A Discourse on Method

Distribution of sentence lengths for Our Friend John Burroughs

That’s only four more texts, and the Descarte text looks odd, but this looks like a viable route to go down for a bit. The Descarte text is odd only because the maximum number of sentences with a particular word length is much lower than the other texts. Based on the following table, it seems that the problem with Descarte may be the low number of words relative to the other texts.

 Next Steps

Besides looking at other distributions and calculating distributions for more texts, we may want to explore some other tools at our disposal, like dimensionality analysis. In a future post, I’ll discuss what we might do to figure out how many independent things are going on to construct a text.

What are some of the tools you’ve used to deconstruct text and see the underlying structures?

Update: Thanks to Travis Brown for pointing out that I could just as well normalize the histogram from the texts. I’ll be doing such normalization going forward when we’re dealing with anything resembling a probability.

Cover: Of Fish and Swimming Swords

I bit the bullet and pushed my novel out to the Kindle, Nook, and Smashwords stores. Feel free to download a copy and write a review! It’s only $2.99, but if you want it for less, comment here sometime in the next week (before August 4th, 2011) with a link to your blog and I’ll send you a coupon for Smashwords. The only thing I’ll ask in return is that you write a substantive and constructive review on your blog.

It will be two or three weeks before the novel is available in the iBookstore or other stores fed by Smashwords.  I’ll post when I see the novel show up in these other stores.

From the blurb:

In a world built around fours, where the trinity of the United States government has been replaced by the Cardinalities of the Muses, Barbara finds a hidden symbol in yet another investigation and suspects foul play. But who can she trust? As she and her family dive into the mystery and horror, they find that they are battling forces that reach to the very core of the world they believe in.

Follow the family of four as they unravel the mystery and discover more about each other.

I used a fairly traditional narrative style that begins as a closed room murder mystery, but I quickly dive into a mix of horror and science fiction exploring conspiracy theory and artificial intelligence. The idea is that a story is a conspiracy theory on the part of the author and reader: the story isn’t in the words on the page, but in the connections the reader makes “between the lines” when reading. Part of the mystery is for the reader to figure out.  I don’t give a neat ending with everything tied with a bow: all of the clues are in the novel, but I don’t have an explicit summary.

I created a world centered on fours: four cardinalities with four colors (which you can see on the cover), four directions, four elements, four symbols. Families are built around four adults instead of our familiar two. There are other fours hidden in the novel that I’ll leave for you to discover.

 Enjoy! And let me know what you think.

The Digital Humanities Summer Institute (DHSI) is in a week.  I’ll be teaching a course on data discovery, management, and presentation using a platform I’ve been developing for the last couple years.  This will be the first time other people will try to use the platform to build a project.  I’ve been writing the workbook for the week-long course and I think we can do it.

For those who aren’t familiar with what I call the Fabulator, I’ve developed a compute engine as an extension to Radiant, an open source content management system.  The goal is to provide a platform for dynamic, data-driven digital humanities project sites that fill the role of the traditional monograph.  These sites make a scholarly argument using interactive web applications instead of static text.  The problem is that libraries don’t want to touch these projects.  No one wants to provide the long-term maintenance required to keep a web application running as the underlying languages and systems change.

The Fabulator system takes a different approach.  Instead of building a project from a set of applications and content (e.g., text, images, or videos), it makes the application side of the project equal with the content.  Now, everything is content and the framework is the player.  This insulates the application side from the vagaries of the operating system and programming languages.

Ah, you say, but there’s still a programming language in there somewhere!  And yes, there is.  But there are a few things about the language that make it workable.  Like PDF and Postscript, it’s domain specific and versioned.  If the language undergoes significant changes, we attach a new version number and know which syntax or set of functions to expect.  I’m designing the language to describe the problem instead of how to carry out the solution.  We still have some work to do in this area, but it’s shaping up well.

By reducing the interactive, algorithmic project site to a collection of pages in a content management system, we get closer to being able to treat the entire project in the same way as we treat PDFs.  We have players/readers for PDFs.  We don’t update all the PDFs if there’s a security hole or a change in languages.  Instead, we update our readers.  Even when we update our readers, though, we can still read the PDFs we could read before.

This move to seeing the underlying framework as just a player for the content that is the project site allows us to focus our long-term maintenance on the framework.  If we have a large number of projects that all use the same framework or player, then our maintenance load is greatly reduced.  Instead of maintaining code for a hundred projects, we support code for one.

Long term, I need to move from Radiant to a system that allows large-scale production and publication of digital projects.  Once I return from DHSI, I’ll be tackling a rewrite of the framework to create a publication workflow: begin with a work area for building the project, move through a submission and review process, and then to final publication.  Hopefully we can create a publication environment for dynamic, interactive, data-driven and possibly crowd sourcing digital humanities projects.

A couple of weeks ago, I gave a talk at the Maryland Institute for Technology in the Humanities (MITH) with the same title.  I’ve linked to the video here so you can see the slides along with my monotone voice.  In this talk, I use imagery and music along with mathematics to explore how we might approach taking ownership of computing in the humanities.

The last slide is wrong.  It should be the following:

close:reading(text_) :=
     word_ \ connotes: semantic:connotations(word_),
     word__ = f:split(' ', text_),
     semantic:sensible?(word__)

close:reading(sentence_ \ time-period: '1860') 

I’ll explore the implications of this code in another post.

I’ll say right off that I don’t know how to create a great book cover.  What I want to explore in this post is my thinking behind the evolving cover for my first novel, Of Fish and Swimming Swords.

When self-publishing, you have to provide covers for each of the formats you’re publishing.  If an electronic edition, you’ll need the equivalent of a front cover.  If going with a POD edition, you’ll need the front, spine, and back of the cover.  Usually, you want the electronic cover to be a version of the POD cover.  For my covers, I’m using Inkscape for vector drawing and the GIMP for final composition and effects.  Both are free, open source applications that have versions for Microsoft Windows, OS X, and most UNIX/Linux distributions.

The first rule of writing is that money always flows to the author, never away.  When self-publishing, this is true to the extent that you don’t need to pay someone else to do something for you.  The one thing you must never pay for is the opportunity to be published.  It’s one thing to have to pay for a copy of your novel.  It’s another to have to buy a minimum number of copies in exchange for getting published.  The publisher must not require anything that costs you money in order to publish.

Cover design and book layout can be done by yourself if you are publishing through a reputable self-publisher, especially if publishing electronically.  However, if you aren’t good with graphics or typesetting, there’s no harm in paying someone else to do the work for you as long as there’s no conflict of interest with the publisher.  You should never feel compelled to use an artist or typesetter recommended by the publisher.

Almost every book or article talking about writing says that you need to capture the reader in the first sentence.  You want to hook the reader so they always want to know what’s next.  You want them to feel compelled to turn the page.  At the end of a chapter, you want them to feel that things are unresolved.  You don’t want them coming to the end of a chapter feeling that they’ve reached a good stopping point.

Before they get to the first sentence, though, they see the cover of the book.  The cover needs to be inviting and just enticing enough that they will want to see the first sentence.  Covers say things about the book.  To make a good cover, you need to know what you’re saying when you design it, and you need to know what reader’s might think.  This is just like writing, but it’s visual instead of textual.

Questions We Want to Ask

Take a moment and read Top 8 Cover Design Tips for Self-Publishers.  Reviewing the tips, we have the following questions we need to ask of our cover:

• Does the cover convey science fiction?
• Does the cover convey a sense of searching and mystery?
• Does the cover convey the surreal ambience of much of the novel?
• Can we remove an element of the cover and still have it work?
• Do we have a background that will work on a white screen?
• Can we read the title on the thumbnail version?
• Is the title easy to read?
• Does the imagery clarify what the reader can expect?
• Do we have too many colors?

First Attempt

First Draft eBook Cover

Let’s take a look at my first attempt at a cover for my novel in light of the above questions.  This is my second draft of this style.  I changed the background from black to white.  We’ll take a look at a dark version after this one.

The cover doesn’t really speak to science fiction specifically. Someone might connect the symbol with something religious or mystical, which would be accurate as far as the symbol itself went, but wouldn’t be representing the genre accurately.  The mysticism that comes with the symbol might convey a sense of mystery and searching, but only in the sense that mysticism is about mystery and searching.

There are a lot of things here that we could take out.  The texturing of the symbol and lighting that makes it look almost like a lifesaver candy.  The background yellow gradient.  In some ways, the graphics have been overdone to the point of being an amateurish exercise of a graphics program.  While I only have four basic colors, I have gradients and shading all over the place in the symbol, and no color anywhere else.

The title actually is readable in the thumbnail, but just barely.  I like the script and the way I could connect the ‘S’ in Swimming with the ‘f’ in Of.  It’s like a ligature, but across lines instead of adjacent letters.

Zero-eth Attempt

Black background version of first cover draft

I said that the white cover above was my first attempt, and that’s mostly true.  I did an earlier version with a black background before changing to a lighter, white background that I felt was a bit friendlier.

I like the symbol better in this version.  It’s a little more abstract.  The coffee stains help it blend in with the rest of the cover.  The only problem with the stain images is that I’d rather they be blood than coffee.  Even though no blood appears in the book, the idea of mystery and mayhem are suggestive of what the novel is about.  This version also works well as a thumbnail, though the coffee stain might make some of the text a little difficult to read.

Even though I’ve moved on to a completely different composition of symbols for my second cover draft, I might return to this idea if I feel my current version isn’t going to work. However, that will mean that I will need to figure out how to reproduce the symbol design here as well as get the coffee stains to work at 600 dpi.  The GIMP plugin crashed last time I tried it at that resolution, but 600 dpi is really the lowest resolution you should do a cover for POD use.  POD publishers say you shouldn’t go below 300 dpi, but that’s because 300 dpi is the bare minimum for printing that looks smooth when viewed from a foot or two away.  I’m hoping to work at 1200 dpi if I can get the GIMP to manage it.

Second Attempt

Novel cover based on color and abstract imagery

My second attempt was a completely different view of what the cover should be showing.  Instead of going for something that was literal to the text, I went with a design that captured the concepts of the novel without showing anything really from the novel.

The style makes me think of the late sixties and early seventies.  It’s very abstract, with the blue and maroon playing the part of the sky, and the yellow being the ground.  The tree is a fractal that represents the computational aspect of the tree portrayed in the book.  The boundary between the blue and maroon mirrors the arc of the tree branches without following it exactly.  The other choice was to cut across the branches.

I’m not happy with the title yet.  The font is reasonable given the other design elements, but I think it vibrates with the other elements behind it.  I want to mirror the flow of the tree and color boundaries with the flow of the title, but I may need to cut across the other lines to make it work.

Of all of the covers, I think this one gets the most things right with respect to the eight tips we read earlier.  It’s a bit mysterious without committing to anything specific.  It’s surreal in a way that parts of the novel are surreal.  It’s representative without being concrete.

What do you think?  Do you have a favorite of the three?  What tweaks do you think I should try?