Clustering from Search

Because of my primitive search engine, I’ve been thinking about some of the ways we can better use search data to a) interpret historical data, and b) improve our understanding of what goes on when we search. As I was saying then, there are two things that search engines let us do that we usually don’t get:

1)  Numeric scores on results
2) The ability to from a set of books to a set of high-scoring words, as well as (the normal direction) from a set of words to a set of high-scoring books.

We can start to do some really interesting stuff by feeding this information back in and out of the system. (Given unlimited memory, we could probably do it all even better with pure matrix manipulation, and I’m sure there are creative in-between solutions). Let me give an example that will lead to ever-elaborating graphics.

An example: we can find the most distinguishing words for the 100 books that use “evolution” the most frequently:

> books = names(sort(TFIDF.from.usage(“evolution”),decreasing=T))

working on item number 1 of 1 (evolution)…

> bookTFIDF(books)[1:10]

evolution    organisms     organism      organic      mammals  vertebrates    phenomena    sociology

0.0039030229 0.0014772785 0.0012406171 0.0010053423 0.0009848822 0.0009823578 0.0009781454 0.0009637798

fig   intestinal

0.0008745183 0.0008426900

Evolution itself is the most common, but we get a bunch of other words in there—most have to do with biological evolution. “Intestinal” probably includes a lot of course-of-diseases descriptions, and “sociology,” of course, is the discourse that I’m most interesting. So that’s interesting, but nothing new.

What I think is kind of cool, though, is that we can tell the computer to go into  those words. We can do a bunch of searches to see what words distinguish books that score highly on a search for “evolution” and “organisms”, and what words distinguish books that score highly on a search for “evolution” and “society.” (Note–I’m just arbitrarily using the top hundred books using a word as my cutoff to find interesting words here, but if I had more processing power, I’d like to try weighting the results on a scale related to their search score).

A lot of those words will be the same. “Organism” and “organisms”, for instance, have a lot of the same vocabulary when we do co-search with evolution:

> bookTFIDF(books[1:100])[1:10]
  evolution    organism   organisms   molecular   psychical     organic
0.002821304 0.002344120 0.001888864 0.001626369 0.001358292 0.001274280
  phenomena       cells integration environment
0.001142712 0.001058429 0.000937243 0.000915636

> bookTFIDF(books[1:100])[1:10]
    evolution     organisms      organism   integration       organic
  0.002881946   0.002109943   0.001685654   0.001314584   0.001276740
        cells     molecular heterogeneity     phenomena           fig
  0.001170736   0.001144163   0.001051732   0.001030783   0.001026932

We can use the information on co-occurrence to cluster words together into patterns of co-use. This is why I was so bold, Hank, as to call what I was doing looking at “discourses”–these sorts of scores let us start to cluster words into general conversational groups.

[In the language of text analysis, of course, I’m drifting towards not discourses, but a simple form of topic modelling. But I’m trying to only submerge myself slowly into that pool, because I don’t know how well fully machine-categorized topics will help researchers who already know their fields. Generally, we’re interested in heavily supervised models on locally chosen groups of texts. Plus, like Pierre Menard, I find it more fun to reinvent the wheel than to be the first one to invent the—what’s something crappy and wheel related?—steering-wheel desk.]

There are a lot of ways to approach the data, and I don’t know the best one. Here are three. Don’t think of these as static visualizations, which isn’t exactly my thing, but as a process that we can redo for any word or words. When we don’t know about a word or phrase, it can be helpful in rapidly immersing us into information about what discourses it occurs in and helping us do things like craft smarter searchers. When we know a lot, there will always be some data that sticks out as odd–figuring out why that seems odd is a route to better understanding.

1. We can ask for 7 clusters using k-means clustering (powerpoint PDF, but it’s the one I found useful), on about a hundred words highly tied to “evolution.”  I’ve invented the labels, but the results themselves and the choice of the words to cluster all came out TF-IDF scores.

> clusters = 7; kmeans(graph,clusters,20,nstart=10)$cluster -> dat; lapply(1:clusters, function (i) {names(dat[dat==i])})

Zoology and microbiology?

[1] “amphioxus”     “animals”       “apes”          “cell”

[5] “cells”         “development”   “embryo”        “evolution”

[9] “evolutionary”  “fishes”        “heredity”      “layer”

[13] “mammals”       “membrane”      “nutrition”     “ontogeny”

[17] “organisms”     “organs”        “physiological” “primitive”

[21] “protoplasm”    “reptiles”      “sexual”        “tertiary”

[25] “theory”        “tissue”        “vertebrate”    “vertebrates”

Anatomy

[1] “anterior”  “cavity”    “dorsal”    “fig”       “gastrula”

[6] “posterior” “ventral”

Chemistry, math, etc.?

[1] “actions”          “aggregate”        “aggregates”

[4] “chemical”         “differentiations” “equilibration”

[7] “equilibrium”      “genesis”          “heterogeneity”

[10] “heterogeneous”    “hypothesis”       “integrated”

[13] “integration”      “molecular”        “molecules”

[16] “motion”           “motions”          “similarly”

[19] “units”            “vegetal”

Philosophy, etc

[1] “chromosomes” “comte”       “cosmic”      “fiske”

[5] “fiskes”      “germlayers”  “intestinal”  “kant”

[9] “obvgootlc”   “philosophy”  “polygyny”    “science”

[13] “scientific”  “spencer”     “tho”         “universe”

[17] “youmans”

“obvgootlc,” I’m sure, is an OCR misreading of the “Digitized by Google” watermarks on a bunch of scans. I’ve given up trying to root those out for the time being, since they don’t corrupt the rest of the data that much.

Darwinism proper

[1] “darwin”     “darwinism”  “darwins”    “geological” “huxley”

[6] “plants”     “selection”  “species”    “variation”

Biological science?

[1] “axis”            “biology”         “complex”

[4] “differentiation” “environment”     “homogeneous”

[7] “inorganic”       “morphological”   “nervous”

[10] “organic”         “organism”        “phenomena”

[13] “processes”       “structures”      “tissues”

[16] “variations”      “vascular”        “yelk”

Social Darwinism!

[1] “activities”    “altruism”      “altruistic”    “biological”

[5] “consciousness” “economic”      “egoistic”      “ethical”

[9] “ethics”        “factors”       “objective”     “psychical”

[13] “psychology”    “social”        “sociological”  “sociology”

[17] “subjective”

That’s K-means clustering. We can also create hierarchical trees using Ward’s clustering algorithm, also in R:

There are a lot more details to get out of a hierarchical chart like this. I think my favorite is that “Huxley” is closer to “Darwinism” than any other word, followed by “evolution.” Remember, all these correlations are limited to books that use ‘ evolution ’ AND the listed word, so a word like “theory” doesn’t necessarily get to have all its other meanings. It’s also interesting to see just how different the anatomical language is from everything else (the last five words on the chart). I also like seeing how Fiske, Kant, Spencer and Comte each appear amidst a slightly different cluster of words in the philosophical grouping near the top. If only the sample ran a little later, this is surely where Dewey would land.

Finally, I wondered if there was some way to capture the strength of ties among words using some sort of force-directed placement algorithm like the ones usually used for social networks. Each word has different strengths of ties to other words. This doesn’t totally work, but it’s a little interesting nonetheless, so I might as well post it. (click to enlarge). It’s also a slightly different set of words, as I’ve kept tweaking with the selection mechanism. With enough work I think one could create some interesting visualizations. But right now, I see the good of this sort of work in helping perfect searches, and I’m not sure how deep the insight we get from this sort of view are.

But we shouldn’t underestimate shallow insights–that’s what computers are best at, and the best of us need them. Reminders of common words, the ability to skim through areas for words in a family, and so on—can be very valuable in the places we might fall flat. (It took one of these, for example, to remind me that a lot of 19th century books that use the word ‘ society ’ do so in the context of missionaries).

Historians do a tremendous volume of searches, and we don’t always remember just what it is we know. To stretch for an analogy: Intermediate pianists should often play a scale in the key before striking out on a piece to reacquaint their fingers with the environment of the new key. Reviewing visualizations can serve something of the same purpose—reminding us what we know before we try to apply that knowledge, so we’re less likely to make common mistakes.

Also, of course, it’s just fun to see the words arrange themselves into networks that seem to instantiate some sort of semiotic web. There’s social Darwinism in the upper left,  geology on the right, and a cluster of words—species, sexual, cell— at the heart. Sometimes it’s just good to confirm what we already know through a different path.