Sharing texts better, part 1: Austrian Newspapers

I wrote them into a folder with level 5 compression in zstd. The new directory, with parquet files and ids, is a tenth the size: 6.4GB vs 63GB for the zipfile I downloaded. Why on earth have I downloaded massive XML files when I just want text? Who really wants this positional text, anyway? I’ve used it a few times over the years–but most people want text, not XML. Zipfiles at least are nice, because I can grab the specific files I want. But they’re also slow in their own right. I start parsing at 22:21, and leave my computer open–looking at the timestamps, I don’t finish the last file until more than two hours later, at 00:31.

This is bonkers. Mediocre zip compression and uselessly XML-encoded data mean that it takes two hours just to look at the data in the most cursory way. It’s important to distribute things in a complete format, but it’s also important not to waste resources making things too hard to parse. With the parquet formatted versions of the data, it takes not two hours but 55 seconds to parse through every file in this set. That’s a major improvement–100 times faster to read, and one-tenth the size. Both of those are big enough differences that they actually affect whether this data is usable or not.

matches = []
from pyarrow import compute as pc
for p in Path("parquet_files").glob("*.parquet"):
    a = parquet.read_table(p)
    which = pc.match_substring(a['pages'], "Gustav Mahler")
    matches.append(a.filter(which))

So–now we’ve got a huge set of text in a fairly navigable form. But we don’t know what the records are. The identifiers are all things like 9200300/BibliographicResource_3000123565676/4; aside from the page number, it’s not clear what any of those mean. My working theory to this point was that 9200300 meant the Neue Freie Presse and BibliographicResource_3000123565676 means the individual issue; but I need to know for sure.

At this point, I start putting the identifiers into the web site and figuring out the layout of the metadata here. It turns out that this is not just one newspaper, but lots–probably everything contributed from the OSB to Europeana. And, stunningly, the order seems to be completely random? I call the web based Europeana API and get a dcTitle field in this order:

["Der Humorist - 1847-01-29"]
["Blätter für Musik, Theater und Kunst - 1871-09-19"]
["Wiener Zeitung - 1841-10-18"]
["Der Humorist - 1841-03-10"]
["Neue Freie Presse - 1871-10-22"]
["Innsbrucker Nachrichten - 1859-11-25"]
["Die Presse - 1867-06-25"]
["Das Vaterland - 1862-09-26"]
["Wiener Zeitung - 1705-02-28"]
["Wiener Zeitung - 1868-12-04"]

There a couple things weird here. One is the random order. I suppose that this could be my fault, because I just used the filenames from the zipfile in the order they appeared, rather than sorting. But that itself is a problem–the zipfile should have more of an inherent order. It is an underappreciated fact that good sorting is good compression; the more natural an order information appears in, the better it will compress. And of course, the fewer files people will have to download. The other is that “title” is wrapped in an array: apparently in the EDM things can have multiple titles. OK, that’s something I can work with.

So now I have a clear plan.

1. Get metadata for every record.

2. Match it to the papers.

3. Write out each newspaper in chronological order.

To get the metadata, I have to find it–there is no metadata in the data dumps. First I do it using the API. https://api.europeana.eu/record/v2/{id}.json?wskey={api_key}' But it quickly becomes clear this won’t scale: Running overnight I’ve only download 35,000 of 1.3 million records. So I go back to the Europeana page and download another enormous zipfile–a 4 gigabyte one with records for the entire set. How this manages to be so large isn’t initially clear to me–perhaps, I think, they’ve bundled the full text into it?

The answer turns out to be that there is massive amounts of text for each record because, chiefly, every records repeats an extremely long definition of ‘ newspaper ’ in many different languages. That this balloons the size so much is a failure of an over-literal use of linked data. Perhaps there would be a way to reference it as an element in a single HTML file, but really, no one cares. This part of the data model will never be used outside a Europeana site–there is some base-covering in distributing it, but it’s a massive inconvenience for researchers to have the following block of text (and something vaguely equivalent in Latvian, Arabic, Russian, etc.) **repeated 1.6 million times in a file that’s supposed to be a metadata dump about newspaper issues:

Many newspapers, besides employing journalists on their own payrolls, also subscribe to news agencies (wire services) (such as the Associated Press, Reuters, or Agence France-Presse), which employ journalists to find, assemble, and report the news, then sell the content to the various newspapers. This is a way to avoid duplicating the expense of reporting.

Now, I understand the need for clear URIs for concepts and the benefits of linked open data. But the nature of linked open data is that any individual record can be ballooned indefinitely. Why is there a definition of ‘ newspaper ’ at such tedious length and not, say a full expansion of the geographic definition of ‘ Graz ’ where it appears? I am sure there is a reason–but I’m equally sure it’s not really a good one.

Toggle to see the metadata for a single newspaper

So now I’ve got to parse these monster XML blobs 1.3 million times. And this time I can’t resort to regex. Ugh. Again, this is something that most researchers will abandon quickly. I’m increasingly XML referred to in the past tense online, as a data format/data movement that failed. Evangelists will surely disagree, and certainly a great deal has been lost. But for my purposes, I need something tabular that can be joined, and XML and tables play extremely poorly together.

But I’ll try. The first step will be to get into JSON-LD format, which is a linked data format that actually works inside of programming languages for non-evangelist humans. It turns out to be something of a pain–maybe ten minutes of vaguely recalling terms before I precisely figure out how to use Harold Solbrig’s rdflib-jsonld extension to the rdflib library to squeeze the data into JSON. Solbrig, thank goodness, has provided a code example. With everything but the format to put in, the transformation is obvious.

from rdflib import Graph, plugin
from rdflib.serializer import Serializer
g = Graph().parse(data=demo, format="xml") #<-took a while to figure this line out!
print(g.serialize(format='json-ld', indent=1))

OK. So all I really need here is the nmewspaper title and the date, so let’s see how to parse it out. Once again, the json-ld is massively large. After wasting 40 minutes trying to figure out if I can implement a general solution to parse out all the various @type entries using a json context into a flatter document, and coming up flat against the difficulties of inferring the many contexts, I decide to just do a quick-and-dirty route that will lose most of the json-ld data here. First, filter to only proxies:

proxies = [f for f in json.loads(d) if 'http://www.openarchives.org/ore/terms/Proxy' in f['@type']]

And then reduce to a dict where we grab the first occurrence of a value or id field if it seems to be a Dublin Core item.

Again, this is requiring a completely different set of skills than the data wrangling above. If I knew a lot about LOD, I could do much better here. But the python libraries I’m finding don’t make this especially easy, so I’m giving up on the LOD dream of being able to put it back together in a multilingual frame.

def parse_row(d):
    proxies = [f for f in json.loads(d) if 'http://www.openarchives.org/ore/terms/Proxy' in f['@type']]
    out = {}
    for k, v in proxies[1].items():
        if "purl.org/dc" in k:
            try:
                out['dc:' + k.split("/")[-1]] = v[0]['@value']
            except KeyError:
                out['dc:' + k.split("/")[-1]] = v[0]['@id']
    return out

{'dc:identifier': 'oai:fue.onb.at:EuropeanaNewspapers_Delivery_3:ONB_00286/1875/ONB_00286_18750610.zip',
 'dc:language': 'deu',
 'dc:relation': 'http://de.wikipedia.org/wiki/Neuigkeits-Welt-Blatt',
 'dc:source': 'http://anno.onb.ac.at/cgi-content/anno?apm=0&aid=nwb&datum=18750610',
 'dc:subject': 'http://d-nb.info/gnd/4067510-5',
 'dc:title': 'Neuigkeits-Welt-Blatt - 1875-06-10',
 'dc:type': 'http://schema.org/PublicationIssue',
 'dc:extent': 'Pages: 4',
 'dc:isPartOf': 'http://data.europeana.eu/item/9200300/BibliographicResource_3000095610170',
 'dc:issued': '1875-06-10',
 'dc:spatial': 'http://d-nb.info/gnd/4066009-6'
 }

This whole process can parse about 40 lines a second. That sounds kind of fast, maybe. But with 1.3 million metadata items it would take nine hours to run, single threaded in Python on my laptop. That is obscene. We can reduce this by batching by issue an getting it down to about an hour–there are “only” 154,000 records in here. But a good metadata format should be able to load a million rows of structured data in under a second, not in nine hours. This data could probably have been released in CSV on the Web, or JSON-LD, or some other format where this process would take a minute or two.

Anyhow–nine hours is too long for me because it’s the morning. I’ll split this up into multiple processes that work on batches of 25,000 at a time, and set it running in a loop.

And I’m back! So now I’ve got data and I’ve got texts. Joining these together is pretty easy–I just pull apart the IIIF ID and merge them in. Now I need to figure out how to distribute these to the student. These are big–too big, probably to simply slap them into an e-mail.

But luckily, I set up a static hosting service on Google a few months ago, so I can just upload them into there. I’ve created files for all of these newspapers now. So we’ve got one for the student, but also for you.