How This Was Built
The architecture here is deliberately opinionated: it isn't a loose bag of scripts that happen to produce a website; it's a designed system with a point of view about honesty, and that view is enforced in the structure itself.
The central principle is a strict division of labour, and the work is split among a chain of specialists — a transcriber, a translator, an archivist, an editor — each with exactly one job and a clean handoff to the next. No single step is trusted to do everything.
This page describes how that works, and what running it taught us about where machine help is strong and where a person's attention is worth most.
First, the transcriber and the translator
The transcriber is a handwriting-recognition model trained on this era's Yiddish script; it turns the strokes on a scan into text. The translators work from that text rather than from the image, so a reading error and a translation error stay separate and traceable.
For most of this project there was one machine reading of the Yiddish, and it looked authoritative because there was nothing to compare it against. A single reading always looks certain.
There are now four independent readings of every Yiddish letter, and comparing them is what makes the difference: our two newest tools agree with each other more closely than either agrees with the reading this site was originally built on. That doesn't prove the newer ones right — machines agreeing with each other never does — but it produces something more useful than confidence. It produces a map of where the older reading most deserves a second look.
What a disagreement looks like in the data
Every handoff is written as structured, labelled records rather than loose text, and nothing is ever overwritten. A new reading is added alongside the old one; the old one stays.
Each letter carries a required confidence field, and uncertainty is recorded at two grains: the whole document — 82 letters are flagged uncertain outright — and the individual span, where thousands of markers in the text show where a particular word couldn't be read with confidence.
Those markers were built to answer one question: could the machine read this? That work taught us there is a second question they don't answer — did the machine read it confidently and get it wrong? A garbled word looks uncertain. A wrong word that happens to be a real word looks like a sentence. Learning to tell those apart is most of what the last month of this project has been about.
What a record actually holds
Every letter is a single file, and every reading of it lives inside that file as a separate labelled entry rather than replacing what was there. If you take one design idea from this page, take that one.
A transcription entry carries four things: which tool produced it, what script it is in, the text, and the spans within it that are uncertain. A translation entry carries the same, plus one field that does most of the honesty work — basis: which transcription it was made from.
That one field is why a reading error and a translation error never get confused. If the English says something surprising, you can walk back to exactly which machine's reading of the handwriting produced it, and look at the scan behind that. Without it you have an English sentence and no way to ask where it came from.
Our earliest letter now holds five readings side by side — the original handwriting model, a Latin-script transliteration, two newer handwriting models, and a vision model's attempt. None overwrote another. That is what made it possible, later, to discover that the first one was the outlier.
The stack, and how small it is
People assume a project like this needs infrastructure. It doesn't, and that is worth saying plainly to anyone deciding whether to start.
The entire pipeline is Python with six dependencies: a test runner, a JSON schema validator, an HTTP library, an image library, an XML parser, and a PDF writer. No database. No server. No framework. The data is JSON files in a folder; the website is static HTML generated from them; the whole thing is version-controlled in git and deploys as files.
For the handwriting we used Transkribus, which hosts models trained on historical scripts and charges per page — roughly one credit per page per model, so reading a hundred and fifty letters with two models costs about what a couple of restaurant meals do. The translation and most of the software itself came from large language models.
The reason to keep it this small is not frugality — it is that every part of it can be inspected. When a number looked wrong, we could open the file it came from. A family archive that outlives its author has to be readable by whoever inherits it, and a folder of JSON and a static site is about as portable as digital things get.
Then, the archivist
The archivist is an extraction stage that reads the finished text and pulls out every person, place, date and photograph the letters mention, filing each one as its own record that cites the letter it came from. It works like a card catalogue with a drawer per kind of thing: one for people, one for places, one for dates, one for photographs.
Nothing enters the catalogue without a source, and a card that loses its source is a defect the system reports rather than a tidy fact that quietly survives.
The important part is that the catalogue is separate from the letters: a letter record holds what that page says; the catalogue holds what the collection knows. So "Reuven" mentioned in nine letters is one card with nine citations, not nine loose mentions — which is what makes it possible to ask questions across the whole archive rather than one page at a time. When a name turns out to be a misreading, you correct the card and every reference to it moves at once.
⚠️ One rule earns its keep more than any other here: never delete, always supersede. A corrected record keeps its old version and gains a pointer to the new one. We have needed that history more than once, and a system that overwrites cannot tell you what it used to think.
Then, the editor
The editor is the role that selects and arranges: the archive decides what is true, the editor decides what is told. That seam exists because the catalogue holds everything the letters support, gaps included — and not all of it belongs on a public page, while the same true fact can still be told well or badly.
This is also where we learned the most, because two passages have since been refined where an editor, reading a damaged line in good faith, resolved an uncertainty the evidence left open — in one, a couple described as "young" are better read as worn-out and old; in another, a man we had walking away from a bad contract on his own initiative was in fact talked out of it by others.
In both cases the machines had marked the passage uncertain and the resolution happened downstream. It's a specific, correctable pattern, and knowing to look for it is worth more than the two corrections themselves. A translator filling a gap from context doesn't know it's filling a gap.
Giving the material shape
From that one catalogue the same material is laid out in different ways — a browsable shelf of letters, a woven story, a family roster, a map of where the letters were sent from — each drawn from the archivist's records rather than written separately. Correct a fact once and every page showing it changes.
This is the difference between a website about an archive and a website generated from one. Nothing on these pages is typed twice. A person's dates appear on their card, in the story, and beside every letter they are named in, and all four come from the same field. There is no version of this site where the family tree says one thing and a letter caption says another, because there is nowhere for the second version to live.
The practical cost is that adding a new kind of page means teaching the system a new shape rather than writing HTML. The practical benefit is that the archive cannot drift away from itself while nobody is looking.
Where the AI fits, and where it deliberately doesn't
Much of this was built with AI assistants working alongside the two of us, and their role is broader than it might seem: the same models both wrote the software — the pipeline, the honesty checks, the site itself — and did the interpretive work of translating letters, drawing out themes, and shaping raw material into a narrative.
The design draws a hard line around what AI is allowed to decide: it reads, translates, drafts and builds; it never gets the final word on what is true. The failure mode being designed against is specific: a model asked to transcribe a page it cannot read will produce fluent, plausible, entirely invented text. One model was dropped from transcription for exactly that, and kept its role as a translator, where the source text is fixed and there is nothing to hallucinate from.
A faithful garble is worth more than a beautiful fabrication, because you can see a garble and fix it.
How the work actually happens, day to day
A small vocabulary of verbs turned out to be the most portable thing here, and you could adopt it on day one, before writing any code at all. What you say to the system once it exists is the part nobody writes down.
Ingest. New material arrives.
Discover. Read new material against everything already here, and report what it finds.
Audit. Ask one question across the whole collection.
Correct. A recorded fact was wrong — change it, and keep the history.
Clarify. The fact stands; we now know more.
Connect. Two records are the same thing, or should link.
Park. A real observation with nowhere to put it yet — keep it visible.
Decline. Ruled out; stop asking.
Retire. Take it off the site without losing it.
Vet a claim. A human judgement call, and the word never stands alone.
Build. Change the machine itself, not the record.
Almost every request a family archive receives is one of those eleven, and naming which one you mean turns a vague ask into a specific operation. "Something's wrong with that date" is a Correct. "Hilary sent us some photos" is an Ingest. "Is this problem anywhere else?" is an Audit — and that last one is the verb that turned a single reported error into fifty-nine.
Two of them matter more than they sound. Park exists because the alternative is losing observations that have nowhere to go yet, and an archive accumulates those constantly. Decline exists because without it, the same rejected idea comes back every few months.
Where everything that isn't in the letters goes
A family archive is never just its documents — it is also an aunt's recollection over dinner, a half-remembered street name, a hunch about why someone left a city, a photograph nobody can date, and a hundred questions that occur to you at midnight and are gone by morning.
None of that fits in a letter record, because none of it is in a letter, so we gave it its own place, and it turned out to be one of the most load-bearing parts of the system.
A landing zone with no standards. Anything that might matter gets dropped into a folder as a dated file with a couple of lines of front matter — what it is, where it came from, who wrote it down. Explicitly no obligation to verify it, categorise it correctly, or write it well. The whole design goal is that capturing something costs nothing, because the alternative is losing it. There are over a hundred and fifty of these now.
Findings, not facts. A memo is what you found; it is not yet what is true. Nothing in it reaches the archive until someone promotes it deliberately, and the distinction is enforced by keeping them in separate places entirely. An interview with a relative can sit for months as an interesting claim, cited as an interview, without ever quietly becoming a date on a person's card. That gap is the whole point — it is where a hunch would otherwise turn into a fact while nobody was watching.
Interviews and outside knowledge. Most of ours are external: emails with relatives, notes on what a historical event actually was, a town's real name versus the one a letter spells phonetically, records found elsewhere. These are what make the letters legible — a reader needs to know what the Panic of 1907 was to understand why a man in Montreal suddenly could not find work.
Research requests, which is the half nobody expects. The same folder holds questions as well as answers. "Every mention of money across the whole collection, organised by period." "Which letters mention a town in Latvia." "Where does the correspondence go quiet, and for how long."
Those are questions no single letter can answer, and reading a hundred and fifty letters looking for one pattern is exactly the work a person will not do twice. Around a quarter of our memos are the output of sweeping the whole corpus for one thing at a time — and several of the archive's real findings came from there rather than from any individual letter. The business-transaction sweep is the clearest example: read one letter and you see a man short of money, read all of them in order and you see a career, from a dollar for four days' labour to international trade deals thirty years later.
⚠️ The habit worth stealing is the cheapest one. When a question occurs to you that the collection might answer, write the question down immediately, in the same place as everything else, without answering it. Half of them turn out to be answerable in an afternoon once someone gets to them, and none of them survive being remembered.
Where human attention goes, and why
We used to describe this as a person checking every reading. The real picture is more interesting, and more useful to anyone attempting something similar.
A family member worked through the earliest letters line by line. Then he noticed that on a language he doesn't read, his own guesses about an unclear word weren't adding information — so he stopped, and left those passages standing as the machines had marked them.
That judgement was right, and it took a month to prove: the two readings we have since refined both came from those first heavily-edited letters, because that particular slip requires a person confident enough to settle a question the evidence leaves open.
So human attention is spent deliberately rather than evenly: a small number of letters have had a full line-by-line pass; the rest carry the machines' own uncertainty markers, visible in the text, pointing anyone who reads Yiddish straight at the passages where their help is worth most. Two readers have already written in. It is the single most effective check this project has.
The tools themselves
The models matter more than the platform they run on, which is the detail most people get backwards when pricing this out.
Transkribus
Handwriting recognition trained on real historical scripts, hosted as a service that charges per page. You pick a published model rather than training one, which is the whole appeal — someone has already done the expensive part on handwriting like yours.
The models matter more than the platform, and this is the detail most write-ups leave out. We started with The Dybbuk, trained largely on Yiddish play manuscripts, and used Yiddish→Latin as a second reading that renders the same page in Latin characters — useful precisely because you can check it without reading Yiddish. Later we added Letter Reader and Mame Loshn Maven, both trained on family correspondence rather than published texts.
The genre difference turned out to matter enormously: our original model was reading personal letters with a model shaped by theatrical manuscripts — a genre mismatch nobody had named until a stranger suggested the alternatives. The two letter-trained models independently reproduced corrections an expert had reported, which is the strongest signal this project has recorded. We also tested one further model, Pinkas Brody, and rejected it: it disagreed with everything, uniformly, which carries no information at all.
Claude
Did double duty, and this is worth being explicit about: the same model wrote the software — the pipeline, the honesty checks, the site itself — and did the interpretive work of translating letters, drawing out themes, and shaping the narrative. Both halves matter for anyone estimating effort. A family archive of this size is now a few weeks of evenings rather than a funded project, and that is almost entirely because the software-writing half is no longer a separate skill you have to hire.
Gemini
A second, independent translator used as a cross-check. Where it and Claude agreed we trusted the reading more; it was never allowed to quietly overrule the primary.
What it costs
Roughly one credit per page per handwriting model, so reading a hundred and fifty letters with two models runs to about the price of a couple of restaurant meals. Add ordinary consumer AI subscriptions for the translation and software work. There is no database, no server, and no framework anywhere in this — six Python libraries, a folder of JSON files, and static HTML.
Git, the unsung tool
Version control is the least glamorous thing on this list and the one we'd argue hardest for. Not as a backup — as the only complete record of what a person decided, and why, at the moment they decided it.
Git is the only thing here that records how we came to think something, rather than what we concluded. Everything else records what is true. That difference turned out to matter more than we expected, and it produced several habits worth stealing.
- Write the reasoning, not the change. A message saying "fixed dates" is worthless six months later. A message saying which dates, what the evidence was, and what you rejected is a research note that happens to be attached to the exact change it describes. We write commit messages as though someone will need to reconstruct our thinking much later — and on this project "much later" turned out to be four weeks.
- Decision archaeology. Twice we needed to know something about our own editorial history that no file in the project could answer: which letters had received real human attention, and how carefully. The metadata we had designed for exactly that question turned out to be nearly backwards — the marker we used to record a review appeared mostly on the letters that had been skimmed. The real answer was in the commit log, in messages written weeks earlier by someone documenting what they were doing as they went.
- Workflow archaeology. Because every change is timestamped and described, the history is also a record of how the work actually went rather than how we remember it going. That is how we established that the careful human editing pass covered the first few letters and then tapered off — not from memory, which would have been generous, but from the shape of the commits themselves: one letter, then one, then three, then two, with the edits per letter shrinking each time. That measurement changed what this whole project believes about where human attention is worth spending.
- The archive of what didn't work. Approaches that failed are still in the history with the reasoning attached. We tried seven things that didn't work, and the record of why is worth more than the record of what did — it stops you re-trying them, and it is the honest answer when someone asks whether you considered X.
- A rollback point before anything risky. Tag the state before a change you might regret, and the question "can we undo this?" stops being frightening. We tag before any edit that touches published copy.
What we're confident about, and what we're not
The shape of the story is solid — half the archive is in English and carries no transcription risk at all, and the narrative rests on people, places and dates that those letters, family documents and ordinary records corroborate. Sam crossed the Atlantic when we say he did, worked the jobs we describe, and wrote to Dora as often as the collection suggests.
What stays less certain is texture — a phrase, a joke, the exact register of a man's voice in a language none of his descendants read. That's worth caring about precisely because it's what the letters are for. It isn't the same as the history being wrong.
And the letters themselves are safe: a hundred and fifty pages photographed at archival resolution, dated, indexed and readable by a family who couldn't read a word of them two years ago. Even if every English word here were rewritten tomorrow, that part stands.
If you read Yiddish and find something we've missed, please tell us. That's the mechanism that has worked best.
If you're starting one of these
The short version, in the order we'd do it again.
- Photograph everything first, at the highest resolution you can manage, before anything else happens to it. The scan is the only thing in the system that cannot be wrong, and everything else is disposable and rebuildable from it.
- Get the text off the page with a model trained on your language, script and century — not a general-purpose one. Translate in a separate step, from the transcription rather than from the image, so a reading error and a translation error stay distinguishable.
- Keep every reading. Do not overwrite. When you can afford a second machine reading, get one, because two independent readings disagreeing is the cheapest error detector there is — and much cheaper than a person's attention, which you should save for the places the disagreement points at.
- Mark uncertainty, and make sure nothing downstream can quietly resolve it. That is the one that cost us both of our real errors.
- Then publish the scan beside the text, and invite correction. Of everything described on this page, the correction invitation has caught the most — two readers who owed us nothing wrote in and improved the archive. No amount of tooling has matched that.
More on all of this, including the parts that didn't work, in the other two pieces here.