11–20 of 31 posts
It inspires me to tackle a project I've been holding off on for many years: OCR my grandmother/great-grandmother's cookbook. It's about 100 pages of collected and annotated recipes from the 1930-1980s.
OCR and AI have become sufficiently capable (as you've demonstrated) to properly scan, index, and classify the recipes into something I can share with relatives online or as an ebook.
Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
Oh boy. #3 on front page, 19k page hits in the first hour. 8243 static html pages, 15728 webp images (10k-50k each). I've never had one of my sites with this much traffic. With everything as static files, website is still holding. Thank you all.
Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
- I think it would be a very large improvement if the actual diary pages/transcriptions were more accessible. I found the LLM summaries completely uncompelling, and did not particularly appreciate having to scroll through 5+ pages of LLM summary to get to the part where I could actually read the diary entries for a given month.
- The dates of the diary entries for many months are broken. For example, in the final month, all of the entries are labelled 1945-03-19. From a cursory examination, I believe the dating broke 24th July 1941 and was broken for every month from there to the end.
- The page for Nov 1941 seems entirely broken. For some reason, the dates labelling the pages are described in a different format that included the name of the month rather than a numeric representation, the pages are out of order, and then all manner of months are mixed in. The first pages are "November 1941", "April 1941", "October 2 1941", "October 3 1941", "November 4 1941", "November 12 1941", "November 7 1941" ... and so on. The LLM summary notes an "Event", a construction project that took place from 1931 to 1934, despite this being the entry for Nov 1941.
Nice work! For others with journals in the U.S., but not feeling up to all the scanning and transcription work, I volunteer with the American Diary Project ( https://americandiaryproject.com/ ) based in Cleveland Ohio. You can donate journals to be archived and shared. It's only been established in the past few years, and all scanning/transcription is done by volunteers, but are currently evaluating more automated pi…
Earlier quoted context omitted.
Oh boy. #3 on front page, 19k page hits in the first hour. 8243 static html pages, 15728 webp images (10k-50k each). I've never had one of my sites with this much traffic. With everything as static files, website is still holding. Thank you all.
A fresh training dataset ;-)
Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
I'm working on a kinda similar project (documenting bank runs from historical newspapers) and also opted for Claude to build a static website. Crazy that the two sites have a very similar look and feel: https://www.finhist.com/bank-runs/index.html . The only big difference is that mine lacks a map, which I should hopefully fix soon (I already have lat and lon and am linking to google maps).
PS: Do you know if mistral works better at OCRing handwritten text than gemini 3? Was planning on going the gemini3 for another project
Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
This is great! I love it when people take bits of history that works be forgotten and put them out in the world (to be further vacuumed up by Internet Archive). Thank you for doing it.
These diary pages come largely from Stirling City, just north of Chico, and later from the Hat Creek district, on Hwy 89 north of Mt. Lassen. Nearby, many historical records were lost in the Paradise Camp Fire, and digitizing some of the records in some of the local museums is something this is a test run for.
—Lance (CSU Chico ‘93 Computer Engineering)Imagine how much unanticipated historical perspectives might become uncovered if everyone uploaded paraphenelia of long deceased ancestors like this; after indexing, and searched as one hyper-amalgamated crowdsourced knowledge graph, can show who was where doing what in say the 1920s, 1930s, 1940s in a way that mainstream history might fall short of capturing.
There are so many interesting stories out there, from the attorney general who summed up the evidence presented at a trial as “the victim lynched himself and his fellow thieves in jail” to the couple with 6 male children who named each successive pair of boys using the same initials (e.g. Carl Ervin, Carwin Earl; Truman William, Tresman Walter; Llewellyn Purcell, Lealyn Percy).
The resources that are already available are amazing (one woman, Violet Toph, assembled thousands of pages of memories and genealogy records for her county in the mid-20th century) but obviously very incomplete. Your idea would be a terrific way to help fill in some of those gaps and encourage people to keep their own memoirs somewhere outside Facebook.
Fun fact: "Government mule" isn't just an expression, it's a real thing. And the U.S. government, including the Forest Service, still employs teams of mules to carry things to places that can't be reached any other way.