Why are the scans not deskewed and in some cases edges cut off ?
Show HN: Scanned 1927-1945 Daily USFS Work Diary
21–30 of 31 posts
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#22Earlier quoted context omitted.
A fresh training dataset ;-)
Yeah. If there are groups that want the high resolution images, talk to me.
Could consider putting it up as a dataset on Kaggle, perhaps? I would think they'd provide hosting for such things?
Archive.org would be another option as a repository for the high-res scans in an accessible / discoverable location.
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#23Very cool. Some feedback: - I think it would be a very large improvement if the actual diary pages/transcriptions were more accessible. I found the LLM summaries completely uncompelling, and did not particularly appreciate having to scroll through 5+ pages of LLM summary to get to the part where I could actually read the diary entries for a given month. - The dates of the diary entries for many months are broken. For…
Addendum: After further consideration, I would like to offer two specific suggestions regarding the first point.
Low effort, minimal change suggestion: a link or table of contents header at the top of each month's page to jump to the diary entries.
Higher effort, bigger change suggestion: I think it would make for a significantly better reading experience if all of the diary pages and their transcriptions for a month were listed sequentially, such that you could seamlessly read them without clicking previous/next page.
I think it's a bit of a waste to have put so much effort into preserving this, but the actual ability to read it is de-prioritised relative to the ability to read an LLM summary.
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#24Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
That's amazing! I'm working on a kinda similar project (documenting bank runs from historical newspapers) and also opted for Claude to build a static website. Crazy that the two sites have a very similar look and feel: https://www.finhist.com/bank-runs/index.html . The only big difference is that mine lacks a map, which I should hopefully fix soon (I already have lat and lon and am linking to google maps). PS: Do you…
That's cool! I've noticed when asking for Claude for a website, it does have a certain look, like our two sites, if you don't give it any more guidance. I'm not sure if that's a good thing or not.
Digitizing history in different ways, with different resources that are unique or only known to small groups, might be a new development area, and that's exciting. As I've shown, and how other people have shared, using AI tools to digitize things which haven't previously been done before is now possible. Are there ways to make this easier for everybody? New techniques to discuss? I don't know, and I'd love to talk about it.
Concerning OCR: I used Mistral because of a posting here describing advancements with handwriting recognition a month or so ago. I didn't actually compare them. And I've got my setup that I can rerun everything again later if there are advancements in the area. Again, another area to keep track of and discuss.
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#25Very cool. Some feedback: - I think it would be a very large improvement if the actual diary pages/transcriptions were more accessible. I found the LLM summaries completely uncompelling, and did not particularly appreciate having to scroll through 5+ pages of LLM summary to get to the part where I could actually read the diary entries for a given month. - The dates of the diary entries for many months are broken. For…
Addendum: After further consideration, I would like to offer two specific suggestions regarding the first point. Low effort, minimal change suggestion: a link or table of contents header at the top of each month's page to jump to the diary entries. Higher effort, bigger change suggestion: I think it would make for a significantly better reading experience if all of the diary pages and their transcriptions for a month…
You're right, and it's a good idea. The summary started out small, as a header to the actual daily pages, but then I realized I could have AI do a lot more work here, including silly things like collect weather references and assemble them together. My prompt kept getting bigger to find trends in the data. But, it takes away from the view-ability of the site, which is not good.
LLM's ability to take 7400+ handwritten entries and try to make a narrative out them is amazing. With all of the AI experiments on HN lately, we're figuring out the power of LLMs, but it most cases, it still needs a human refining touch, and we need to remember that. Or else it just looks like AI slop.
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#26Earlier quoted context omitted.
Addendum: After further consideration, I would like to offer two specific suggestions regarding the first point. Low effort, minimal change suggestion: a link or table of contents header at the top of each month's page to jump to the diary entries. Higher effort, bigger change suggestion: I think it would make for a significantly better reading experience if all of the diary pages and their transcriptions for a month…
You're right, and it's a good idea. The summary started out small, as a header to the actual daily pages, but then I realized I could have AI do a lot more work here, including silly things like collect weather references and assemble them together. My prompt kept getting bigger to find trends in the data. But, it takes away from the view-ability of the site, which is not good. LLM's ability to take 7400+ handwritten…
I certainly don't think it's a bad thing to try to refine the information into a more digestible form. I think, for example, the dedicated "People", "Places", "Events", and "Map" sections are well-organized and interesting[1]. I would simply prefer if the presentation of this information did not detract from the ability to read the diary itself, as it does on the month pages. I am rather fond of reading historical diaries as part of a general curiosity about the past, and reading the experiences as they were written is as interesting to me if not more so than the aggregate information, personally.
[1] Although, of course, there is the question of reliability. For example, the "Boy Scouts" page says Boy Scouts have 2 mentions, but has references to 3 diary entries! Also, on further examination, Sep 1931 has broken dates (meaning my previous theory about it breaking only after Jul 1941 was wrong), and some pages appear to be out of order.
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#27Also, just to clarify, I scanned all 7488 pages in personally (Fujitsu ScanSnap ix500). With Claude's help, I found some undocumented SANE features to auto crop and fix the scans, then had a Python script in Linux auto scan them and put them into a Postgres database as I went. Other scripts would add transcription, summaries, and auto index everything. "mistral-ocr-latest" did really good handwriting transcription, c…
Oh boy. #3 on front page, 19k page hits in the first hour. 8243 static html pages, 15728 webp images (10k-50k each). I've never had one of my sites with this much traffic. With everything as static files, website is still holding. Thank you all.
Before your server catches a fire and burns down the originals: please also send them to archive.org
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#28Ai really loves that card style
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#29Earlier quoted context omitted.
That's amazing! I'm working on a kinda similar project (documenting bank runs from historical newspapers) and also opted for Claude to build a static website. Crazy that the two sites have a very similar look and feel: https://www.finhist.com/bank-runs/index.html . The only big difference is that mine lacks a map, which I should hopefully fix soon (I already have lat and lon and am linking to google maps). PS: Do you…
That's cool! I've noticed when asking for Claude for a website, it does have a certain look, like our two sites, if you don't give it any more guidance. I'm not sure if that's a good thing or not. Digitizing history in different ways, with different resources that are unique or only known to small groups, might be a new development area, and that's exciting. As I've shown, and how other people have shared, using AI t…
Thanks for the insights! I'll try Mistral as well.. Gemini worked well for me so far but which model is SOTA is changing quite frequently these days
Re: Show HN: Scanned 1927-1945 Daily USFS Work Diary
#30Earlier quoted context omitted.
This is great! I love it when people take bits of history that works be forgotten and put them out in the world (to be further vacuumed up by Internet Archive). Thank you for doing it.
Beej! Thank you very much! Your networking guides have long been a great contribution to everybody, and collectively improves what we know. These diary pages come largely from Stirling City, just north of Chico, and later from the Hat Creek district, on Hwy 89 north of Mt. Lassen. Nearby, many historical records were lost in the Paradise Camp Fire, and digitizing some of the records in some of the local museums is so…
Very cool! I'm struggling to recall if I ran into you at CSUC, though--close timing. My memory isn't what it used to be. :)