Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

snewpapers.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#11

Post by brettnbutter »

zzleeper wrote: post #8
brettnbutter wrote: post #6

Earlier quoted context omitted.

Nice collection you have there. Just asked the Sleuth for some examples of that, and here's one to add to your Unional National one: https://www.finhist.com/bank-runs/episodes/19827.html https://snewpapers.com/components/0b22f0ca-60d2-4d63-be99-74... Yes I agree the layouts are the trickiest part. I tried a few and ended up using some of the Paddle Paddle models for document layout analysis and orientation and such,…

Great! Was thinking about PP but because I only ran an order of magnitude fewer articles (under 1mm pages; by piggybacking on Dell's OCR) I relied on Arcanum ( https://www.arcanum.com/en/newspaper-segmentation/about/ ) which was cheap enough (but I think not cheap enough at your scale). Cheers!

Hmm, I just tried to upload the jpgs of some of todays samples to Arcanum via https://www.arcanum.com/en/newspaper-segmentation/try-it/ and it didn't work. I'll try it again later, but it seems based on a cursory look that it wouldn't return info that I would need to correct it if I didn't like the output, and that I'd still have to stitch the individual pages back together myself?

Probably much cheaper than my process though...

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#12

Post by brettnbutter »

nastrofa wrote: post #10

It would be really cool to create different analysis across the time - Each month's / year's top news headline - Left / Right swings of publishers

Great idea. This should be fairly easy to do with the embedding vectors I have for the semantic search, using some clustering tools. Adding it to my backlog now!

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#13

Post by brettnbutter »

StilesCrisis wrote: post #7
brettnbutter wrote: post #2

A few examples you can click on without having to authenticate or click the free trial (no cc if you do though and I won't bother you or chase you with spam etc...) https://snewpapers.com/components/b2d40c08-db63-40e8-890f-09... https://snewpapers.com/components/0fabc8e4-a60b-4f31-9ad1-b0... https://snewpapers.com/components/cdde790f-4e97-4f2d-a2c2-95...

I see an obvious typo in the first one: "wickked deeds of witchecran" (should be craft) I can see why the OCR is a challenge here, and spellcheck is a lost cause, but I'm surprised an LLM cleanup pass didn't detect this?

I see what you mean now with the witchcran. I do instruct it not to correct old-time spellings, and only to tidy up obvious mistakes. I guess it got confused on this one

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#15

Post by brettnbutter »

I'm opening up https://snewpapers.com/today-in-history to the world right now. Per @benwillis's advice below, I will figure out how to make a section of the data searchable for free as well, but this is the best I can do today!

Thank you everyone for taking the time to look at snewpapers today, enjoy!

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#16

Post by whythismatters »

brettnbutter wrote: post #15

I'm opening up https://snewpapers.com/today-in-history to the world right now. Per @benwillis's advice below, I will figure out how to make a section of the data searchable for free as well, but this is the best I can do today! Thank you everyone for taking the time to look at snewpapers today, enjoy!

>for free

I took a quick look but could not find any pricing info on the website, could you clarify a bit?

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#17

Post by brettnbutter »

whythismatters wrote: post #16
brettnbutter wrote: post #15

I'm opening up https://snewpapers.com/today-in-history to the world right now. Per @benwillis's advice below, I will figure out how to make a section of the data searchable for free as well, but this is the best I can do today! Thank you everyone for taking the time to look at snewpapers today, enjoy!

>for free I took a quick look but could not find any pricing info on the website, could you clarify a bit?

When you're anonymous or google/email authd, you can now see today-in-history stuff. There should be a gentle nudge to sign up when you view a component or go to pages that require activating the free trial (no cc required) which opens up search / sleuth / custom collections. That takes you to /subscribe with all the options

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#18

Post by brettnbutter »

whythismatters wrote: post #16
brettnbutter wrote: post #15

I'm opening up https://snewpapers.com/today-in-history to the world right now. Per @benwillis's advice below, I will figure out how to make a section of the data searchable for free as well, but this is the best I can do today! Thank you everyone for taking the time to look at snewpapers today, enjoy!

>for free I took a quick look but could not find any pricing info on the website, could you clarify a bit?

I'm going to try to figure out how to make a subset of the data searchable for free (for anyone anonymous or not without having to start free trial) as well in the next few days, so people can play with that too before committing to anything.

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#19

Post by seanb »

brettnbutter wrote: post #17
whythismatters wrote: post #16

Earlier quoted context omitted.

>for free I took a quick look but could not find any pricing info on the website, could you clarify a bit?

When you're anonymous or google/email authd, you can now see today-in-history stuff. There should be a gentle nudge to sign up when you view a component or go to pages that require activating the free trial (no cc required) which opens up search / sleuth / custom collections. That takes you to /subscribe with all the options

I'm not going to sign up for a free trial unless I can see the pricing information up front.

Re: Show HN: Large Scale Article Extract of Newspapers 1730s-1960s

#20

Post by brettnbutter »

seanb wrote: post #19
brettnbutter wrote: post #17

Earlier quoted context omitted.

When you're anonymous or google/email authd, you can now see today-in-history stuff. There should be a gentle nudge to sign up when you view a component or go to pages that require activating the free trial (no cc required) which opens up search / sleuth / custom collections. That takes you to /subscribe with all the options

I'm not going to sign up for a free trial unless I can see the pricing information up front.

I think I see the confusing part now. The /subscribe page only is visible once you google auth or enter email / confirm your email exists. It's $9.99/month for the default selection though. In stripe it has the no CC option, so if you click the button it will just auto cancel after a week anyway
Post reply on HN