Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Classifying all of the pdfs on the internet

snats.xyz

Post reply on HN
Print view View original

81–90 of 117 posts

Re: Classifying all of the pdfs on the internet

#81

Post by specialist »

fsckboy wrote: post #57
gsck wrote: post #31

Earlier quoted context omitted.

RTBF isn't about having your information wiped from the internet. Its a safe assumption any public information about you is completely out of your control as soon as its public. RTBF is about getting companies to get rid of any trace of you so they cannot use that data, not removing all traces about you across the internet.

> RTBF isn't about having your information wiped from the internet. your take is misleading enough to be considered wrong. It's "don't use public information about me in search engines, I don't want people to find that information about me", not simply "don't use my information for marketing purposes" https://en.wikipedia.org/wiki/Right_to_be_forgotten first paragraph of the article: The right to be forgotten (RTBF)…

Once demographic data cannot be crawled or cached by 3rd parties, we get RTBF for free.

Re: Classifying all of the pdfs on the internet

#82

Post by samatman »

moralestapia wrote: post #74
mellosouls wrote: post #53

Earlier quoted context omitted.

Like I said, I didn't downvote and took the time to answer your question. I didn't take the time to sugarcoat it. You are interpreting bluntness as hostility; that's ultimately an issue for you to resolve.

You don't have to sugarcoat it. You just have to read this site's guidelines and follow them. Ez pz.

> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

https://news.ycombinator.com/newsguidelines.html

Re: Classifying all of the pdfs on the internet

#83

Post by abhi_p »

guiomie wrote: post #47

Interesting and fun article! I've been experimenting with various LLMs/GenAI solutions to extract tabular data from PDFs with underwhelming results. It seems like they are good at extracting strings of text and summarizing (e.g what was the total price? when was this printed?) but extracting reliably into a CSV has a decent margin of error.

Disclosure: I'm an employee.

Give the Aryn partitioning service a shot: https://www.aryn.ai/post/announcing-the-aryn-partitioning-se...

We recently released it and we've a few examples here: https://sycamore.readthedocs.io/en/stable/aryn_cloud/get_sta... that show you how to turn the tabular data from the pdf into a pandas dataframe(which you can then turn into csv).

Re: Classifying all of the pdfs on the internet

#84

Post by loa_in_ »

namrog84 wrote: post #52
llm_trw wrote: post #28

Back in 2006 there were multiple 1tb collections of textbooks as torrents. I imagine the size and number has only grown since then.

That was before hoarding and building questionable businesses around them became a thing. I remember it being really easy to find textbooks, solution manuals, and related pdf and other stuff as late as 2008 far easier than 6-8 years later. The main difference were sites like chegg and many other sites started slurping them up to resell in some way.

It doesn't take away the torrents, no?

Re: Classifying all of the pdfs on the internet

#85

Post by loa_in_ »

Treesrule14 wrote: post #78
mehulashah wrote: post #76

Classification is just a start. Wondering if it's worth doing something more -- like turning all of the text into Markdown or HTML? Would anyone find that interesting?

There are a lot of webcrawlers where the chief feature is turning the website into markdown, I don't quite understand what they are doing for me thats useful since I can just do something like `markdownify(my_html)` or whatever, all this to say is that I wouldn't find this useful, but also clearly people think this is a useful feature as part of an LLM pipeline.

You don't want the footer or navigation in the output. Ideally you want the main content of the page, if it exists. How do you assign header level if they're only differentiated by CSS left-margin in a variety of units? How do you interpret documents that render properly but are hardly correct HTML?

Re: Classifying all of the pdfs on the internet

#87

Post by Treesrule14 »

loa_in_ wrote: post #85
Treesrule14 wrote: post #78

Earlier quoted context omitted.

There are a lot of webcrawlers where the chief feature is turning the website into markdown, I don't quite understand what they are doing for me thats useful since I can just do something like `markdownify(my_html)` or whatever, all this to say is that I wouldn't find this useful, but also clearly people think this is a useful feature as part of an LLM pipeline.

You don't want the footer or navigation in the output. Ideally you want the main content of the page, if it exists. How do you assign header level if they're only differentiated by CSS left-margin in a variety of units? How do you interpret documents that render properly but are hardly correct HTML?

Thanks, I guess, none of that stuff seemed super useful to cut systematically, but I'm gonna run some tests.

Re: Classifying all of the pdfs on the internet

#88

Post by nativeit »

llm_trw wrote: post #28

Back in 2006 there were multiple 1tb collections of textbooks as torrents. I imagine the size and number has only grown since then.

I personally have about 350GB worth of old service manuals, data sheets, catalogs, and periodicals. Mostly related to electronics and engineering. All from torrent sources from ~2-years ago (when I wanted to mess with GraphQL and some OSR resources).

Re: Classifying all of the pdfs on the internet

#89

Post by josh-sematic »

Very cool! At Airtrain we’ve also found embeddings can be very valuable for building classification models. If you’re looking to play around with a large amount of text and embeddings we actually recently deduped and embedded all of fineweb-edu (also mentioned in the article) and put the resulting dataset on Hugging Face: https://huggingface.co/datasets/airtrain-ai/fineweb-edu-fort...

Re: Classifying all of the pdfs on the internet

#90

Post by CaptainFever »

loa_in_ wrote: post #84
namrog84 wrote: post #52

Earlier quoted context omitted.

That was before hoarding and building questionable businesses around them became a thing. I remember it being really easy to find textbooks, solution manuals, and related pdf and other stuff as late as 2008 far easier than 6-8 years later. The main difference were sites like chegg and many other sites started slurping them up to resell in some way.

It doesn't take away the torrents, no?

One of my pet peeves is the way people use words like slurp, hoover, take, vaccuum, suck up, or steal, when in reality they mean copy.

I mean if Chegg manages to sell something you can get for free, then all the more power to them lol. Though we could probably do more to educate the younger generation on the magic of torrents. Ignoring angry textbook publishers, of course.

Post reply on HN