Live data from Hacker News

Classifying all of the pdfs on the internet

snats.xyz

91–100 of 117 posts

Re: Classifying all of the pdfs on the internet

#91

Did some similar work with similar visualizations ~2009, on ~5.7M research articles (PDFs, private corpus) from scientific publishers Elsevier, Springer: Newton, G., A. Callahan & M. Dumontier. 2009. Semantic Journal Mapping for Search Visualization in a Large Scale Article Digital Library. Second Workshop on Very Large Digital Libraries at the European Conference on Digital Libraries (ECDL) 2009. https://lekythos.li…

How do you decide who is listed first? And does the ampersand symbolize something that the word and doesn't, or is that just citation style?

Re: Classifying all of the pdfs on the internet

#92

This seems like cool work but with a ton of "marketing hype speak" that immediately gets watered down by the first paragraph. Ordering of statements. 1. (Title) Classifying all of the pdfs on the internet 2. (First Paragraph) Well not all, but all the PDFs in Common Crawl 3. (First Image) Well not all of them, but 500k of them. I am not knocking the project, but while categorizing 500k PDFs is something we couldnt ne…

Moreover, the classification was not done on 500,000 PDF files themselves, but rather on the metadata of those 500,000 PDFs.

Re: Classifying all of the pdfs on the internet

#93
post #39
post #15

Earlier quoted context omitted.

RTBF was a ludicrous concept before AI and these new crawlers. Only EU bureaucracts would have the hubris to believe you could actually, comprehensively remove information from the Internet. Once something is spread, it is there, forever.

> Once something is spread, it is there, forever. Really depends on the content. Tons of websites are going down everyday, link rot is a real thing. Internet archive or people don't save nearly everything. Something I should do more often is saving mhtml copies of webpages I find interesting.

  > Something I should do more often is saving mhtml copies of webpages I find interesting.
They consume so much disc space. I wish that their was some intermediate format that would have a file size only two orders of magnitude larger than the webpage text, yet provide enough formatting to be useful.

Re: Classifying all of the pdfs on the internet

#96
post #94
post #55

Hi! Author here, I wasn't expecting this to be at the top of HN, AMA

Hi snats, great article. You mention the accuracy of the various techniques you used, could you explain more about how you calculated the accuracy? Were the pdfs already categorized? Thanks!

hi! i used the average accuracy over the entire dataset made originally made by the llm

Re: Classifying all of the pdfs on the internet

#97

One of the now-underdiscussed features of embeddings is that you can indeed use any existing statistical modeling techniques on them out of the box, and as a bonus avoid the common NLP preprocessing nuances and pitfalls (e.g. stemming) entirely. This post is a good example on why going straight to LLM embeddings for NLP is a pragmatic first step, especially for long documents.

You can apply statistical techniques to the embeddings themselves? How does that work?

Re: Classifying all of the pdfs on the internet

#98

One of the now-underdiscussed features of embeddings is that you can indeed use any existing statistical modeling techniques on them out of the box, and as a bonus avoid the common NLP preprocessing nuances and pitfalls (e.g. stemming) entirely. This post is a good example on why going straight to LLM embeddings for NLP is a pragmatic first step, especially for long documents.

You can apply statistical techniques to the embeddings themselves ? How does that work?

You can apply statistical techniques to anything you want. Embeddings are just vectors of numbers which capture some meaning, so statistical analysis of them will work fine.

Re: Classifying all of the pdfs on the internet

#99
post #98

Earlier quoted context omitted.

You can apply statistical techniques to the embeddings themselves ? How does that work?

You can apply statistical techniques to anything you want. Embeddings are just vectors of numbers which capture some meaning, so statistical analysis of them will work fine.

Don't most statistical techniques rely on specific structure in the spaces containing the objects they operate on, in order to be useful?

Re: Classifying all of the pdfs on the internet

#100
post #91

Did some similar work with similar visualizations ~2009, on ~5.7M research articles (PDFs, private corpus) from scientific publishers Elsevier, Springer: Newton, G., A. Callahan & M. Dumontier. 2009. Semantic Journal Mapping for Search Visualization in a Large Scale Article Digital Library. Second Workshop on Very Large Digital Libraries at the European Conference on Digital Libraries (ECDL) 2009. https://lekythos.li…

How do you decide who is listed first? And does the ampersand symbolize something that the word and doesn't, or is that just citation style?

In biomedical research or tangential fields, author order generally follows these guidelines:

First author(s): the individual(s) who organized and conducted the study. Typically there is only a single first author, but nowadays there are often two first authors. This is because the amount of research required to generate “high impact” publications simply can’t be done by a single person. Typically, the first author is a Ph.D. student or lab scientist.

Middle authors: Individuals that provide critical effort, help, feedback, or guidance for the study and publication of the research. Different fields/labs have varying stringencies for what is considered “middle authorship worthy”. In many labs, simply being present and helping with the research warrants authorship. In other labs, you need to contribute a lot of energy to the project to be included as an author.

Senior author(s): The primary investigators (PI’s) or lead researchers that run the lab that conducted and published the study. The senior authors are typically the ones that acquire funding and oversee all aspects of the published research. PI’s have varying degrees of hands-on management.

There is some variation in whether the central research question for a manuscript is developed by the first vs. the senior author, but usually it’s the senior author. Also, the first and senior authors typically write the manuscript and seek edits/feedback from middle authors. In other cases, there can be dedicated writers that write the manuscript, who sometimes do/don’t get middle authorship. A main takeaway is: the general outline I’ve provided above is not strictly adhered to.

I’ll take some liberty to apply this outline to this article at hand:

First Author: G. Newton (OP). The scientist who mostly likely conducted all of the data mining and analysis. He likely wrote the article as well.

Middle Author: A. Callahan. It seems like this author was a grad student at the time the article was written. She likely performed essential work for the paper’s publication. This could’ve been: helping with the analysis, data mining, or ideation.

Senior Author: M. Dumontier. A data science professor, now at Maastricht U. He’s a highly cited scientist!

Lastly… if you check out the acknowledgements, you can see three additional names. These people likely helped with setting up compute access, editing, or general ideation.

This is a cool manuscript! Hopefully this overview isn’t TMI and provides some insight into the biomedical/data science publication process.

Post reply on HN