Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trafilatura: Python tool to gather text on the Web

github.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Trafilatura: Python tool to gather text on the Web

#11

Post by rmbyrro »

evanmcgillivray wrote: post #6

What is the gap between this and beautiful soup?

This tool can extract data in a structured format from virtually any website, with any HTML structure.

With Beautiful Soup, you'd need to explicitly tell where each piece of data exists referencing HTML tags, ids, classes, etc. For each website you'd want to process.

Re: Trafilatura: Python tool to gather text on the Web

#13

Post by dleeftink »

dominick-cc wrote: post #3

I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.

Not sure how it fares nowadays, but I used to employ Mercury Reader/API for this, now called Postlight Reader[1]. While not perfect, I found it to work for most daily reading needs.

[1]: https://reader.postlight.com/

Re: Trafilatura: Python tool to gather text on the Web

#14

Post by bnewbold »

This tool is so great for robustly dealing with content in old and poorly formatted HTML. There are a lot of similar tools for extracting "the main text" from free-form HTML, but this was the most reliable in my experience, especially when dealing with web archives containing hand-written HTML back to the 1990s, working with non-English languages, etc.

Re: Trafilatura: Python tool to gather text on the Web

#16

Post by selcuka »

dominick-cc wrote: post #3

I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.

Sounds trivial to implement using this library with a bit of glue code for the web bits:

https://archiw.fly.dev/

Re: Trafilatura: Python tool to gather text on the Web

#17

Post by jamil7 »

dominick-cc wrote: post #3

I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.

There’s been a few such things over the years. I even built one for iOS/iPad that’s still in the store. I found that doing the parsing client side is preferable because so many sites have paywalls and render some of their content with JS. I never did much with the app because it’s hard to monetize, but I maintain it occasionally.

Re: Trafilatura: Python tool to gather text on the Web

#18

Post by lou1306 »

mxuribe wrote: post #7
dominick-cc wrote: post #3

I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.

You sort of, kind of, maybe just asked for roughly what RSS (Really Simple Syndication) provides...although your wish is more of a "pull", while RSS is more of a "push" in content access/distribution. :-) Don't get me wrong, I'm in agreement with you. I wish every website, web app, well, pretty much everything digital had an automated RSS feed available to consume and subscribe to!

With RSS you are at the mercy of the server, though. The content creator may only syndicate an excerpt of the whole article, remove pictures or formatting, yada yada. But yes the Web would be so much nicer if more websites provided at least some form of content syndication...

Re: Trafilatura: Python tool to gather text on the Web

#20

Post by adbarba »

dmillar wrote: post #8
evanmcgillivray wrote: post #6

What is the gap between this and beautiful soup?

Maybe bs4 + newspaper3k rolled into one? But still, what's the gap?

Regarding content extraction it's more accurate than newspaper3k (especially for languages other than English) and it entails more information: metadata, text, and comments. It works out of the box in most cases so no need to write a particular scraper for a given websites, which saves time. If you care about 2-3 websites and are willing to write and maintain scraping scripts then bs4/lxml/whatever is also fine.

It also features functions and a command-line interface to collect data on your own (say find recent news using feeds). So it's not merely about text extraction in the end but also text discovery.

Post reply on HN