Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trafilatura: Python tool to gather text on the Web

github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Trafilatura: Python tool to gather text on the Web

#21

Post by adbarba »

dleeftink wrote: post #13
dominick-cc wrote: post #3

I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.

Not sure how it fares nowadays, but I used to employ Mercury Reader/API for this, now called Postlight Reader[1]. While not perfect, I found it to work for most daily reading needs. [1]: https://reader.postlight.com/

Concerning tooling I'd say you have two different worlds, JavaScript and Python, each with a series of tools to tackle such tasks. It's not easy to compare them directly because of varying software environments and I haven't had a chance to test JS tools thoroughly.

For the sake of completeness: Mozilla's Readability [1] is obviously a reference in the JS world.

[1]: https://github.com/mozilla/readability

Re: Trafilatura: Python tool to gather text on the Web

#22

Post by mxuribe »

lou1306 wrote: post #18
mxuribe wrote: post #7

Earlier quoted context omitted.

You sort of, kind of, maybe just asked for roughly what RSS (Really Simple Syndication) provides...although your wish is more of a "pull", while RSS is more of a "push" in content access/distribution. :-) Don't get me wrong, I'm in agreement with you. I wish every website, web app, well, pretty much everything digital had an automated RSS feed available to consume and subscribe to!

With RSS you are at the mercy of the server, though. The content creator may only syndicate an excerpt of the whole article, remove pictures or formatting, yada yada. But yes the Web would be so much nicer if more websites provided at least some form of content syndication...

Agreed, one would definitely be at the mercy of the author/content creator...but I often feel like someone who is willing to offer an RSS feed probably would likely enable easier consumption of their content even if one needs to actually visit the website. Of course tthat is a very broad generalization I'm making. But you're certainly not wrong.

Re: Trafilatura: Python tool to gather text on the Web

#23

Post by d4rkp4ttern »

adbarba wrote: post #19

Author here, nice to see the package on the HN's front page this morning and thanks for the kind words! Just created an account to participate in the discussion, I'll try to answer your questions.

I’ve been using this package and like it a lot.

One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website. I’ve not found a satisfactory packaged solution for this. Clearly a tough problem in general but wondered if people have found good libs to help with this. I’ve heard of solutions involving playwright etc.

Re: Trafilatura: Python tool to gather text on the Web

#24

Post by adbarba »

d4rkp4ttern wrote: post #23
adbarba wrote: post #19

Author here, nice to see the package on the HN's front page this morning and thanks for the kind words! Just created an account to participate in the discussion, I'll try to answer your questions.

I’ve been using this package and like it a lot. One problem I’d like to find a solution for is how to get past cookie pop ups when scraping a website. I’ve not found a satisfactory packaged solution for this. Clearly a tough problem in general but wondered if people have found good libs to help with this. I’ve heard of solutions involving playwright etc.

Thanks! Here is what I put together in the docs, you could basically preprocess/render/filter the webpages with the software of your choice and then pass the result to trafilatura: https://trafilatura.readthedocs.io/en/latest/troubleshooting...
Post reply on HN