Trafilatura: Python tool to gather text on the Web
1–10 of 25 posts
Re: Trafilatura: Python tool to gather text on the Web
#2Cool tool, I used it for a scrapinh project and it performed quite well for extracting clean text and the date.
Re: Trafilatura: Python tool to gather text on the Web
#3I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.
Re: Trafilatura: Python tool to gather text on the Web
#4I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.
Archive box (https://archivebox.io/) will create a local dump of any site in a multitude of formats from raw html, printed PDF, and extracted body text. Also has option to request internet archive to trigger a scrape of the page.
Re: Trafilatura: Python tool to gather text on the Web
#5I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.
check out nghota.com api. It is able to pull out the main text from most non-ecommerce web pages and return that to you in json.
Re: Trafilatura: Python tool to gather text on the Web
#6What is the gap between this and beautiful soup?
Re: Trafilatura: Python tool to gather text on the Web
#7I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.
You sort of, kind of, maybe just asked for roughly what RSS (Really Simple Syndication) provides...although your wish is more of a "pull", while RSS is more of a "push" in content access/distribution. :-) Don't get me wrong, I'm in agreement with you. I wish every website, web app, well, pretty much everything digital had an automated RSS feed available to consume and subscribe to!
Re: Trafilatura: Python tool to gather text on the Web
#8What is the gap between this and beautiful soup?
Maybe bs4 + newspaper3k rolled into one? But still, what's the gap?
Re: Trafilatura: Python tool to gather text on the Web
#9I wish there was a web service that used this tool to scrape nicely-formatted plain text from any website, then archive it and serve it as a super basic web reader.
check out nghota.com api. It is able to pull out the main text from most non-ecommerce web pages and return that to you in json.
In general I'd be curious to try this, but your homepage is not very convincing.
The "demo" doesn't look like typing, it's a fade right, and it's painfully slow. And then, there's no library, it's just 'import requests', so even the demo is extra long. (Why not show curl then?)
Also, are there any benchmarks? Why should I take the time to evaluate this myself against existing open-source tools? It seems like that should be your responsibility, not mine, to spend the time doing a detailed comparison and evaluation. In a way that feels open and trustworthy.
I respect what you are doing and share this feedback from the heart.
Re: Trafilatura: Python tool to gather text on the Web
#10What is the gap between this and beautiful soup?
The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features
Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.