Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
Show HN: Pulpie – Models for Cleaning the Web
11–20 of 38 posts
Re: Show HN: Pulpie – Models for Cleaning the Web
#12Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
If I had to reckon, it's because the web comes in very many shapes, and outsourcing that work to a generalist LLM/SLM like GPT Nano is expensive, and doing it deterministically will never catch all the edge cases as well as a purpose-built encoder when run at webscale.
Re: Show HN: Pulpie – Models for Cleaning the Web
#13Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
Looks like they are including Trafilatura in the comparison tables, which I've used before with pretty decent results, but it still has trouble with some pages. Looks like the pulpie f1 scores are quite a bit better, especially for the hard cases.
Would be curious how it runs on more modest hardware though, I'm using it for a small bookmark archiving tool and being able to run it on my small mini-pc homelab would be nice.
Re: Show HN: Pulpie – Models for Cleaning the Web
#14Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
We see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.
Re: Show HN: Pulpie – Models for Cleaning the Web
#15So this is tailored towards kind of a "reader view" for models right? Can it handle images, tables, shadow DOMs too? Like there are 3 use cases I have now - one is a simple text view for models to understand it, one is a "web clip" mode which would ideally preserve images and media, and one is to extract tabular data from web pages. Which ones is this good at?
Re: Show HN: Pulpie – Models for Cleaning the Web
#16Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
We see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.
[dead]
Re: Show HN: Pulpie – Models for Cleaning the Web
#17How does this work on pages that require JavaScript in order to render?
Re: Show HN: Pulpie – Models for Cleaning the Web
#18So this is tailored towards kind of a "reader view" for models right? Can it handle images, tables, shadow DOMs too? Like there are 3 use cases I have now - one is a simple text view for models to understand it, one is a "web clip" mode which would ideally preserve images and media, and one is to extract tabular data from web pages. Which ones is this good at?
Images pass through as they are considered main content. Same with tables.
Pulpie will return all main content on a page as HTML/Markdown. I’m not sure I fully understand “which one this is good at?”. perhaps you can try the model on hugging face and let me know if the results look good?
Re: Show HN: Pulpie – Models for Cleaning the Web
#19I did some research on this about 10 years ago. I spent 2 days hand labelling data from scraped news sites. Then built a good old fashioned Random Forest model to classify html nodes based on some feature engineering. turns out the P tag and the number-of-words threshold get you 90% of the way there, on news sites anyway. Great thing about RF models is they tell you which features are the most important. fun little project (apart from the 2 days of data labelling).
Re: Show HN: Pulpie – Models for Cleaning the Web
#20[flagged]