Amazing I was just looking for something like this to be able to import web page content into Whisperit
Show HN: Pulpie – Models for Cleaning the Web
21–30 of 38 posts
Re: Show HN: Pulpie – Models for Cleaning the Web
#22How does this work on pages that require JavaScript in order to render?
You'd typically use a headless browser to generate the fully rendered page, then capture the rendered output for use with the model.
Re: Show HN: Pulpie – Models for Cleaning the Web
#23Does it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
Re: Show HN: Pulpie – Models for Cleaning the Web
#24Does it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
Yes. I tried it with https://www.allbirds.com/products/womens-cruiser-canvas on our HF space and Pulpie worked great.
Re: Show HN: Pulpie – Models for Cleaning the Web
#25Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
We see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.
I think instead of "performance" you must mean "accuracy". Traditional deterministic conversion will always be faster and cheaper than running through a model, even if it is less accurate.
Re: Show HN: Pulpie – Models for Cleaning the Web
#26Does it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
Yes. I tried it with https://www.allbirds.com/products/womens-cruiser-canvas on our HF space and Pulpie worked great. HF Space: https://huggingface.co/spaces/feyninc/pulpie
Well allbirds is an AI company now so I guess that makes sense
Re: Show HN: Pulpie – Models for Cleaning the Web
#27Earlier quoted context omitted.
We see far better performance with models. Heuristics break on richer content like codeblocks, formulae, quotes, etc. In our testing, our model was 25 F1 points better than Trafilatura.
I think instead of "performance" you must mean "accuracy". Traditional deterministic conversion will always be faster and cheaper than running through a model, even if it is less accurate.
Ah yes, I meant accuracy.
Re: Show HN: Pulpie – Models for Cleaning the Web
#28Re: Show HN: Pulpie – Models for Cleaning the Web
#29This looks really cool
Re: Show HN: Pulpie – Models for Cleaning the Web
#30Very interesting