121–130 of 177 posts
Using an LLM for the actual parsing, that's simultaneously overkill while risking your results being polluted with hallucinations.
Earlier quoted context omitted.
You must get a lot of test emails to that FANTASTIC gmail address. Funny how it might even be worth some decent money.
That's not literally his e-mail :D. He means that you have to replace it with his HN username. It would have been better to write it like this: [HN username]@gmail.com
OpenAI recently announced a Batch API [1] which allows you to prepare all prompts and then run them as a batch. This reduces costs as its just 50% the price. Used it a lot with GPT-4o mini in the past and was able to prompt 3000 Items in less than 5min. Could be great for non-realtime applications. [1] https://platform.openai.com/docs/guides/batch
For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…
The LLM is resistant to website updates that would break normal scraping If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the dat…
For structured content (e.g. lists of items, simple tables), you really don’t need LLMs. I recently built a web scraper to automatically work on any website [0] and built the initial version using AI, but I found that using heuristics based on element attributes and positioning ended up being faster, cheaper, and more accurate (no hallucinations!). For most websites, the non-AI approach works incredibly well so I’d m…
The LLM is resistant to website updates that would break normal scraping If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the dat…
> Turns out, a simple table from Wikipedia (Human development index) breaks the model because rows with repeated values are merged
As others have mentioned here you might get better results cheaper (this probably wasn't the point of the article, so just fyi) if you preprocess the html first. I personally have had good results with trafilatura[1], which I don't see mentioned yet. [1] https://trafilatura.readthedocs.io/en/latest/
I second trafilatura greatly. This will save a huge amount of money to just send the text to the LLM. I used it on this recent project (shameless plug): https://github.com/philippe2803/contentmap . It's a simple python library that creates a vector store for any website, using a domain XML sitemap as a starting point. The challenge was that each domain has its own HTML structure, and to create a vector store, we need…
I also forgot to mention another interesting scraper that's an LLM based service. A quick search here tells me it was mentioned once by simonw, but I think it should be better known just for the convenience! Prepend "r.jina.ai" to any URL to extract text. For ex., check out [2] or [3].
[1] https://aclanthology.org/2021.acl-demo.15.pdf
[2] https://r.jina.ai/news.ycombinator.com/
[3] (this discussion) https://r.jina.ai/news.ycombinator.com/item?id=41428274
Earlier quoted context omitted.
If you haven't considered it, you can also use the direct wikitext markup, from which the HTML is derived. Depending on how you use it, the wikitext may or may not be more ingestible if you're passing it through to an LLM anyway. You may also be able to pare it down a bit by heading/section so that you can reduce it do only sections that are likely to be relevant (eg. "Life and career") type sections. You can also do…
> reduce it do only sections that are likely to be relevant (eg. "Life and career") True but I also managed to do this from HTML. I tried getting pages wikitext through the API but couldn't find how to. Just querying the HTML page was less friction and fast enough that I didn't need a dump (although when AI becomes cheap enough, there is probably a lot of things to do from a wikipedia dump!). One advantage of using o…
Earlier quoted context omitted.
Only works insofar as sites are being nice. A lot of sites do things like: render all text via JS, render article text via API, paywall content by showing a preview snippet of static text before swapping it for the full text (which lives in a different element), lazyload images, lazyload text, etc etc. DOM parsing wasn't enough for Google's SEO algo, either. I'll even see Safari's "reader mode" fail utterly on site a…
It's possible to capture the DOM by running a headless browser (i.e. with chromedriver/geckodriver), allowing the js execute and then saving the HTML. If these readers do not use already rendered HTML to parse the information on the screen, then...
It's a big web out there, there's even more heinous stuff. Even identifying what the main content is can be a challenge.
And reader mode has the benefit of being ran by the user. Identifying when to run a page-simplifying action on some headlessly loaded URL can be tricky. I imagine it would need to be like: load URL, await load event, scroll to bottom of page, wait for the network to be idle (and possibly for long tasks/animations to finish, too)