Live data from Hacker News

Scraperr – A Self Hosted Webscraper

github.com

91–96 of 96 posts

Re: Scraperr – A Self Hosted Webscraper

#91

Earlier quoted context omitted.

With Pascal being my first "adult" language, not used in 20 years ... it is surprising how readable that code is. Makes me wish for such simpler times.

that fits, I wrote the first code for xidel almost 20 years ago and it still uses Pascal because I didn't plan to change it but just wanted to show people what I programmed 20 years ago

What environment are you using nowadays to develop in Pascal? Editor, possible plugins etc?

Re: Scraperr – A Self Hosted Webscraper

#92
post #87
post #52

Earlier quoted context omitted.

You need to drive and fine tune a Ferrari because it feeds your family. The OP just drive a nice little car, because it is fun to drive and he enjoys it. He could extract another +5% of torque by fine tuning, but he does not care, this is not where his joy is and where he wants to spend his time.

To be completely fair to the person you're responding to, they're talking about pages that take 20 seconds to load. On a regular website that hopes to get visitors from search engines, say, that is indeed insane and the fix is not to squeeze out 5% by fine tuning, the fix is to re-architect the thing I don't mind people asking why it is this slow (whether I can't or why I don't re-architect it) or suggest fixes, but…

Well said! At work, I deal with these kinds of issues, and they get messy. I've spent hours this week discussing and debating table indexes and caching parameters for a system that's been running for fifteen years but we think we can improve. There's a diminishing return to putting tons of your time into every little thing, especially when a project is not your livelihood. It sounds like you've taken a thoughtful approach to your system architecture. If it works for you, it works!

Re: Scraperr – A Self Hosted Webscraper

#93
post #86

Earlier quoted context omitted.

Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…

Right... In case it helps to understand: compare it to something like weather models. You can't "just add indexes" to make it fast, but you can store the result of an hourly run and serve that to people in milliseconds. In my case, nobody's paying me to serve them that 'weather report' so it is what it is > Like I said then, you need indexes on those columns Dude, like I said ("that's not a correct assumption" in res…

But but my GCP-PDE exam said every project needs massive and expensive infrastructure to optimize every tiny detail, minimize latency, and deliver five nines of availability! If your cloud bill isn't five figures every month, are you even a dev? /s

Re: Scraperr – A Self Hosted Webscraper

#94

Earlier quoted context omitted.

What sort of pages require 20 seconds to generate? This is extremely slow by most web standards and even your users would be frustrated by this. It sounds like poorly designed database queries with unindexed joins. Google will also abandon page loads that take too long, and will demote rankings for that page (or the entire site!)

So what about flight searches where we have to query several 3rd party providers, and can take 45 seconds to get results from all of them (out of my control). I can dynamically update the page (and do) but a scraper would have to wait 20-45 seconds to get the 'cheapest' flight from my site. I can add async the queries and have the fastest pipes, but if the upstream providers take their time (they need to query their…

Oof, reminds me of the data pipelines I maintain that pull data from Salesforce. I've optimized the heck out of all of our internal database queries and they're blazing fast. Then my pipeline has to wait patiently for the SOQL response.

Re: Scraperr – A Self Hosted Webscraper

#95
post #68

Earlier quoted context omitted.

Thank you, I'll experiment with that. Tips and advice welcome!

Another cool trick is to deny all the content types you don't care about in your playwright. so if you only want text why bother allowing requests for fonts, css, svgs, images, videos, etc Just request the html and cap down all the other stuff PS: I also think this has the nice side-effect of you consuming less resources (that you didnt care about/need anyways) from the server, so win win

That is a great tip, thank you!

Re: Scraperr – A Self Hosted Webscraper

#96

Earlier quoted context omitted.

that fits, I wrote the first code for xidel almost 20 years ago and it still uses Pascal because I didn't plan to change it but just wanted to show people what I programmed 20 years ago

What environment are you using nowadays to develop in Pascal? Editor, possible plugins etc?

FreePascal / Lazarus

There was actually just a large discussion thread in the Lazarus forum wondering about why it is not more popular, with almost 300 comments. But now the thread got locked. That might be a reason.

I used Delphi on Windows 98. But that became outdated, so I ported it to FreePascal. FreePascal has a lot of integrated libraries, but I do not really use anything I did not already use with Delphi.

Post reply on HN