How do you plan on tackling anti-bot blocking?
You shouldn't
Show HN: Crul – Query Any Webpage or API
11–20 of 81 posts
Re: Show HN: Crul – Query Any Webpage or API
#12Ah, I see it's pronounced more like "Krull" than "cruel".
Re: Show HN: Crul – Query Any Webpage or API
#13Ah, I see it's pronounced more like "Krull" than "cruel".
lol - yeah. https://www.imdb.com/title/tt0085811/?ref_=tt_urv
We've been thinking of ways to make the pronunciation a bit clearer, maybe a mascot or something. Open to ideas!
Re: Show HN: Crul – Query Any Webpage or API
#14How does crul handle the dynamic nature of the web? Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is renamed or moved as part of a site redesign, what happens? I've worked on a bunch of tools in the past that do similar things, and structural changes were the kryptonite for all of them. A secondary problem is when you use particular content as…
At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break. We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work. But the expanding and querying concepts also…
IMO, this is the hardest part of maintaining a web scraper. We had ~100 scripts to scrape ~1000 clients' sites and it was, at minimum, 50 hours a week to keep up with changes.
The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.
Re: Show HN: Crul – Query Any Webpage or API
#15Earlier quoted context omitted.
At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break. We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work. But the expanding and querying concepts also…
IMO, this is the hardest part of maintaining a web scraper. We had ~100 scripts to scrape ~1000 clients' sites and it was, at minimum, 50 hours a week to keep up with changes. The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.
This makes sense and I am curious about this. Was there consistency between those 1k client sites or were they all rather different? Mind if I reach out?
Re: Show HN: Crul – Query Any Webpage or API
#16How do you plan on tackling anti-bot blocking?
It's a tricky question. Part of it is looking at APIs as the main source of data for scheduled queries/data feeds. Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and open up multiple tabs it becomes a little more challenging. We have the concept of domain policies which allow you to control how quickly/slowly you access something. There are also some pu…
In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.
Re: Show HN: Crul – Query Any Webpage or API
#17Really elegant work guys. You've identified the pain points well. I'd be curious how you've structured your infrastructure to scale though. But I imagine that's secret sauce.
What are some of your experiences with async networking, btw? Did you find that it didn't really handle concurrency well in practice? Or were there other issues? I've written a lot of async networking code but always found it horrible to profile.
Re: Show HN: Crul – Query Any Webpage or API
#18> status: complete (28.284 seconds / 14 results)
Why was it so slow? Is there a default delay between requests or something?
Re: Show HN: Crul – Query Any Webpage or API
#19Really elegant work guys. You've identified the pain points well. I'd be curious how you've structured your infrastructure to scale though. But I imagine that's secret sauce. What are some of your experiences with async networking, btw? Did you find that it didn't really handle concurrency well in practice? Or were there other issues? I've written a lot of async networking code but always found it horrible to profile…
Thank you and your comment really warms our hearts. It's been fun building in the "cave" but comes with self doubt.
We've built using a microservice architecture to allow us to scale out the parts that need to scale, mainly the workers, which interact with a queue, although we'll need to move from the parts that are currently nodejs for some perf gains and a smaller footprint. All those microservices are consolidated for the desktop variants.
Network concurrency is mostly throttled by our domain policy manager (named "gonogo" - lol) at 1 req/per outbound domain a sec. It's a little slow for a default, but also configurable and provides a nice guardrail for api request limits, etc. Overall async networking has been quite tricky, esp with retries, etc., and we're still iterating on it. Agreed on the profiling difficulties.
Re: Show HN: Crul – Query Any Webpage or API
#20> status: complete (28.284 seconds / 14 results) Why was it so slow? Is there a default delay between requests or something?
Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature.
We found that it becomes too easy to break API request limits or spam a website otherwise.
However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering of the data part is smoother/faster.