Looks interesting. Customers of our data wrangling tool (Easy Data Transform) are asking to be able to pull data out of REST APIs, but we don't currently support this. So it could be an interesting way to bridge the gap.
Show HN: Crul – Query Any Webpage or API
21–30 of 81 posts
Re: Show HN: Crul – Query Any Webpage or API
#22Looks interesting. Customers of our data wrangling tool (Easy Data Transform) are asking to be able to pull data out of REST APIs, but we don't currently support this. So it could be an interesting way to bridge the gap.
Would love to chat and try a few use cases together! At first glance I see you can drag in csv files, which could be generated by crul and either manually downloaded or scheduled and written to the filesystem.
Crul is a really easy way of populating tools that need data, whether it's just a one time thing for a demo/static data set or a scheduled data feed, so this kind of usage makes sense to us.
Re: Show HN: Crul – Query Any Webpage or API
#23Looks interesting. Customers of our data wrangling tool (Easy Data Transform) are asking to be able to pull data out of REST APIs, but we don't currently support this. So it could be an interesting way to bridge the gap.
Would love to chat and try a few use cases together! At first glance I see you can drag in csv files, which could be generated by crul and either manually downloaded or scheduled and written to the filesystem. Crul is a really easy way of populating tools that need data, whether it's just a one time thing for a demo/static data set or a scheduled data feed, so this kind of usage makes sense to us.
That might be interesting. I will try to find some time to have a play with Crul. Do you support pulling data from an API on a schedule?
Re: Show HN: Crul – Query Any Webpage or API
#24Earlier quoted context omitted.
Would love to chat and try a few use cases together! At first glance I see you can drag in csv files, which could be generated by crul and either manually downloaded or scheduled and written to the filesystem. Crul is a really easy way of populating tools that need data, whether it's just a one time thing for a demo/static data set or a scheduled data feed, so this kind of usage makes sense to us.
That might be interesting. I will try to find some time to have a play with Crul. Do you support pulling data from an API on a schedule?
Yes that is possible, although we currently have the scheduler set as an enterprise feature. We should look into a free trial, I enjoyed the flow of the EasyDataTransform installation with the free trial option.
Re: Show HN: Crul – Query Any Webpage or API
#25Earlier quoted context omitted.
It's a tricky question. Part of it is looking at APIs as the main source of data for scheduled queries/data feeds. Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and open up multiple tabs it becomes a little more challenging. We have the concept of domain policies which allow you to control how quickly/slowly you access something. There are also some pu…
In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.
In my experience, the thing that makes me actually have to lug out my ol' headless browser is that a lot of websites are starting to implement obfuscated cryptographic puzzles in their JS, making it really difficult to emulate without just running it in a browser.
Re: Show HN: Crul – Query Any Webpage or API
#26Earlier quoted context omitted.
That might be interesting. I will try to find some time to have a play with Crul. Do you support pulling data from an API on a schedule?
Yes that is possible, although we currently have the scheduler set as an enterprise feature. We should look into a free trial, I enjoyed the flow of the EasyDataTransform installation with the free trial option.
>I enjoyed the flow of the EasyDataTransform installation with the free trial option
We're taking a different apporach to our enterprisey competitiors. ;0)
>we currently have the scheduler set as an enterprise feature.
Understandable.
Re: Show HN: Crul – Query Any Webpage or API
#27> status: complete (28.284 seconds / 14 results) Why was it so slow? Is there a default delay between requests or something?
Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature. We found that it becomes too easy to break API request limits or spam a website otherwise. However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering o…
> due to the caching layers
Every time I see that, the "2 hardest things" springs to mind. Is there a clear-caches option, or I guess the opposite question: does that process honor the HTTP caching semantics? Scrapy actually has a bunch of configurable knobs for that (use RFC2616 Policy ( https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... ), write your own policy, or a ton of other stuff: https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... )
Re: Show HN: Crul – Query Any Webpage or API
#28Earlier quoted context omitted.
Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature. We found that it becomes too easy to break API request limits or spam a website otherwise. However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering o…
> due to the caching layers Every time I see that, the "2 hardest things" springs to mind. Is there a clear-caches option, or I guess the opposite question: does that process honor the HTTP caching semantics? Scrapy actually has a bunch of configurable knobs for that (use RFC2616 Policy ( https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... ), write your own policy, or a ton of other stuff: https://docs.sc…
Agreed, caching does come with its own set of quirks and mind-numbing bugs, crul does have a caching override flag at the command/stage level which alleviates some of this: https://www.crul.com/docs/queryconcepts/common-flags#--cache
Your provided links are interesting and something for us think about some more. Honestly, I would be quite interested in hearing more about your experiences.
Re: Show HN: Crul – Query Any Webpage or API
#29Earlier quoted context omitted.
In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.
In my experience, the thing that makes me actually have to lug out my ol' headless browser is that a lot of websites are starting to implement obfuscated cryptographic puzzles in their JS, making it really difficult to emulate without just running it in a browser.
Starting to? That's been going on for at least the last 3-4 years. Akamai tends to rely on that more than Cloudflare and in my recent experience Akamai is winning that game and browser emulation alone, headless or not, isn't going to bypass Akamai. Recently I have seen browser emulation not be effective at all for bypassing bot detection.
The only kind of emulation where I have seen success is mobile and in that case you need to run a device emulator.
Re: Show HN: Crul – Query Any Webpage or API
#30That's a pretty slick tool you guys built!