Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crul – Query Any Webpage or API

crul.com

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Show HN: Crul – Query Any Webpage or API

#41

Post by portInit »

KomoD wrote: post #36
portInit wrote: post #20

Earlier quoted context omitted.

Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature. We found that it becomes too easy to break API request limits or spam a website otherwise. However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering o…

> although currently an enterprise feature. Wait, so we literally can't go faster than 1 req/s unless we pay? I have to say I'm pretty disappointed :/

Sorry to hear that - we do need to think about this. It's our first pass at product tiers and features and we may need to adjust.

Scheduling and Domain Policies were the main features we chose to gate initially as they don't affect core functionality other than performance and deployment.

Re: Show HN: Crul – Query Any Webpage or API

#42

Post by nine_k »

1vuio0pswjnm7 wrote: post #34

Please share some examples of webpages with data to be wrangled that support this statement: "The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues." Every web user's needs are different. One person might have a task that they struggle to accomplish while another might have one that presents no major challenges. As a web user…

Do you have an easy way to transform e.g. a typical Amazon product page into nicely structured data? Not that it's trying to be very video-gamey.

Re: Show HN: Crul – Query Any Webpage or API

#44

Post by portInit »

codetrotter wrote: post #43

This is very cool! Does the Docker image contain everything, or does part of it rely on server side things hosted by crul?

Thank you! Docker image has everything for functionality, there's just an update check that pings our end to check for updates.

Re: Show HN: Crul – Query Any Webpage or API

#45

Post by ElijahLynn »

I just signed up and the email confirmation ended up at http://links.outseta.com/ls/click?upn=xKo-2FU5fxLX67yddEnUva... which I get a page that says "Malware and Phishing This site is blocked because it is a known security threat. Please contact your network administrator to gain access."

I can't tell if the message is from Brave or from the ISP though.

Re: Show HN: Crul – Query Any Webpage or API

#46

Post by ElijahLynn »

ElijahLynn wrote: post #45

I just signed up and the email confirmation ended up at http://links.outseta.com/ls/click?upn=xKo-2FU5fxLX67yddEnUva... which I get a page that says "Malware and Phishing This site is blocked because it is a known security threat. Please contact your network administrator to gain access." I can't tell if the message is from Brave or from the ISP though.

Just tried on mobile hotspot, same message. And on Firefox, same message.

Re: Show HN: Crul – Query Any Webpage or API

#48

Post by 1vuio0pswjnm7 »

nine_k wrote: post #42
1vuio0pswjnm7 wrote: post #34

Please share some examples of webpages with data to be wrangled that support this statement: "The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues." Every web user's needs are different. One person might have a task that they struggle to accomplish while another might have one that presents no major challenges. As a web user…

Do you have an easy way to transform e.g. a typical Amazon product page into nicely structured data? Not that it's trying to be very video-gamey.

What should the structure look like. If it is CSV what are the columns, i.e., what specific data does it need include.

Taking a quick look at the Amazon site these product pages appear to be enormous in size. Interestingly, the website requires a "viewport-width" header. Otherwise one gets directed to a CAPTCHA.

The product page I checked already has some structered data in the form of JSON, including keys such as

   "title":"xxxxxxxxxx"
   "displayPrice":"$000.00"
   "priceAmount":000.00
   "currencySymbol":"$"
   "integerValue":"000"
   "decimalSeparator":"."
   "fractionalValue":"00"
   "symbolPosition":"left"
   "asin": "xxxxxxxxx"
   "asin":"xxxxxxxxxx"
   "acAsin":"xxxxxxxxxx"
   "buyingOptionTypes":["NEW"]
   "productAsin":"xxxxxxxxxx"
   "mediaAsin":"xxxxxxxxxxx"
   "parentAsin":"xxxxxxxxx"
   "asinList":"xxxxxxxxxx"
Thus, CSV with product name, price and ASIN would appear to be easy. No need to mess with the HTML.

Other data such as, e.g., delivery time, seller, where the item ships from and number left in stck can be extracted from the HTML.

Delivery time is in a that contains "data-csa-c-delivery-time".

Seller, shipping info and number left in stock are under a with class="a-size-base _p13n-desktop-sims-fbt_fbt-desktop_shipping-info-show-box__17yWM"

One needs to decide what data one wants from the page.

The way to present an example on which to evaluate a "new" solution such as the one in this thread is to present a problem, e.g.,

Get data items x, y and z from website xyzexample.com.

In the majorty of cases I see submitted to HN, it is impossible to benchmark these "new" solutions against existing ones because no example websites are ever provided.

Re: Show HN: Crul – Query Any Webpage or API

#49

Post by portInit »

ElijahLynn wrote: post #46
ElijahLynn wrote: post #45

I just signed up and the email confirmation ended up at http://links.outseta.com/ls/click?upn=xKo-2FU5fxLX67yddEnUva... which I get a page that says "Malware and Phishing This site is blocked because it is a known security threat. Please contact your network administrator to gain access." I can't tell if the message is from Brave or from the ISP though.

Just tried on mobile hotspot, same message. And on Firefox, same message.

That's unfortunate. A quick look suggests ISP but maybe not with the hotspot. Will do a little more digging.

Re: Show HN: Crul – Query Any Webpage or API

#50

Post by dan_rock_wilson »

chatmasta wrote: post #37
portInit wrote: post #7

Earlier quoted context omitted.

Will absolutely reach out! Our experience has been that just getting data is often really challenging, so we've really focused on that piece, and being able to easily share with destinations that are purpose built for analytics, viz, etc. Thanks for checking it out!

I'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could connect to postgres://data.splitgraph.com:5432 and SELECT from a table backed by crul (either "mounted" for live querying, and/or ingested once/periodically for subsequent querying). The user just needs to provide parameters for the table; it'…

How did you integrate with all those services (the third party service APIs, not asking about the DBs)? I've seen a bunch of sites do this and I'm curious if there's some open source library I should be using, so that I don't need to write from scratch each time I'd like to integrate with another service.

Edit: just saw "airbyte" in a bunch of places, which I assume answers my question. So updated question: airbyte works well for ya?

Post reply on HN