Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crul – Query Any Webpage or API

crul.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Show HN: Crul – Query Any Webpage or API

#71

Post by portInit »

imiric wrote: post #69

This looks very cool. Congrats on the launch! Have you thought about open sourcing it? I realize you want to sustain a business, but an open core model with commercial features would allow you to do that. A free tool like this would be a welcome addition to the ecosystem, most of which is FOSS already, and would help build a community around your product.

We certainly have, it is admittedly quite new to both of us, so we have been exploring the best way to introduce something like this, as well as tying up technical bits. The open core model with commercial features is certainly appealing.

Open to any perspectives on this.

Re: Show HN: Crul – Query Any Webpage or API

#72

Post by djhn »

I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could speed up my workflow.

1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just the itemns? I found that it just gives me thousands of columns (and truncates the results), where I would have wanted, say, two thousand rows of 23 columns. I couldn't wrangle the JSON within crul.

2. Many smaller sites still use wordpress, and may manually lay out items in a visual hierarchy that isn't immediately obvious in the structure of the HTML. Then you have to go in and parse every "row" and every "column" using xpath or css selectors. Crul wasn't particularly helpful for this either.

3. A scraping workflow requiring authentication, post request for searching, get request for picking the correct result, post request for downloading pdf and parsing said pdf... well, I couldn't get this to work at all.

Hope you'll end up solving such problems automagically and I end up paying you for it.

Re: Show HN: Crul – Query Any Webpage or API

#73

Post by portInit »

djhn wrote: post #72

I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could speed up my workflow. 1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just…

Appreciate you taking the time to play around with crul and share your thoughts! They're incredibly valuable.

1. Although it has limitations, were you able to try the normalize command? https://www.crul.com/docs/queryconcepts/api-normalization

2. Will need to think about this some more.

3. Although we don't yet handle pdfs, the rest of the flow is one we aiming to accomplish with crul. Other than pdf, the pieces should be there and would love to understand this further

Re: Show HN: Crul – Query Any Webpage or API

#74

Post by yevpats »

Congrats team! Getting data from and to places is a fundamental Computer Science challenge (or in marketing terms ELT) that involves mostly data interoperability but also things like rate limits, transformations, horizontal and vertical scaling, schema definitions, incremental syncs, at least one delivery guarantee. Those are rarely things that can be automatically defined and needs user configuration - so Im really curious how you solve it under the hood or is it only working for subset of problems?

Asking all those as we are working on CloudQuery (https://github.com/cloudquery/cloudquery) so been dealing with a lot of those underlying challenges as well.

Re: Show HN: Crul – Query Any Webpage or API

#75

Post by imiric »

portInit wrote: post #71
imiric wrote: post #69

This looks very cool. Congrats on the launch! Have you thought about open sourcing it? I realize you want to sustain a business, but an open core model with commercial features would allow you to do that. A free tool like this would be a welcome addition to the ecosystem, most of which is FOSS already, and would help build a community around your product.

We certainly have, it is admittedly quite new to both of us, so we have been exploring the best way to introduce something like this, as well as tying up technical bits. The open core model with commercial features is certainly appealing. Open to any perspectives on this.

I've been involved with successful commercial projects using the open core model. Feel free to reach out over email if you have any questions. My contact info is in the profile.

Re: Show HN: Crul – Query Any Webpage or API

#76

Post by RobotToaster »

djhn wrote: post #72

I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could speed up my workflow. 1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just…

Doesn't WordPress have a built in API?

Re: Show HN: Crul – Query Any Webpage or API

#77

Post by djhn »

RobotToaster wrote: post #76
djhn wrote: post #72

I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could speed up my workflow. 1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just…

Doesn't WordPress have a built in API?

If you have a link to some high quality Wordpress API hacks/dorks for scraping, I'm all ears. I think my problem pages are usually made in some sort of page builder, like Elementor, and the content is a static soup of HTML like it came straight out of FrontPage2003.

Re: Show HN: Crul – Query Any Webpage or API

#78

Post by djhn »

portInit wrote: post #73
djhn wrote: post #72

I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could speed up my workflow. 1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just…

Appreciate you taking the time to play around with crul and share your thoughts! They're incredibly valuable. 1. Although it has limitations, were you able to try the normalize command? https://www.crul.com/docs/queryconcepts/api-normalization 2. Will need to think about this some more. 3. Although we don't yet handle pdfs, the rest of the flow is one we aiming to accomplish with crul. Other than pdf, the pieces shou…

1: It does expand a level of hierarchy if you already know what you're looking for (from manually getting the data). Is there a way to omit columns that have more levels or keep them as list columns?

2: Probably too niche to be worth it for you :)

3: Parsing pdfs automagically isn't easy, but handling downloads and images and storing them in a bucket would go quite far (I guess there's a way to do that, but I didn't immediately see a "simple" example).

It sounds like it's potentially a great tool, the only open question is really, is it worth studying the docs and implementing a process in crul as opposed to whatever language I'm familiar with already?

Re: Show HN: Crul – Query Any Webpage or API

#79

Post by ElijahLynn »

portInit wrote: post #49
ElijahLynn wrote: post #46

Earlier quoted context omitted.

Just tried on mobile hotspot, same message. And on Firefox, same message.

That's unfortunate. A quick look suggests ISP but maybe not with the hotspot. Will do a little more digging.

Thanks, once I got back to my home network it worked (same browser). So definitely looks ISP-related.
Post reply on HN