Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crul – Query Any Webpage or API

crul.com

Post reply on HN
Print view View original

1–10 of 81 posts

Show HN: Crul – Query Any Webpage or API

#1

Post by portInit »

Hi HN, we’re Carl and Nic, the creators of crul (https://www.crul.com), and we’ve been hard at work for the last year and a half building our dream of turning the web into a dataset. In a nutshell crul is a tool for querying and building web and api data feeds from anywhere to anywhere.

With crul you can crawl and transform web pages into csv tables, explore and dynamically query APIs, filter and organize data, and push data sets to third party data lakes and analytics tools. Here’s a demo video, we’ve been told Nic sounds like John Mayer (lol) (https://www.crul.com/demo-video)

We’ve personally struggled wrangling data from the web using puppeteer/playwright/selenium, jq or cobbling together python scripts, client libraries, and schedulers to consume APIs. The reality is that shit is hard, doesn’t scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues. The tools we love to hate.

Crul’s value prop is simple: Query any Webpage or API for free.

At its core, crul is based on the foundational linked nature of Web/API content. It consists of a purpose built map/expand/reduce engine for hierarchical Web/API content (kind of like postman but with a membership to Gold's Gym) with a familiar parser expression grammar that naturally gets the job done (and layered caching to make it quick to fix when it doesn’t on the first try). There’s a boatload of other features like domain policies, scheduler, checkpoints, templates, REST API, Web UI, vault, OAuth for third parties and 20+ stores to send your data to.

Our goal is to open source crul as time and resources permit. At the end of the day it’s just the two of us trying to figure things out as we go! We’re just getting started.

Crul is one bad mother#^@%*& and the web is finally yours!

Download crul for free as a Mac OS desktop application or as a Docker image (https://www.crul.com) and let us know if you love it or hate it. (https://forms.gle/5BXb5bLC1D5QG7i99) And come say hello to us on our slack channel - we’re a friendly bunch! (https://crulinc.slack.com/)

Nic and Carl (https://www.crul.com/early-days)

Show HN: Crul – Query Any Webpage or API
crul.com

Re: Show HN: Crul – Query Any Webpage or API

#3

Post by dishwishy »

I just started playing with CRUL recently to try to map out different media download links on various websites, in an effort to avoid clicking around looking for hidden content. The language is very robust, but just the `filter` command alone is crazy powerful for just exploring things quickly and intuitively.

Re: Show HN: Crul – Query Any Webpage or API

#4

Post by portInit »

dishwishy wrote: post #3

I just started playing with CRUL recently to try to map out different media download links on various websites, in an effort to avoid clicking around looking for hidden content. The language is very robust, but just the `filter` command alone is crazy powerful for just exploring things quickly and intuitively.

Thanks! The find (https://www.crul.com/docs/commands/find) command works really well if you are trying to construct a filter expression and just want to quickly look for results containing a particular string so you can see the defining attributes/column+row values.

There's a short writeup of this pattern here: https://www.crul.com/docs/examples/how-to-find-filters

Re: Show HN: Crul – Query Any Webpage or API

#5

Post by jensneuse »

Hey, just watched the video. This looks super useful! I'm the founder of WunderGraph (https://wundergraph.com) and we allow our users to easily integrate multiple data sources into a virtual graph, which they can then access using GraphQL. You can add various data sources, like GraphQL, Federation, OpenAPI, Databases, etc... I was just thinking, wouldn't it be cool if we could find an easy way to add a "Crul" datasource? If you're interested, please DM me in our discord (https://wundergraph.com/discord). I'd love to have a conversation!

Re: Show HN: Crul – Query Any Webpage or API

#6

Post by jawns »

How does crul handle the dynamic nature of the web?

Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is renamed or moved as part of a site redesign, what happens?

I've worked on a bunch of tools in the past that do similar things, and structural changes were the kryptonite for all of them.

A secondary problem is when you use particular content as a reference point, and that content is later updated. Now your reference point is gone!

Re: Show HN: Crul – Query Any Webpage or API

#7

Post by portInit »

jensneuse wrote: post #5

Hey, just watched the video. This looks super useful! I'm the founder of WunderGraph ( https://wundergraph.com ) and we allow our users to easily integrate multiple data sources into a virtual graph, which they can then access using GraphQL. You can add various data sources, like GraphQL, Federation, OpenAPI, Databases, etc... I was just thinking, wouldn't it be cool if we could find an easy way to add a "Crul" datas…

Will absolutely reach out! Our experience has been that just getting data is often really challenging, so we've really focused on that piece, and being able to easily share with destinations that are purpose built for analytics, viz, etc.

Thanks for checking it out!

Re: Show HN: Crul – Query Any Webpage or API

#8

Post by portInit »

jawns wrote: post #6

How does crul handle the dynamic nature of the web? Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is renamed or moved as part of a site redesign, what happens? I've worked on a bunch of tools in the past that do similar things, and structural changes were the kryptonite for all of them. A secondary problem is when you use particular content as…

At this point we're considering it a foundational concept to build around - web content changes, so our best option currently is to make the query as easy as possible to change, and alert when things break.

We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work.

But the expanding and querying concepts also make a lot of sense with APIs, which tend to be a little more stable.

Re: Show HN: Crul – Query Any Webpage or API

#10

Post by portInit »

asdadsdad wrote: post #9

How do you plan on tackling anti-bot blocking?

It's a tricky question. Part of it is looking at APIs as the main source of data for scheduled queries/data feeds.

Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and open up multiple tabs it becomes a little more challenging. We have the concept of domain policies which allow you to control how quickly/slowly you access something. There are also some puppeteer level options that could be relevant, even a headful toggle.

We have not invested too much time into this yet as we focused on getting the core functionality working. We think there are use cases (particularly with APIs) that don't run into this problem, but if it comes up more often we'll come up with some options.

Post reply on HN