That's a pretty slick tool you guys built!
Thank you! Really means a lot to us
Show HN: Crul – Query Any Webpage or API
31–40 of 81 posts
Re: Show HN: Crul – Query Any Webpage or API
#32Do you have an example of how to turn an html table element into a CSV? I saw the open and the scrape commands, but wasn't sure where to go from there.
Re: Show HN: Crul – Query Any Webpage or API
#33Earlier quoted context omitted.
It's a tricky question. Part of it is looking at APIs as the main source of data for scheduled queries/data feeds. Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and open up multiple tabs it becomes a little more challenging. We have the concept of domain policies which allow you to control how quickly/slowly you access something. There are also some pu…
In my experience bot detection is moving more towards looking at network activity and IP reputations. Using a proxy will go along way, it's easy to implement, and the cost can easily be passed onto the customer.
In my experience it's starting to moving towards "use a headless browser with patched attributes" or nothing else will work.
edit: I have quite a bit of experience with Akamai and other vendors =)
Re: Show HN: Crul – Query Any Webpage or API
#34Please share some examples of webpages with data to be wrangled that support this statement:
"The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues."
Every web user's needs are different. One person might have a task that they struggle to accomplish while another might have one that presents no major challenges. As a web user, I transform web pages to CSV or SQL. I log HTTP and network requests. I do this for free using open source software. No web browser needed. No docker image needed. Works on both Linux and BSD.
For me, the web is a dataset from which I retrieve data/information. "Tech" companies want to the web to be more like a video game, with visuals and constant interactivity.
Re: Show HN: Crul – Query Any Webpage or API
#35Do you have an example of how to turn an html table element into a CSV? I saw the open and the scrape commands, but wasn't sure where to go from there.
Ah! We didn't quite get an html table command in this release but it will be in the next one.
Here's a query that shows an option, but the table command will be far more straightforward.
open https://www.w3schools.com/html/html_tables.asp --dimension || filter "(nodeName == 'TD')" || groupBy boundingClientRect.top || table _group.0.innerText _group.1.innerText _group.2.innerText
Re: Show HN: Crul – Query Any Webpage or API
#36> status: complete (28.284 seconds / 14 results) Why was it so slow? Is there a default delay between requests or something?
Yeah the default domain throttle policy is 1 req per second per domain. Configurable through domain policies https://www.crul.com/docs/features/domain-policies - although currently an enterprise feature. We found that it becomes too easy to break API request limits or spam a website otherwise. However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering o…
> although currently an enterprise feature.
Wait, so we literally can't go faster than 1 req/s unless we pay?
I have to say I'm pretty disappointed :/
Re: Show HN: Crul – Query Any Webpage or API
#37Hey, just watched the video. This looks super useful! I'm the founder of WunderGraph ( https://wundergraph.com ) and we allow our users to easily integrate multiple data sources into a virtual graph, which they can then access using GraphQL. You can add various data sources, like GraphQL, Federation, OpenAPI, Databases, etc... I was just thinking, wouldn't it be cool if we could find an easy way to add a "Crul" datas…
Will absolutely reach out! Our experience has been that just getting data is often really challenging, so we've really focused on that piece, and being able to easily share with destinations that are purpose built for analytics, viz, etc. Thanks for checking it out!
I'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could connect to postgres://data.splitgraph.com:5432 and SELECT from a table backed by crul (either "mounted" for live querying, and/or ingested once/periodically for subsequent querying). The user just needs to provide parameters for the table; it's up to the FDW how to interpret those parameters.
It would take some thinking and planning, and it's possibly not even a good idea ;) But generally any "data source" is packageable as an FDW as long as you can model it in such a way that you can reasonably implement certain functions for operations like table scans. For most FDWs, this is easy and the tradeoff of a large query is usually limited to excess bandwidth and latency while the query executor reads the result from the FDW. But with a live source pointing to a crawler instance, a table scan could in the worst case mean waiting for the crawler to parse the responses to hundreds of rate-limited network requests. So it's probably better to ingest the data once (and/or periodically) for a particular crul "table" (whatever you decide that means) rather than to query it live.
Fortunately, you can still write an FDW as the adapter layer, because Splitgraph ingests data on a schedule by querying the FDW of the live data source (while tolerating a long-running query). Alternatively (or additionally) you could write an Airbyte adapter which we also support, but only for ingestion - if you want live queryable tables then an FDW is necessary.
We've been interested in adding something like this (think Apify + Postgres) for a while. If done well it could be really cool. Let me know if you want to talk about it: miles@splitgraph.com
Re: Show HN: Crul – Query Any Webpage or API
#38Please share some examples of webpages with data to be wrangled that support this statement: "The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues." Every web user's needs are different. One person might have a task that they struggle to accomplish while another might have one that presents no major challenges. As a web user…
Thanks for this, we're still trying to figure out these details ourselves.
Related to the quote, we've seen interest for API data wrangling, where prepackaged data feeds can be cumbersome to edit, or other implementation details become challenging like credential management, domain throttling, scheduling, checkpointing, export, etc.
It's also interesting for webpage data when you need to use a particular page as an index of links to filter and crawl. We've tried to build an abstraction layer around that.
Initially, we were mainly focused on webpages, and wanted to bypass the visuals of the browser and use a headless browser to fulfill network requests, render js, etc. then convert the page to flat table of enriched elements. With the APIs as another data set there is some work for us to do around language.
We're now trying to figure out what workflows are most relevant for crul to optimize around, as well, honestly - we just built what we thought was cool. Some features/workflows will certainly be more straightforward with existing tools and software - especially for a technically savvy user.
Re: Show HN: Crul – Query Any Webpage or API
#39Earlier quoted context omitted.
Will absolutely reach out! Our experience has been that just getting data is often really challenging, so we've really focused on that piece, and being able to easily share with destinations that are purpose built for analytics, viz, etc. Thanks for checking it out!
I'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could connect to postgres://data.splitgraph.com:5432 and SELECT from a table backed by crul (either "mounted" for live querying, and/or ingested once/periodically for subsequent querying). The user just needs to provide parameters for the table; it'…
We'll have to see how close our current postgres integration is! Would like to understand more and will reach out.
Re: Show HN: Crul – Query Any Webpage or API
#40Earlier quoted context omitted.
I'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could connect to postgres://data.splitgraph.com:5432 and SELECT from a table backed by crul (either "mounted" for live querying, and/or ingested once/periodically for subsequent querying). The user just needs to provide parameters for the table; it'…
We'll have to see how close our current postgres integration is! Would like to understand more and will reach out.
Awesome, looking forward to it.
I just got to the section on destinations ("Stores"). Very cool. If you're building an Enterprise plan where you manage the infrastructure for your customers, we can deploy a dedicated white-labeled deployment of Splitgraph Cloud to any of the three major clouds. Perhaps you could find a use for that in your backend infrastructure.
ETA: Also, if you can write to Postgres, you can write to the Splitgraph DDN [0] with most DML and DDL statements, including INSERT, and CREATE TABLE. So even without any FDW, you might be able to add Splitgraph as a "destination" for your users (who would for the most part just need to provide an API keypair).