Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet • Hacker News Board

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#61

Post by fshr » Sun, Sep 25, 2022, 6:32 PM UTC

gavinray wrote: post #12

bachmeier wrote: post #3

As I commented on a recent similar discussion, these tools can't be used for update or insert. As useful as querying might be, it's terribly misleading to claim to "run SQL" if you can't change the data, since that's such a critical part of an SQL database.

Question: I've built something that supports full CRUD, and queries that span multiple data sources with optimization and pushdown What kind of headline would make you want to read/try such a thing? (I'm planning on announcing it + releasing code on HN but have never done so before)

Hi Gavin; that sounds interesting! I saw @eirikbakke make a comment about https://www.ultorg.com earlier. It appears to also support editing the underlying data. I'm curious to see how you've each tackled these tricky topics.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#62

Post by andygrove » Sun, Sep 25, 2022, 10:58 PM UTC

houqp wrote: post #40

sakras wrote: post #39

Earlier quoted context omitted.

Ah nice, thank you for sharing that. I hadn’t seen it before, and congrats on beating out Spark that hard, I hope it continues to improve! As an aside, maybe it would make sense to publish a new blog post somewhere so that the top hit on Google for “DataFusion benchmark” isn’t that post I linked.

Haha, yeah, we should definitely put a little bit more efforts into SEO :) Everyone is so focused on the hard-core engineering at the moment. I think Matthew from the community is actually working on a new comprehensive benchmark for us at the moment, which I hope will be published soon.

I will update these old pages on my blog and redirect them!

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#63

Post by mattewong » Mon, Sep 26, 2022, 3:36 AM UTC

ebfe1 wrote: post #24

This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)

another shameless plug: https://github.com/liquidaty/zsv

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#64

Post by jaxn » Tue, Sep 27, 2022, 12:43 AM UTC

tomrod wrote: post #34

jaxn wrote: post #29

Earlier quoted context omitted.

Maybe in your database. Do you have any validation of that claim in a larger context?

Purely the power law. That would be an interesting thing to figure out though. Maybe a github crawl. EDIT: I stand corrected based on github code files (which might better represent application CRUD queries versus use by analysts, more thought required!) SELECT: 7.3M code results [0] INSERT: 8.9M code results [1] UPDATE: 5.5M code results [2] DELETE: 5.0M code results [3] [0] https://github.com/search?q=select++exten…

Reads are easy to cache at various layers (query cache, application, web, etc). Inserts and updates must go to the database.

So even a read-heavy application could have more writes than reads due to caching.