Live data from Hacker News

Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

github.com

41–50 of 64 posts

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#41
post #27
post #26

Earlier quoted context omitted.

Shameless plug. A desktop app: https://superintendent.app

we need more shameless plug on this thread ... the more benchmark, the better! I think this would be a fun weekend afternoon :)

Oh, okay... https://lnav.org is a log file viewer for the terminal that integrates with SQLite so you can use SQL to query your log files.

No benchmarks, though.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#42
post #27
post #26

Earlier quoted context omitted.

Shameless plug. A desktop app: https://superintendent.app

we need more shameless plug on this thread ... the more benchmark, the better! I think this would be a fun weekend afternoon :)

Another shameless plug, then: https://www.ultorg.com

This one is another desktop app, with a slightly different approach: in the CSV file case, the data is parsed and dumped into a bundled PostgreSQL database, which you can then query with a visual query interface.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#43
post #6

AWS Athena offers something similar. You can build tables off of structured text files (like log files) in S3 and run SQL queries.

What’s the performance like though?

It really, really, realllllyyyy, depends on how you setup your prefix/"folder" structure and the underlying file format. Though that's almost certainly true here.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#44
I have a use case, where my company's application logs will be shipped to S3 in a directory structure such as application/timestamp(one_hour)_logs.parquet. We want to build a simple developer focussed UI, where we can query for a given application for a time range and retrieve a bunch of s3 blobs in that time range and brute force search for the desired string. I see that roapi offers a REST interface for a fixed set of files but I would like to dynamically glob newer files. Are there are alternatives that can be used too ? Thanks

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#46
post #26
post #24

This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)

Shameless plug. A desktop app: https://superintendent.app

It looks really nice.

I’ve spend what feels like too much time looking for a windows app that I can use to occasionally preview CSV files before moving them into WSL.

I just think $50/year is too much for occasional use.

Since this posts has initiated several other shameless plugs, maybe I can swoop in and ask for shameless recommendations for something cheap in this space.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#47
post #4

The one thing everyone here is missing so far is that it's a Rust binary, distributed on PyPi. That's brilliant.

I’m all in on using PyPI for binary distribution. Couple that with Python Venv and you have a brilliant system for per project dependancies. I created this project for distributing Node via PyPI: https://pypi.org/project/nodejs-bin/

Why not rather distribute Python as NPM dependency?

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#48
post #24

This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)

I am currently evaluating dsq and its partner desktop app DataStation. AIUI, the developer of DataStation realised that it would be useful to extract the underlying pieces into a standalone CLI, so they both support the same range of sources.

dsq CLI - https://github.com/multiprocessio/dsq DataStation desktop app - https://datastation.multiprocess.io/

Two alternative CLI tools that I looked at:

sq - https://sq.io/ octosql - https://github.com/cube2222/octosql

Honourable mentions to:

Miller - https://miller.readthedocs.io/en/latest/index.html Dasel - https://daseldocs.tomwright.me/

These don't use SQL.

VisiData is also amazing for data navigation, although it requires some effort to get the model:

https://www.visidata.org/

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#49
post #26

Earlier quoted context omitted.

Shameless plug. A desktop app: https://superintendent.app

It looks really nice. I’ve spend what feels like too much time looking for a windows app that I can use to occasionally preview CSV files before moving them into WSL. I just think $50/year is too much for occasional use. Since this posts has initiated several other shameless plugs, maybe I can swoop in and ask for shameless recommendations for something cheap in this space.

Thank you for your kind words. Could you elaborate your use case?

Drop me an email at tanin@ superintendent.app, we should be able to work something out regarding pricing.

Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet

#50
post #48
post #24

This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)

I am currently evaluating dsq and its partner desktop app DataStation. AIUI, the developer of DataStation realised that it would be useful to extract the underlying pieces into a standalone CLI, so they both support the same range of sources. dsq CLI - https://github.com/multiprocessio/dsq DataStation desktop app - https://datastation.multiprocess.io/ Two alternative CLI tools that I looked at: sq - https://sq.io/ oc…

xsv is invaluable for processing big csv files: https://github.com/BurntSushi/xsv
Post reply on HN