Command-line data analytics • Hacker News Board

Re: Command-line data analytics

#21

Post by cube2222 » Thu, Nov 03, 2022, 3:33 PM UTC

mildbyte wrote: post #19

cube2222 wrote: post #18

Earlier quoted context omitted.

That's what I thought about ROAPI as well, until I benchmarked it, and it ended up being very slow[0]. [0]: https://news.ycombinator.com/item?id=32970495

It could be the NDJSON parser (DF source: [0]) or could be a variety of other factors. Looking at the ROAPI release archive [1], it doesn't ship with the definitive `columnq` binary from your comment (EDIT: it does, I was looking in the wrong place! https://github.com/roapi/roapi/releases/tag/columnq-cli-v0.3... ), so it could also have something to do with compilation-time flags. FWIW, we use the Parquet format with…

Yes, DataFusion itself is definitely fast, no denying that.

Re: Command-line data analytics

#22

Post by ptsneves » Thu, Nov 03, 2022, 4:09 PM UTC

Was I the only one thinking of something like google analytics but for command line? A system of usability telemetry for command line utilities might be useful?

Re: Command-line data analytics

#23

Post by Beltalowda » Thu, Nov 03, 2022, 4:11 PM UTC

mytherin wrote: post #10

cube2222 wrote: post #4

SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions! As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2]. DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats. dsq is based on SQLite which means it has to load data into SQLite first, but then gives y…

You may also want to have a look at the DuckDB command line client [1]. The shell itself is based on the SQLite client, and DuckDB can be used to natively query CSV and Parquet files. Using extensions, DuckDB can also query SQLite and Postgres databases, and query files over HTTPS and S3. The command line client also has some nifty features like syntax highlighting, and context-aware auto-complete that is coming in t…

One thing I really miss in the DuckDB is that it doesn't insert the entire query if you use C-p/Up arrow for multi-line queries (it just cycles through the lines of the query). This behaviour is inherited from SQLite, and it trips me up every time even after years of SQLite CLI usage.

Re: Command-line data analytics

#24

Post by qxip » Thu, Nov 03, 2022, 4:20 PM UTC

avogar wrote: post #9

SPyQL looks very promising, great work! I can't help but mention clickhouse-local tool: https://clickhouse.com/docs/en/operations/utilities/clickhou... clickhouse-local is a single binary that enables you to perform fast data processing using SQL - effectively database features without a database. This tool supports the full breadth of ClickHouse functions, many popular file formats and recently added automatic schem…

I couldn't agree more. clickhouse-local is great as a CLI tool as well as a relay for web driven functions, delivering all the clickhouse functionality and speed for ad-hoc tasks with local or remote storage on S3, Parquet files, etc.

Disclaimer: I do NOT work for ClickHouse :)

Re: Command-line data analytics

#25

Post by thriftwy » Thu, Nov 03, 2022, 4:41 PM UTC

Naturally, awk/sort/grep are often much more powerful than fiddling with fully qualified SQL.

Re: Command-line data analytics

#26

Post by snthpy » Fri, Nov 04, 2022, 5:11 AM UTC

samuell wrote: post #13

SPyQL looks fantastic! The thing that worried me when looking into SQL-tools for CSV-files on the commandline, is the plethora of tools available, and it being hard to find one that feels solid and well-supported enough to become a "default" tool for many daily tasks. I want to avoid investing a lot of time learning the ins and outs of a tool that might stop being developed in a year from now. I wish for something th…

It's a valid concern. I'm also working on a similar tool (which I'm not ready to reveal yet) and you're quite right. Only time will tell I guess.