Command-line data analytics
danielcmoura.com
Command-line data analytics
1–10 of 26 posts
Re: Command-line data analytics
#2See also Jeroen Janssens' Data Science at the Command Line:
Re: Command-line data analytics
#3Looks really interesting, certainly something i will enjoy playing with.
Great work
Re: Command-line data analytics
#4SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions!
As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2].
DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats.
dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time.
OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate and display a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table.
[0]: https://github.com/apache/arrow-datafusion
[1]: https://github.com/multiprocessio/dsq
[2]: https://github.com/cube2222/octosql
Disclaimer: Author of OctoSQL
Re: Command-line data analytics
#5Once your data is at a certain size, it might be worth considering tools that does the job quickly enough while still being simple to use. This comparison is very interesting:
https://colab.research.google.com/github/dcmoura/spyql/blob/...
Disclaimer: Work at ClickHouse, whose tool is part of the benchmarking efforts linked to above.
Re: Command-line data analytics
#6Does SPyQL have any advantages over clickhouse-local?
Re: Command-line data analytics
#7SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions! As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2]. DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats. dsq is based on SQLite which means it has to load data into SQLite first, but then gives y…
Here is a comparison of various SQL engines in command line:
https://github.com/dcmoura/spyql/blob/master/notebooks/json_...
Re: Command-line data analytics
#8Once your data is at a certain size, it might be worth considering tools that does the job quickly enough while still being simple to use. This comparison is very interesting: https://colab.research.google.com/github/dcmoura/spyql/blob/... Disclaimer: Work at ClickHouse, whose tool is part of the benchmarking efforts linked to above.
Author of the benchmark and of SPyQL here.
ClickHouse is fantastic. Amazing performance. SPyQL is built on top of Python but still can be faster than jq and several other tools as shown in the benchmark. SPyQL can handle large datasets but Clickhouse local should always show better performance.
SPyQL CLI is more oriented to work in harmony with the shell (piping), to be very simple to use and to leverage the Python ecosystem (you can import Python libs and use them in your queries).
Re: Command-line data analytics
#9SPyQL looks very promising, great work!
I can't help but mention clickhouse-local tool: https://clickhouse.com/docs/en/operations/utilities/clickhou...
clickhouse-local is a single binary that enables you to perform fast data processing using SQL - effectively database features without a database. This tool supports the full breadth of ClickHouse functions, many popular file formats and recently added automatic schema inference. You can query not only local files, but also remote files (from S3/HDFS/static files accessed by URL). Moreover, clickhouse-local tool has interactive mode where you can create tables, play with data and do almost everything that you can do wih ordinary database. And let's not forget, this tool is written in C++, so it's incredibly fast.
Disclaimer: Work at ClickHouse
Re: Command-line data analytics
#10SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions! As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2]. DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats. dsq is based on SQLite which means it has to load data into SQLite first, but then gives y…
You may also want to have a look at the DuckDB command line client [1]. The shell itself is based on the SQLite client, and DuckDB can be used to natively query CSV and Parquet files. Using extensions, DuckDB can also query SQLite and Postgres databases, and query files over HTTPS and S3.
The command line client also has some nifty features like syntax highlighting, and context-aware auto-complete that is coming in the next release.
[1] https://duckdb.org/docs/installation/
Disclaimer: working on DuckDB