Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
1–10 of 64 posts
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#2This looks really cool! Especially using datafusion underneath means that it probably is blazingly fast.
If you like this, I recommend taking a look at OctoSQL[0], which I'm the author of.
It's plenty fast and easier to add new data sources for as external plugins.
It can also handle endless streams of data natively, so you can do running groupings on i.e. tailed JSON logs.
Additionally, it's able to push down predicates to the database below, so if you're selecting 10 rows from a 1 billion row table, it'll just get those 10 rows instead of getting them all and filtering in memory.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#3As I commented on a recent similar discussion, these tools can't be used for update or insert. As useful as querying might be, it's terribly misleading to claim to "run SQL" if you can't change the data, since that's such a critical part of an SQL database.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#4The one thing everyone here is missing so far is that it's a Rust binary, distributed on PyPi. That's brilliant.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#5As I commented on a recent similar discussion, these tools can't be used for update or insert. As useful as querying might be, it's terribly misleading to claim to "run SQL" if you can't change the data, since that's such a critical part of an SQL database.
The title is an editorialization. The project is very careful to emphasize that it is for reading data:
> Create full-fledged APIs for slowly moving datasets without writing a single line of code.
Even the name of the project "ROAPI" has "read only" in the name.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#6AWS Athena offers something similar. You can build tables off of structured text files (like log files) in S3 and run SQL queries.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#7As I commented on a recent similar discussion, these tools can't be used for update or insert. As useful as querying might be, it's terribly misleading to claim to "run SQL" if you can't change the data, since that's such a critical part of an SQL database.
What you're really saying is that the database presented in OP is not useful because it only handles DQL.
1. SQL can be thought of as being composed of several smaller lanuages: DDL, DQL, DML, DCL.
2. columnq-cli is only a CLI to a query engine, not a database. As such, it only supports DQL by design.
3. I have the impression that outside of data engineering/DBA, people are rarely taught the distinction between OLTP and OLAP workloads [1]. The latter often utilizes immutable data structures (e.g. columnar storage with column compression), or provides limited DML support, see e.g. the limitations of the DELETE statement in ClickHouse [2], or the list of supported DML statements in Amazon Athena [3]. My point -- as much as this tool is useless for transactional workloads, it is perfectly capable of some analytical workloads.
[1] Opinion, not a fact.
[2] https://clickhouse.com/docs/en/sql-reference/statements/dele...
[3] https://docs.aws.amazon.com/athena/latest/ug/functions-opera...
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#8As I commented on a recent similar discussion, these tools can't be used for update or insert. As useful as querying might be, it's terribly misleading to claim to "run SQL" if you can't change the data, since that's such a critical part of an SQL database.
I think it is worth pointing out that this tool does support querying Delta Lake (the author of ROAPI is also a major contributor the native Rust implementation of Delta Lake). Delta Lake certainly supports transactions, so ROAPI can query transactional data, although the writes would not go through ROAPI.
Re: Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheet
#9The one thing everyone here is missing so far is that it's a Rust binary, distributed on PyPi. That's brilliant.
Can you explain the advantages of this vs cargo?