51–60 of 64 posts
This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)
Earlier quoted context omitted.
I am currently evaluating dsq and its partner desktop app DataStation. AIUI, the developer of DataStation realised that it would be useful to extract the underlying pieces into a standalone CLI, so they both support the same range of sources. dsq CLI - https://github.com/multiprocessio/dsq DataStation desktop app - https://datastation.multiprocess.io/ Two alternative CLI tools that I looked at: sq - https://sq.io/ oc…
xsv is invaluable for processing big csv files: https://github.com/BurntSushi/xsv
I have a use case, where my company's application logs will be shipped to S3 in a directory structure such as application/timestamp(one_hour)_logs.parquet. We want to build a simple developer focussed UI, where we can query for a given application for a time range and retrieve a bunch of s3 blobs in that time range and brute force search for the desired string. I see that roapi offers a REST interface for a fixed set…
This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)
This is cool...Totally reminded me about several tools pop up on HN every now and then in the past for similar task so i did a quick search: clickhouse-local - https://news.ycombinator.com/item?id=22457767 q - https://news.ycombinator.com/item?id=27423276 textql - https://news.ycombinator.com/item?id=16781294 simpql- https://news.ycombinator.com/item?id=25791207 We need a benchmark i think..;)
I have a use case, where my company's application logs will be shipped to S3 in a directory structure such as application/timestamp(one_hour)_logs.parquet. We want to build a simple developer focussed UI, where we can query for a given application for a time range and retrieve a bunch of s3 blobs in that time range and brute force search for the desired string. I see that roapi offers a REST interface for a fixed set…
I have a use case, where my company's application logs will be shipped to S3 in a directory structure such as application/timestamp(one_hour)_logs.parquet. We want to build a simple developer focussed UI, where we can query for a given application for a time range and retrieve a bunch of s3 blobs in that time range and brute force search for the desired string. I see that roapi offers a REST interface for a fixed set…
AWS Athena offers something similar. You can build tables off of structured text files (like log files) in S3 and run SQL queries.
What’s the performance like though?
Is not a BI tool tho, if queries have high variability on the where clause and you can't leverage indexes then you're looking at minutes as response time.
If the data is not in structures but plain csv/json, all bets are off.
I've not yet tested it at terabyte scale albeit it should happily scale up there.
This looks really cool! Especially using datafusion underneath means that it probably is blazingly fast. If you like this, I recommend taking a look at OctoSQL[0], which I'm the author of. It's plenty fast and easier to add new data sources for as external plugins. It can also handle endless streams of data natively, so you can do running groupings on i.e. tailed JSON logs. Additionally, it's able to push down predic…
A simple group by
time columnq sql --table books_10m.ndjson "SELECT AVG(overall) FROM books_10m"
The equivalent in OctoSQL takes less than 3 seconds.
I retract my statement about this project being blazingly fast, though I imagine it's just the JSON parser that requires optimization.
Earlier quoted context omitted.
xsv is invaluable for processing big csv files: https://github.com/BurntSushi/xsv
qsv is a fork of this, as qsv is pretty much unmaintained now (I don't mean to sound negative, BurntSushi did an AMAZING job and I love the work they did).