Cool project. On a side note, Apache Drill (https://drill.apache.org/) is also worth checking out. The homepage doesn't make it very apparent, but I've used it to query CSVs, etc., using SQL.
Show HN: OctoSQL – Query and join multiple databases and files, written in Go
21–30 of 74 posts
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#22Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…
Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html
I wonder if you or the other Postgres users here could give me some tips.
What would be the best way to use the data from a huge Excel file into my web apps?
Currently I'm converting it to CSV and use `BULK INSERT` with MSSQL. I know that with MSSQL I can also use Excel files as "external database" but AFAIK, Excel files are not indexed and are super slow when used as an "external database".
I wouldn't mind switching to Postgres. I actually prefer Postgres.
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#23When joining tables from 2 different data sources, is the join performed by loading two tables in full into memory ? If yes - any plans on improving this, so that it does not OOM or kill the remote database ?
If no - how do you handle this case ?
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#24When joining tables from 2 different data sources, is the join performed by loading two tables in full into memory ? If yes - any plans on improving this, so that it does not OOM or kill the remote database ? If no - how do you handle this case ?
No, currently we support only lookup joins. We're planning to add table/hash-joins based on table-size heuristics.
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#25Cool project. On a side note, Apache Drill ( https://drill.apache.org/ ) is also worth checking out. The homepage doesn't make it very apparent, but I've used it to query CSVs, etc., using SQL.
Yeah an Drill has been around for a few years. Will likely continue to use Drill over octocat (no offense OP)
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#26Really cool, thanks for sharing. You all might want to look at Apache Calcite ( http://calcite.apache.org/ ) as well for inspiration, which has similar functionality as a subset of its features!
Apache Drill vs Calcite? Seem very similar.
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#27Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…
> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This exactly describes Drill ( https://drill.apache.org/ ) which can query any data source under the sun (RDBMS, NoSQL, files, cluster…
Thanks, didn't know about it! I'll make sure to read up on it!
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#28Really cool, thanks for sharing. You all might want to look at Apache Calcite ( http://calcite.apache.org/ ) as well for inspiration, which has similar functionality as a subset of its features!
Apache Drill vs Calcite? Seem very similar.
Lots of Apache projects have very similar counterparts, which has always puzzled me - I wonder if there are publicly available official selection criteria for new Apache projects?
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#29Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…
At Hasura, we recently released the preview of Remote Joins that allows you to join different GraphQL APIs. So, all you need is a graphql wrapper over your DB/sources (Postgres comes in-built) and you can then join whatever sources.
https://blog.hasura.io/remote-joins-a-graphql-api-to-join-da...
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#30A small tangential comment: In the README you mention octopi. Although not wrong as per the Oxford English Dictionary which lists octopuses, octopi and octopodes (in that order), normally it is better to use octopuses.