Consider adding a server/daemon with jobs and scheduling. This would let people copy and reconcile data between systems, or prep data at night, etc... It could also save indexes and table statistics between queries.
Show HN: OctoSQL – Query and join multiple databases and files, written in Go
31–40 of 74 posts
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#32A small tangential comment: In the README you mention octopi. Although not wrong as per the Oxford English Dictionary which lists octopuses, octopi and octopodes (in that order), normally it is better to use octopuses.
I did that intentionally cause I think octopi sounds kind of cute :)
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#33Earlier quoted context omitted.
Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html
I wonder if you or the other Postgres users here could give me some tips. What would be the best way to use the data from a huge Excel file into my web apps? Currently I'm converting it to CSV and use `BULK INSERT` with MSSQL. I know that with MSSQL I can also use Excel files as "external database" but AFAIK, Excel files are not indexed and are super slow when used as an "external database". I wouldn't mind switching…
THe process is similar, you would convert to csv and then use the \copy command to bulk insert the data. In pg 12, the copy command has grown a lot of very useful features like WHERE filtering, so if you get the chance go for 12:
https://www.cybertec-postgresql.com/en/tech-preview-improvin...
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#34Consider adding a server/daemon with jobs and scheduling. This would let people copy and reconcile data between systems, or prep data at night, etc... It could also save indexes and table statistics between queries.
That's definitely our plan for the future!
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#35A small tangential comment: In the README you mention octopi. Although not wrong as per the Oxford English Dictionary which lists octopuses, octopi and octopodes (in that order), normally it is better to use octopuses.
Why not let that word live it's own little life?
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#36Earlier quoted context omitted.
Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html
I wonder if you or the other Postgres users here could give me some tips. What would be the best way to use the data from a huge Excel file into my web apps? Currently I'm converting it to CSV and use `BULK INSERT` with MSSQL. I know that with MSSQL I can also use Excel files as "external database" but AFAIK, Excel files are not indexed and are super slow when used as an "external database". I wouldn't mind switching…
[deleted]
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#37Earlier quoted context omitted.
Apache Drill vs Calcite? Seem very similar.
Lots of Apache projects have very similar counterparts, which has always puzzled me - I wonder if there are publicly available official selection criteria for new Apache projects?
This doesn't bother the Apache Software Foundation. So long as a project is able to follow the "Apache Way" (licensing, treatment of source artifacts, organizational structure of project members and community, etc.), then the project can exist in the ASF without conflict of other similar projects.
The ASF's mission is not focused on bringing one best-of-breed product in every category. More, the ASF is focused on the stewardship and delivery of the software. They are more interested in the organization and community surrounding the project than the code itself.
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#38Earlier quoted context omitted.
Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html
I wonder if you or the other Postgres users here could give me some tips. What would be the best way to use the data from a huge Excel file into my web apps? Currently I'm converting it to CSV and use `BULK INSERT` with MSSQL. I know that with MSSQL I can also use Excel files as "external database" but AFAIK, Excel files are not indexed and are super slow when used as an "external database". I wouldn't mind switching…
This link covers it in detail:
http://www.postgresqltutorial.com/import-csv-file-into-posgr...
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#39When joining tables from 2 different data sources, is the join performed by loading two tables in full into memory ? If yes - any plans on improving this, so that it does not OOM or kill the remote database ? If no - how do you handle this case ?
No, currently we support only lookup joins. We're planning to add table/hash-joins based on table-size heuristics.
How do you look up csv records?..
Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go
#40Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…
Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html
Do fdw/standard connectors support pushdown predicates well? Documentation doesn't say much about that.