Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: OctoSQL – Query and join multiple databases and files, written in Go

github.com

Post reply on HN
Print view View original

41–50 of 74 posts

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#41

Post by truth_seeker »

riku_iki wrote: post #40
truth_seeker wrote: post #3

Earlier quoted context omitted.

Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html

Do fdw/standard connectors support pushdown predicates well? Documentation doesn't say much about that.

Yep. PG 9.5 and later supports it.

Use "EXPLAIN VERBOSE" before your query to get more info.

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#42

Post by MoOmer »

jadbox wrote: post #26
MoOmer wrote: post #6

Really cool, thanks for sharing. You all might want to look at Apache Calcite ( http://calcite.apache.org/ ) as well for inspiration, which has similar functionality as a subset of its features!

Apache Drill vs Calcite? Seem very similar.

Drill depends on Calcite for SQL parsing and cost-based query optimization!

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#43

Post by tempguy9999 »

brunoqc wrote: post #22
truth_seeker wrote: post #3

Earlier quoted context omitted.

Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html

I wonder if you or the other Postgres users here could give me some tips. What would be the best way to use the data from a huge Excel file into my web apps? Currently I'm converting it to CSV and use `BULK INSERT` with MSSQL. I know that with MSSQL I can also use Excel files as "external database" but AFAIK, Excel files are not indexed and are super slow when used as an "external database". I wouldn't mind switching…

As an MSSQL guy who rarely connects it to excel, if your workflow allows it consider pulling the entirety of the excel sheet into a table in the db and working on it just from there. What's huge for excel isn't for MSSQL.

As a decades-long user of MSSQL guy who is getting sick of being rpeatedly done over by MS, could you explain why you prefer postgres? That would be incredibly interesting, TIA

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#44

Post by just_myles »

truth_seeker wrote: post #3
cube2222 wrote: post #2

Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…

Generally, I make use of file_fdw module of Postgres to achieve this. https://www.postgresql.org/docs/11/file-fdw.html

My god. I didn't know postgres can do this. Thank you!

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#45

Post by dlphn___xyz »

minxomat wrote: post #18
cube2222 wrote: post #2

Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…

> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This exactly describes Drill ( https://drill.apache.org/ ) which can query any data source under the sun (RDBMS, NoSQL, files, cluster…

i have a reoccurring use case where i have huge csv files that i need to aggregate / summarize using a pretty old netbook. the only way i can get this done is through spark/scala. would drill be a better option?

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#46

Post by minxomat »

dlphn___xyz wrote: post #45
minxomat wrote: post #18

Earlier quoted context omitted.

> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This exactly describes Drill ( https://drill.apache.org/ ) which can query any data source under the sun (RDBMS, NoSQL, files, cluster…

i have a reoccurring use case where i have huge csv files that i need to aggregate / summarize using a pretty old netbook. the only way i can get this done is through spark/scala. would drill be a better option?

Sure. Create a filesystem data source in the Drill config (https://drill.apache.org/docs/file-system-storage-plugin/) with the directories where your CSVs are in. Then query them however you'd like. Use the CLI if you want to produce another CSV or REST/JDBC if you need the data somewhere else. If the query is complex and long-running you can monitor it's state in the Drill web UI.

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#47

Post by bryanrasmussen »

cube2222 wrote: post #2

Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…

microsoft logparser https://www.microsoft.com/en-us/download/details.aspx?id=246... was good for querying various text formats from command line with an SQL-like syntax

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#48

Post by kfk »

Hey, at work we have been forced to use Denodo. How does this compare to Denodo? Seems a lot better and doesn’t try to do any “data virtualization”. I am a bit scared by the query optimizer because the Denodo one is super buggy. Overall I think the concept is great and this might be a great companion to etl tools.

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#49

Post by minxomat »

kfk wrote: post #48

Hey, at work we have been forced to use Denodo. How does this compare to Denodo? Seems a lot better and doesn’t try to do any “data virtualization”. I am a bit scared by the query optimizer because the Denodo one is super buggy. Overall I think the concept is great and this might be a great companion to etl tools.

If vis is the focus, I'd recommend Postgres FDW over anything else here, just because PG is supported by almost every vis software (metabase, redash, many Apache projects, Tableau, Google Data Studio (free) etc.)

Re: Show HN: OctoSQL – Query and join multiple databases and files, written in Go

#50

Post by arendtio »

cube2222 wrote: post #2

Hey, one of the authors here. The motivation behind this project is that I always wanted a simple commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use). This started as an university project,…

Pretty cool stuff.

Does it support writing to (single) data sources too or only reading queries?

Post reply on HN