Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Range joins in DuckDB

duckdb.org

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Range joins in DuckDB

#21

Post by ignoramous »

folago wrote: post #9
eatonphil wrote: post #2

I've been beating my head trying to get duckdb to statically link into a Go program (I'm neither an expert with cgo nor ld). If anyone else has been able to do this I'd love to see your build steps. https://github.com/marcboeker/go-duckdb produces a non-static binary by default.

Did you try Zig? https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...

Per this post [0] by Andrew Kelley, Zig's lead developer, projects with "large dependency trees" are better off using other tools than rely on Zig's cross-compile magic.

DuckDB needs Python3 to build as well, so not sure how easy it might be to get it cross-compile with Zig CC.

[0] https://archive.is/7SuAf

Re: Range joins in DuckDB

#22

Post by eatonphil »

irq-1 wrote: post #20
eatonphil wrote: post #7

Earlier quoted context omitted.

I don't have the source code in a good state to publish yet but here's where I'm at. At some point before this CGO_LDFLAGS does work and the header is found (omit the -ldflags args). But when it goes to statically link it can no longer find the header. CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"' # github.com/marcbo…

Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"

That's not my code.

Re: Range joins in DuckDB

#23

Post by ignoramous »

minaguib wrote: post #18
orthoxerox wrote: post #16

DuckDB has become my preferred tool for hardcore data wrangling. Excel is fine for like 80% of data processing tasks, but the remaining twenty percent are a pain, especially when you're CPU bound on a remote desktop. Smuggling the DuckDB JDBC driver onto said remote machine was the most productive infosec violation I've ever committed.

I don't know why I waited so long to try it. I wrangle a ton of raw and aggregate data locally every day. I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. I guess I didn't wanna write code. Funny thing is I love SQL. But with `duckdb_cli` it's a game-changer. I'm truly truly impressed.

> I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel.

Have you, at any point, considered / used dgsh [0] or a similar tool? If so, how has been your experience with it?

[0] https://news.ycombinator.com/item?id=13352659

Re: Range joins in DuckDB

#24

Post by eatonphil »

eatonphil wrote: post #22
irq-1 wrote: post #20

Earlier quoted context omitted.

Put the file in quotes. Angle brackets are for built-in files. #include "duckdb.h"

That's not my code.

But also, just to double check, I modified the vendored code and no difference:

  CGO_LDFLAGS="-L$(pwd)/duckdb/src/include" CGO_CFLAGS="-I$(pwd)/duckdb/src/include" go build -ldflags '-extldflags " -lstdc++ -lm -lduckdb -static"'
  # github.com/marcboeker/go-duckdb
  vendor/github.com/marcboeker/go-duckdb/connection.go:4:10: fatal error: duckdb.h: No such file or directory
    4 | #include "duckdb.h"
      |          ^~~~~~~~~~
  compilation terminated.

Re: Range joins in DuckDB

#25

Post by minaguib »

ignoramous wrote: post #23
minaguib wrote: post #18

Earlier quoted context omitted.

I don't know why I waited so long to try it. I wrangle a ton of raw and aggregate data locally every day. I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. I guess I didn't wanna write code. Funny thing is I love SQL. But with `duckdb_cli` it's a game-changer. I'm truly truly impressed.

> I've had a 10-year habit of massaging via unix CLI tools and pipes then moving to excel. Have you, at any point, considered / used dgsh [0] or a similar tool? If so, how has been your experience with it? [0] https://news.ycombinator.com/item?id=13352659

Interesting. Never tried dgsh before, but yea it looks like my poor-man's CLI workflow for a lot of tasks

Re: Range joins in DuckDB

#26

Post by eatonphil »

ignoramous wrote: post #21
folago wrote: post #9

Earlier quoted context omitted.

Did you try Zig? https://dev.to/kristoff/zig-makes-go-cross-compilation-just-...

Per this post [0] by Andrew Kelley, Zig's lead developer, projects with "large dependency trees" are better off using other tools than rely on Zig's cross-compile magic. DuckDB needs Python3 to build as well, so not sure how easy it might be to get it cross-compile with Zig CC. [0] https://archive.is/7SuAf

Also, the issue isn't cross compiling it's just static linking.

Re: Range joins in DuckDB

#27

Post by ignoramous »

eatonphil wrote: post #26
ignoramous wrote: post #21

Earlier quoted context omitted.

Per this post [0] by Andrew Kelley, Zig's lead developer, projects with "large dependency trees" are better off using other tools than rely on Zig's cross-compile magic. DuckDB needs Python3 to build as well, so not sure how easy it might be to get it cross-compile with Zig CC. [0] https://archive.is/7SuAf

Also, the issue isn't cross compiling it's just static linking.

Gotcha. Targeting musl instead of glibc with Zig CC should get you a statically-linked binary, though, unsure if duckdb and its deps play nice with musl.

Personally, a duckdb golang binary interests me. But: I haven't yet mustered enough patience to sit through a time-consuming duckdb build.

Post reply on HN