Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

github.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#11

Post by codetrotter »

ComputerGuru wrote: post #10
parhamn wrote: post #8

Earlier quoted context omitted.

> Just the usual cgo performance overhead? No, the performance is certainly orders of magnitude faster than serializing over std streams on a subprocess (c ffi calls in cgo are 10s of nanoseconds). But one of the big draws of golang is the write-once-compile-anywhere toolchain and calls cgo makes that harder.

But compiling the non-go assistant process is not going to be any easier than cgo, right?

In the mentioned https://gitlab.com/cznic/sqlite there would not be any assistant process, right?

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#12

Post by parhamn »

ComputerGuru wrote: post #10
parhamn wrote: post #8

Earlier quoted context omitted.

> Just the usual cgo performance overhead? No, the performance is certainly orders of magnitude faster than serializing over std streams on a subprocess (c ffi calls in cgo are 10s of nanoseconds). But one of the big draws of golang is the write-once-compile-anywhere toolchain and calls cgo makes that harder.

But compiling the non-go assistant process is not going to be any easier than cgo, right?

Well its a different access pattern. As the underlying library points out:

> It is used in programming environments that do not allow calling C API functions.

Also I guess which one is easier will be subjective. The steps are sorta similar:

Step 1) install sqlite or squinn on the base system (the latter might be harder)

Step 2) if sqllite use cgo, if squinn just use go (the former might be harder but more performant)

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#13

Post by scosman »

infogulch wrote: post #6
scosman wrote: post #4

Earlier quoted context omitted.

What’s the case against cgo for SQLite? Just the usual cgo performance overhead? It seems like a pretty good cgo use case: a decent amount of work, which is typically slow enough that cgo overhead isn’t perf critical (because DB usually means disk reads), a super robust and well tested C library with a super well maintained cgo wrapper (mattn).

mattn's sqlite3 is probably the most ideal use case for cgo imaginable. But it can still be annoying to set up cgo to build across many platforms. I think Go should just pull a Zig and just embed a full blown C compiler into go build.

Exactly. Good to know I didn’t miss a trade off in the mix.

If I’m choosing to use a C framework (SQLite) I’m okay signing up for the environment costs. Prefer that over abstractions in an intermediate layer that might not be maintained in a few years.

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#14

Post by arp242 »

llimllib wrote: post #5
scosman wrote: post #4

Earlier quoted context omitted.

What’s the case against cgo for SQLite? Just the usual cgo performance overhead? It seems like a pretty good cgo use case: a decent amount of work, which is typically slow enough that cgo overhead isn’t perf critical (because DB usually means disk reads), a super robust and well tested C library with a super well maintained cgo wrapper (mattn).

the other pain point I know of is that it's hard to cross-compile. You can do it with zig[1], but it's still not pleasant. 1: https://zig.news/kristoff/building-sqlite-with-cgo-for-every...

And Zig only covers a pretty limited range of platforms to start with.

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#15

Post by cvilsmeier »

scosman wrote: post #4
eatonphil wrote: post #2

If you really want to use SQLite without anything that isn't Go (since this project involves forking and communicating with a non-Go SQLite in a separate process over pipes), there's a Go translation of SQLite's C source. :) https://gitlab.com/cznic/sqlite https://datastation.multiprocess.io/blog/2022-05-12-sqlite-i...

What’s the case against cgo for SQLite? Just the usual cgo performance overhead? It seems like a pretty good cgo use case: a decent amount of work, which is typically slow enough that cgo overhead isn’t perf critical (because DB usually means disk reads), a super robust and well tested C library with a super well maintained cgo wrapper (mattn).

Sqinn author here. Nothing against CGO, but I develop/deploy on Win/Linux, and cross-compiling CGO is very painful. Regarding performance: To my own surprise, Sqinn out-performs mattn (and others) for normal workloads, see https://github.com/cvilsmeier/sqinn-go-bench

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#16

Post by eatonphil »

cvilsmeier wrote: post #15
scosman wrote: post #4

Earlier quoted context omitted.

What’s the case against cgo for SQLite? Just the usual cgo performance overhead? It seems like a pretty good cgo use case: a decent amount of work, which is typically slow enough that cgo overhead isn’t perf critical (because DB usually means disk reads), a super robust and well tested C library with a super well maintained cgo wrapper (mattn).

Sqinn author here. Nothing against CGO, but I develop/deploy on Win/Linux, and cross-compiling CGO is very painful. Regarding performance: To my own surprise, Sqinn out-performs mattn (and others) for normal workloads, see https://github.com/cvilsmeier/sqinn-go-bench

I think it's a somewhat unfair (though who cares if it's unfair) comparison because you aren't using the database/sql interface and mattn does.

If you drop that interface, you get much better performance.

See: https://github.com/eatonphil/gosqlite for example.

Edit: Nevermind, you did include crawshaw (which doesn't use database/sql) in your benchmark!

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#17

Post by jerf »

codetrotter wrote: post #11
ComputerGuru wrote: post #10

Earlier quoted context omitted.

But compiling the non-go assistant process is not going to be any easier than cgo, right?

In the mentioned https://gitlab.com/cznic/sqlite there would not be any assistant process, right?

No, but that has the disadvantage of being C compiled into Go, then being compiled into native executable.

I'm actually surprised by how readable this came out; props to the Go->C compiler author. But you can guess that pushing this sort of thing through the Go compiler is going to cause some slowdowns due to sheer paradigm mismatch: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_lin...

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#18

Post by hamandcheese »

eatonphil wrote: post #2

If you really want to use SQLite without anything that isn't Go (since this project involves forking and communicating with a non-Go SQLite in a separate process over pipes), there's a Go translation of SQLite's C source. :) https://gitlab.com/cznic/sqlite https://datastation.multiprocess.io/blog/2022-05-12-sqlite-i...

Would it be possible to link a pure go codebase to pre-compiled sqlite binaries and not need to worry about cgo when cross compiling?

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#19

Post by throwaway894345 »

jerf wrote: post #17
codetrotter wrote: post #11

Earlier quoted context omitted.

In the mentioned https://gitlab.com/cznic/sqlite there would not be any assistant process, right?

No, but that has the disadvantage of being C compiled into Go, then being compiled into native executable. I'm actually surprised by how readable this came out; props to the Go->C compiler author. But you can guess that pushing this sort of thing through the Go compiler is going to cause some slowdowns due to sheer paradigm mismatch: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_lin...

I don’t think the paradigm is particularly mismatched, right? If you translate C to Go, it would be pretty much best case for Go (neither language likes lots of small allocations). But Go lacks a lot of optimizations that most C compilers support, like return value optimization, aggressive inlining, etc. C also has lighter-weight function calls and so on that you pay for in Go.

Maybe this is what you mean by paradigm mismatch, but usually I would think of something like translating an allocation-heavy Java app into Go as paradigm mismatch.

Re: Show HN: Sqinn-Go is a Golang library for accessing SQLite databases in pure Go

#20

Post by irusensei »

Groxx wrote: post #9
parhamn wrote: post #8

Earlier quoted context omitted.

> Just the usual cgo performance overhead? No, the performance is certainly orders of magnitude faster than serializing over std streams on a subprocess (c ffi calls in cgo are 10s of nanoseconds). But one of the big draws of golang is the write-once-compile-anywhere toolchain and calls cgo makes that harder.

To be a bit more specific here: pure Go binaries are trivial to cross-compile and they Just Work™ basically all the time. `GOOS="darwin" GOARCH="arm64" go build .` and you're done. Just iterate over the combinations you care about, they'll all work. As soon as you or a library touches cgo, you have to deal with finding and setting up C cross-compilation tooling for your target(s), dynamic linking details, tons of stu…

Golang works on Plan9. It can even bootstrap itself. A few months ago I was trying to setup some server software on 9Front for giggles and while most stuff worked I couldn't past the Sqlite CGO dependencies.
Post reply on HN