Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go port of SQLite without CGo

gitlab.com

Post reply on HN
Print view View original

61–70 of 122 posts

Re: Go port of SQLite without CGo

#61

Post by bombolo »

badtuple wrote: post #32
djbusby wrote: post #24

Earlier quoted context omitted.

Really tho, how much is build time vs runtime? Maybe is just work small projects but I rarely care about build time. Am I missing something?

Honestly, on a non-toy project, build times with cgo are _brutal_. I agree with you usually, but when a build time on a beefy computer switches from under a second to >1min you notice it. Linters and IDEs get slow when they check for errors, tests run slow, feedback drags, and all your workflows that took advantage of Go's fast compile times are now long enough that your flow and mental context disappear. I'm way mor…

Why would you have to recompile sqlite every time?

I guess you just need to compile the .a once and then just reuse it?

If you're rebuilding it every single time, your build is set up wrong.

Re: Go port of SQLite without CGo

#62

Post by pkolaczk »

cdogl wrote: post #5
philosopher1234 wrote: post #3

Earlier quoted context omitted.

What makes it slower?

It’s hard for Go to beat C in a straight benchmark - Go is garbage collected and allocations often have extra overhead due to the memory layout of interfaces.

Not only Go compiler is not as good at optimisation as C/C++/Rust/Zig/D compilers, but also Go FFI to C has significant overhead, which means there is often not much point in using low-level Linux APIs for performance advantage. Same problem like with Java. And in a database system you quite likely want a fine degree of control with non-standard stuff.

Re: Go port of SQLite without CGo

#63

Post by pkolaczk »

djbusby wrote: post #25
cube2222 wrote: post #10

Earlier quoted context omitted.

Other than what the other responders said, since it uses a C to Go compiler, I'd imagine the Go code that's generated isn't as fast as handwritten Go code could be.

C -> GPT -> Go -> Edit -> commit?

-> fail to compile / crash at startup

Re: Go port of SQLite without CGo

#64

Post by shp0ngle »

Note that this is not “go port”. It’s C transpiled to go with their custom C compiler, and it’s transpiled differently for each architecture (as it’s using unsafe go heavily)

I look at it like a neat exercise but not really something to use in production. (I actually use it in tests though.)

Re: Go port of SQLite without CGo

#65

Post by willi59549879 »

lopkeny12ko wrote: post #34
ikiris wrote: post #31

Earlier quoted context omitted.

Why would someone at google ever use sqlite?

Why wouldn't they? Are you implying that, over the course of decades of Google's existence and thousands of software projects, not once has there ever had the need for a lightweight SQL store that can be embedded with a server or CLI tool? Your comment truly does SQLite a disservice. SQLite is probably one of the most impressive pieces of software to have ever graced this earth.

Google's practice is that every service they build is built that way that it can scale to millions of users. On servers i don't think they would use sqlite unlike in android. If i build stuff i usually start with sqlite.

Re: Go port of SQLite without CGo

#66

Post by pkolaczk »

adastra22 wrote: post #50
eschaton wrote: post #28

What’s the point of something like this instead of just using SQLite? Seriously, “foo but in bar!” is not exactly the sort of thing that solves real problems for most people.

CGO is a real problem for many people. Due to how Go handles its async-by-default concurrency model, every call to an FFI library requires setting up a stack and marshaling data. It ends up being rather expensive. So the "without CGo" stuff is about retaining the fast performance people expect of these highly optimized C libraries.

How does it really solve the FFI issue? Now instead of an FFI boundary between the app and the database, you have an FFI boundary between the database (now implemented in Go) and the operating system C API. And the database quite likely performs multiple calls into the C API per each user app query, so it might be in fact worse.

Re: Go port of SQLite without CGo

#67

Post by masklinn »

tetete wrote: post #38

Why would you want to do this? SQLLITE is so stable. Why would you switch to some port that will be dead in a year?

Implementation decisions of Go have various advantages in terms of control and memory use but they make linking to binary artefacts problematic. Not complicated, but it impacts the toll chain’s flexibility drastically. The FFI is also extremely slow.

That is why the go ecosystem tends to reimplement everything in go.

Re: Go port of SQLite without CGo

#68

Post by 59nadir »

icholy wrote: post #26
cdogl wrote: post #5

Earlier quoted context omitted.

It’s hard for Go to beat C in a straight benchmark - Go is garbage collected and allocations often have extra overhead due to the memory layout of interfaces.

In addition, a lot of optimizations (like loop unrolling) are left on the table to preserve Go's fast compile times.

I was surprised with the announcement that Go's standard compiler had register allocation for function arguments[0] added to it. This seemed like table stakes for a very long time but it really illustrated just how Go's compiler is much, much simpler than most in many areas.

On one hand it makes you wonder how much could be squeezed out of Go and how many basic things they're still leaving out of the compiler. On the other it tells part of the story of why Go compiles fast with the backdrop that despite this lack of work by the compiler people use Go productively all day and deliver services they are happy with from a performance perspective.

[0] - https://go.dev/doc/go1.17#compiler (2021-08)

Re: Go port of SQLite without CGo

#69

Post by masklinn »

pkolaczk wrote: post #66
adastra22 wrote: post #50

Earlier quoted context omitted.

CGO is a real problem for many people. Due to how Go handles its async-by-default concurrency model, every call to an FFI library requires setting up a stack and marshaling data. It ends up being rather expensive. So the "without CGo" stuff is about retaining the fast performance people expect of these highly optimized C libraries.

How does it really solve the FFI issue? Now instead of an FFI boundary between the app and the database, you have an FFI boundary between the database (now implemented in Go) and the operating system C API. And the database quite likely performs multiple calls into the C API per each user app query, so it might be in fact worse.

Interactions with the OS are managed by the go runtime, and on Linux it does direct syscalls (except when it tries to use vdso).

Every program needs to do syscalls at some point, so that’s built in. It is not “in fact worse”.

Re: Go port of SQLite without CGo

#70

Post by pkolaczk »

masklinn wrote: post #69
pkolaczk wrote: post #66

Earlier quoted context omitted.

How does it really solve the FFI issue? Now instead of an FFI boundary between the app and the database, you have an FFI boundary between the database (now implemented in Go) and the operating system C API. And the database quite likely performs multiple calls into the C API per each user app query, so it might be in fact worse.

Interactions with the OS are managed by the go runtime, and on Linux it does direct syscalls (except when it tries to use vdso). Every program needs to do syscalls at some point, so that’s built in. It is not “in fact worse”.

That applies only to the stuff exposed by the go stdlib, which is the lowest common denominator between various systems. Databases often use many system specific, advanced calls, to gain more control.
Post reply on HN