Live data from Hacker News

Go port of SQLite without CGo

gitlab.com

91–100 of 122 posts

Re: Go port of SQLite without CGo

#91

Beware: This library seems to have bugs that cause it to break on some writes. Recently held an event that heavily used this library and it broke on us half way through, had to wipe the database and switch it back to the cgo version (the data inside was mostly ephemeral, which was quite a relief).

Hey, sorry to hear that. I'm a contributor on the project and if you're able to open an issue (https://gitlab.com/cznic/sqlite/-/issues/new) with any info you have it would be very appreciated.

Re: Go port of SQLite without CGo

#92
post #79

Note that this is not “go port”. It’s C transpiled to go with their custom C compiler, and it’s transpiled differently for each architecture (as it’s using unsafe go heavily) I look at it like a neat exercise but not really something to use in production. (I actually use it in tests though.)

It is definitely neat, but I think it's more than an exercise. I believe he runs it through the extensive SQLite test suite (I've seen that before, though I can't see where that's stated now).

The bits in https://gitlab.com/cznic/sqlite/-/blob/master/tcl_test.go run the full suite of SQLite TCL tests as part of a Go test. That's in turn run as part a CI setup with results at https://modern-c.appspot.com/-/builder/?importpath=modernc.o....

Re: Go port of SQLite without CGo

#93
post #24

Earlier quoted context omitted.

Really tho, how much is build time vs runtime? Maybe is just work small projects but I rarely care about build time. Am I missing something?

Honestly, on a non-toy project, build times with cgo are _brutal_. I agree with you usually, but when a build time on a beefy computer switches from under a second to >1min you notice it. Linters and IDEs get slow when they check for errors, tests run slow, feedback drags, and all your workflows that took advantage of Go's fast compile times are now long enough that your flow and mental context disappear. I'm way mor…

In my experience, encapsulating the access to sqlite in a go package helps a lot with avoiding recompilation of the c source, which indeed is brutally slow. It acutally seems to be way slower than compiling with gcc from the command line. Anyone knows why this is the case?

Re: Go port of SQLite without CGo

#94
post #57

Earlier quoted context omitted.

Sure, but then DevOps aren't doing their jobs. No devs on production servers unless for tracking down issues, or servers configured for remote development. Which in both cases, don't require cross compilation to start with.

This comment leads me to believe that you’re used to working in one type of organisation to the point where you’re projecting it on your entire understanding of this incredibly broad industry.

Indeed, Fortune 500 consulting.

Re: Go port of SQLite without CGo

#95
post #42

Earlier quoted context omitted.

Cross compiling with zig is so easy. For example I use sqlite in Go projects on arm. The incantation is just: export GOARCH=arm64 export CC=zig cc -target aarch64-linux-musl export CXX=zig cc -target aarch64-linux-musl

Cross compilation is always easy when there are no dependencies to the target platform SDKs. When it goes beyond basic libc stuff, then it is when the fun starts.

It seems fairly straightforward - copy over the SDK header files and libraries to the project and then add these envs to the build:

    CC="/path/to/cross/compiler" CGO_CFLAGS=—I/path/to/sdk/header/files/include" CGO_LDFLAGS="-L/path/to/sdk/obj/files/lib”
I suppose with .so files, you might have to specify "-Wl,-rpath" if the SDK libraries don’t reside under /usr/lib or /usr/local/lib on the target system. But I haven’t actually tried any of this yet, so I could be wrong. Are there any gotchas that I’m missing?

Re: Go port of SQLite without CGo

#96
post #42

Earlier quoted context omitted.

Cross compilation is always easy when there are no dependencies to the target platform SDKs. When it goes beyond basic libc stuff, then it is when the fun starts.

It seems fairly straightforward - copy over the SDK header files and libraries to the project and then add these envs to the build: CC="/path/to/cross/compiler" CGO_CFLAGS=—I/path/to/sdk/header/files/include" CGO_LDFLAGS="-L/path/to/sdk/obj/files/lib” I suppose with .so files, you might have to specify "-Wl,-rpath" if the SDK libraries don’t reside under /usr/lib or /usr/local/lib on the target system. But I haven’t…

Now do the same as Windows UWP SDK, or iOS SDK.

You're missing all the tooling and OS specific steps required to produce a proper package.

Re: Go port of SQLite without CGo

#98
post #38

Why would you want to do this? SQLLITE is so stable. Why would you switch to some port that will be dead in a year?

Each call out to a C library from Go locks an OS thread rather than participating in the go routine scheduling. If stuff happens that makes those threads slow or have some issue, they just sit here. I had a server with C Kafka libraries and each time the network glitched the CPU would spike up wards and not decline. With pure go the scheduler is able to just swap out and ignore code that is stuck some where (unless it were in a busy loop).

With this SQLite implementation I can stick it in my server (as a background, near real time mirror of memory state to disk) without worrying that the main loop will get clogged or my real time path will be hurt.

I use glabarez’ wrapper which makes it have an API like other Go databases: https://github.com/glebarez/sqlite/ and hey have been very responsive to issues I raised.

I have been running load tests against it for a few weeks, and it is quite solid.

Re: Go port of SQLite without CGo

#99
post #47

Earlier quoted context omitted.

Pretty much any database is NOT written in Go, so what's the difference?

Also Bolt is pure go which is a highly performant KV store

I tested that for a while but it seemed to suffer lock contention with a lot of go routines trying to use one database at the same time.

Re: Go port of SQLite without CGo

#100
post #85

Earlier quoted context omitted.

You’re running graviton arm locally? It’s all docker containers in the end, so that’s what you get.

No, and why should I? I got into UNIX development back when the whole team shared a development server and we used telnet and X Windows for our "IDE". No one was running SPARC, RISC System/6000, PA-RISC, MIPS, Eclipse MV locally. When everyone keeps worshipping UNIX, maybe it is time to learn how grey beards used to develop for it.

> it starts by having a local environment that matches the OS of the servers

> No, and why should I?

These two statements are in direct conflict. CPU architecture is as much part of “matching the server” as operating system.

When I built software for SPARC, I was also running SPARC locally…

Post reply on HN