Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go port of SQLite without CGo

gitlab.com

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Go port of SQLite without CGo

#111

Post by ikiris »

willi59549879 wrote: post #65
lopkeny12ko wrote: post #34

Earlier quoted context omitted.

Why wouldn't they? Are you implying that, over the course of decades of Google's existence and thousands of software projects, not once has there ever had the need for a lightweight SQL store that can be embedded with a server or CLI tool? Your comment truly does SQLite a disservice. SQLite is probably one of the most impressive pieces of software to have ever graced this earth.

Google's practice is that every service they build is built that way that it can scale to millions of users. On servers i don't think they would use sqlite unlike in android. If i build stuff i usually start with sqlite.

It's much more that servers are ephemeral, so the DBs are as a service from another team.

Would you run sqlite in kubernetes? (if so... whyyyyyyy?)

Re: Go port of SQLite without CGo

#112

Post by eschaton »

mariusor wrote: post #60
eschaton wrote: post #28

What’s the point of something like this instead of just using SQLite? Seriously, “foo but in bar!” is not exactly the sort of thing that solves real problems for most people.

What exactly do you think "sqlite" is? By the way you asked the question, it seems like you're assuming it's a separate database that you execute alongside your application, similarly to postgres/mariadb/etc. That's not the case though.

I’ve owned the SQLite integration in a major OS, I know quite well what it is. (And I’m also quite familiar with the difference between static and dynamic/shared libraries too, thanks.)

Rephrasing my question, why wouldn’t you just use the SQLite library from Go, instead of cross-compiling it from C to Go? It sounds like the non-enthusiast “reimplement everything in my favorite language” answer is that Go’s FFI is a pain, even for C.

Re: Go port of SQLite without CGo

#113

Post by Mawr »

59nadir wrote: post #68
icholy wrote: post #26

Earlier quoted context omitted.

In addition, a lot of optimizations (like loop unrolling) are left on the table to preserve Go's fast compile times.

I was surprised with the announcement that Go's standard compiler had register allocation for function arguments[0] added to it. This seemed like table stakes for a very long time but it really illustrated just how Go's compiler is much, much simpler than most in many areas. On one hand it makes you wonder how much could be squeezed out of Go and how many basic things they're still leaving out of the compiler. On the…

It goes to show the impact of optimizing from top-down instead of bottom-up can have. Go was designed with performance in mind ("mechanical sympathy") as opposed to most other GC languages for which performance was an afterthought.

If you get high-level language constructs right, you'll get great performance by default, even with a simple compiler:

- structs instead of objects

- value types

- exposed pointers

- straightforward allocation semantics

Most compilers out there are busy fighting bad language designs. There's only so much a compiler can do when faced with fat objects hidden behind a spider web of pointers. So they have to do these microoptimizations to claw back at least some performance.

Re: Go port of SQLite without CGo

#114

Post by [deleted] »

djbusby wrote: post #24
collinvandyck76 wrote: post #23

Earlier quoted context omitted.

at segment we benchmarked https://github.com/segmentio/ctlstore against this driver. We saw about a 50% hit to read performance, so we didn't move forward with it, but the improvements in service build times were really appealing.

Really tho, how much is build time vs runtime? Maybe is just work small projects but I rarely care about build time. Am I missing something?

[deleted]

Re: Go port of SQLite without CGo

#115

Post by deepsun »

mdaniel wrote: post #109
yieldcrv wrote: post #35

Earlier quoted context omitted.

line 213 if you’re on mobile

Don't feel bad, GitLab's line navigation thing doesn't work reliably on desktop, either. I guess they put that on the list of the other 15,000 issues that are "on the backlog"

I dunno, works ideally for me on both mobile and desktop, both Firefox.

Re: Go port of SQLite without CGo

#116

Post by seanp2k2 »

tptacek wrote: post #8

If you cross-compile (which everybody using macOS to developer for Linux servers does), it's annoying to introduce SQLite to your project, because immediately the conventional `GOOS= GOARCH= go build` trick stops working. But, in case it's helpful, it's also really easy to set up a full cross-compiling environment, either with Zig or with Filippo's musl-cross: https://words.filippo.io/easy-windows-and-linux-cross-com…

As someone who has had to develop desktop software (ugh) with true cross-platform support for MacOS, Linux, and Windows (thanks Animation and VFX industry) using golang, I can assure everyone that it is not “really easy” to make things like Qt compile for different OSes on different OSes. IIRC trying to build for M1 Mac on AMD64 Windows was the worst.

Re: Go port of SQLite without CGo

#117

Post by seanp2k2 »

tptacek wrote: post #8

If you cross-compile (which everybody using macOS to developer for Linux servers does), it's annoying to introduce SQLite to your project, because immediately the conventional `GOOS= GOARCH= go build` trick stops working. But, in case it's helpful, it's also really easy to set up a full cross-compiling environment, either with Zig or with Filippo's musl-cross: https://words.filippo.io/easy-windows-and-linux-cross-com…

As someone who has had to develop desktop software (ugh) with true cross-platform support for MacOS, Linux, and Windows (thanks Animation and VFX industry) using golang, I can assure everyone that it is not “really easy” to make things like Qt cross-compile with golang. IIRC building for Mac on Windows was the worst.

Re: Go port of SQLite without CGo

#118

Post by pkolaczk »

lanstin wrote: post #98
tetete wrote: post #38

Why would you want to do this? SQLLITE is so stable. Why would you switch to some port that will be dead in a year?

Each call out to a C library from Go locks an OS thread rather than participating in the go routine scheduling. If stuff happens that makes those threads slow or have some issue, they just sit here. I had a server with C Kafka libraries and each time the network glitched the CPU would spike up wards and not decline. With pure go the scheduler is able to just swap out and ignore code that is stuck some where (unless i…

Doesn't Go runtime offer anything like spawn_blocking in Rust async runtimes? Spawn_blocking allows to run any blocking operation without blocking the main executor threads, by spawning blocking operation into a separate thread-pool. This way there is no reason to worry "that the main loop will get clogged or my real time path will be hurt."

Re: Go port of SQLite without CGo

#119

Post by randomwebdev »

It is really a cool project. We are using it as a default driver in PocketBase and although it is not a "drop-in" replacement of the CGO alternative `mattn/go-sqlite3` (different dsn format, some differences in the error messages, etc.), with a small abstraction it works fine for most cases and greatly simplify cross compilation.

Performance wise I haven't done intensive benchmark tests yet, but from my local experiments last year it performed ~1.5-2x slower than the CGO version for some queries (it is especially noticeable with LIKE expressions on large string data), but as mentioned previously, for most use cases it is already good enough.

Here are the most recent benchmarks I could find (I think from one of the maintainers of the lib) comparing the CGO version and the pure Go port - https://docs.google.com/spreadsheets/d/1YOP1D_ZhuR-ednQhTH6S...

Re: Go port of SQLite without CGo

#120

Post by yencabulator »

cdogl wrote: post #5
philosopher1234 wrote: post #3

Earlier quoted context omitted.

What makes it slower?

It’s hard for Go to beat C in a straight benchmark - Go is garbage collected and allocations often have extra overhead due to the memory layout of interfaces.

However, it is absolutely plausible for Go-calling-Go to beat Go-calling-C-via-CGo, since CGo has plenty of overhead.
Post reply on HN