Live data from Hacker News

Do you even need a database?

dbpro.app

251–260 of 311 posts

Re: Do you even need a database?

#251
I love reading posts like these.

I will still reach for a database 99% or the time, because I like things like SQL and transactions. However, I've recently been working on a 100% personal project to manage some private data; extracting insights, graphing trends, etc. It's not high volume data, so I decided to use just the file system, with data backed at yaml files, with some simple indexing, and I haven't run into any performance issues yet. I probably never will at my scale and volume.

In this particular case having something that was human readable, and more importantly diffable, was more valuable to me than outright performance.

Having said that, I will still gladly reach for a database with a query language and all the guarantees that comes with 99% of the time.

Re: Do you even need a database?

#252

people wildly underestimate the os page cache and modern nvme drives tbh. disk io today is basically ram speeds from 10 years ago. seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit. a jsonl file and a single go binary will literally outlive most startup runways. also, the irony of a database gui company writing a post about how you dont a…

> people wildly underestimate the os page cache and modern nvme drives

And worse, overestimate how safe is their data!

All this fancy thing about not using a RDBMS could had been true only if the APIs and actual implementation across ALL the IO path were robust and RELIABLE.

But is not!

EVERY LAYER LIES

ALL of them

ALL OF TIME

That is why the biggest reason building a real database (whatever the flavor) is that there is no way to avoid pay performance taxes all over the place because you can't believe the IO and having a (single | some files) getting hammered over and over make this painfully obvious.

One of the most sobering experiences is that you write your IO with all the care in the world, let the (your brand new) DB run for hours, on good, great, hardware, and in less than a week you will find that that breaks in funny ways.

P.D: Was part of a team doing a db

Re: Do you even need a database?

#253
Let's put it this way. I always end up needing the functionality and ACID guarantees of a database. I always wish I had a database. But some times I'm forced to use the project's legacy data stores (often flat-file data lakes) and watch every wheel get reinvented as we struggle to glue consistency, transactions, a bespoke query language, etc. onto an unwilling pile of unstructured data.

Re: Do you even need a database?

#254
The sqlite benchmark is not very optimised.

Just adding:

    db.SetMaxOpenConns(runtime.NumCPU())
    db.SetMaxIdleConns(runtime.NumCPU())
Made the performance on my machine go from 27,700.99 r/s to 89,687.36 r/s.

I also tried making the get user a prepared statement, and storing the timestamp as an unix timestamp integer, but that didn't make much difference for me.

Re: Do you even need a database?

#256

The article is fine, but I wanted to call this out. "Every database you have ever used reads and writes to the filesystem, exactly like your code does when it calls open()." Technically not true. Applications like SQLite use mmap to map the file into a locally addressable memory space. This lets you skip the syscalls when reading and writing. The kernel can map that data in dynamically much faster than a userland pro…

> The kernel can map that data in dynamically much faster than a userland process can.

Not necessarily. The kernel's mmap implementation has quite a strong bias towards certains kinds of access patterns; deviate from them and it can become slower than read(2).

We tried using mmap(2) for audio file i/o in Ardour, and it got notably less bandwidth than just using read(2).

Re: Do you even need a database?

#257

The article is fine, but I wanted to call this out. "Every database you have ever used reads and writes to the filesystem, exactly like your code does when it calls open()." Technically not true. Applications like SQLite use mmap to map the file into a locally addressable memory space. This lets you skip the syscalls when reading and writing. The kernel can map that data in dynamically much faster than a userland pro…

> The kernel can map that data in dynamically much faster than a userland process can. Not necessarily. The kernel's mmap implementation has quite a strong bias towards certains kinds of access patterns; deviate from them and it can become slower than read(2). We tried using mmap(2) for audio file i/o in Ardour, and it got notably less bandwidth than just using read(2).

I'm curious if you tried different madvise strategies and if any of them worked better than others?

Re: Do you even need a database?

#258

I love SQLite, I love the idea, I love having something mature and lightweight, but like the author I discovered it's overkill or dare I say insufficient for certain use cases. I was building a client side dictionary app with search functionality and thought that using sqlite's wasm port would be the perfect solution. I used SQLite for a couple years and it started to wear on me. The database files were bigger than t…

The author of stringzilla, Ash Vardanian, has a bunch of really cool talks/lectures/demos. Highly recommend checking him out if you like that kind of thing :)

Re: Do you even need a database?

#259
> The index format is simple: one line per record, exactly 58 bytes: :\n.

It would be much better to write all of this as binary data, omitting separators.

• Since it’s fixed-width and simple, inspecting the data is still pretty easy—there are tools for working with binary data of declared schema, or you could write a few-liner to convert it yourself. You don’t lose much by departing ASCII.

• You might want to complicate it a little by writing a version tag at the start of the file or outside it so you can change the format more easily (e.g. if you ever add a third column). I will admit the explicit separators do make that easier. You can also leave that for later, it probably won’t hurt.

• UUID: 36 bytes → 16 bytes.

• Offset: 20 bytes (zero-padded base-ten integer) → 8 bytes.

• It removes one type of error altogether: now all bit patterns are syntactically valid.

• It’ll use less disk space, be cheaper to read, be cheaper to write, and probably take less code.

I also want to register alarm at the sample code given for func FindUserBinarySearch. To begin with, despite a return type of (*User, error), it always returns nil error—it swallows all I/O errors and ignores JSON decode errors. Then:

  entryID := strings.TrimRight(string(buf[:36]), " ")
That strings.TrimRight will only do anything if your data is corrupted.

  cmp := strings.Compare(entryID, id)
Not important when you control the writing, but worth noting that UUID string comparison is case-insensitive.

  offsetStr := strings.TrimLeft(string(buf[37:57]), "0")
Superfluous. ParseInt doesn’t mind leading zeroes, and it’ll probably skip them faster than a separate TrimLeft call.

  dataOffset, _ := strconv.ParseInt(offsetStr, 10, 64)
That’s begging to make data corruption difficult to debug. Most corruption will now become dataOffset 0. Congratulations! You are now root.

Re: Do you even need a database?

#260
post #258

I love SQLite, I love the idea, I love having something mature and lightweight, but like the author I discovered it's overkill or dare I say insufficient for certain use cases. I was building a client side dictionary app with search functionality and thought that using sqlite's wasm port would be the perfect solution. I used SQLite for a couple years and it started to wear on me. The database files were bigger than t…

The author of stringzilla, Ash Vardanian, has a bunch of really cool talks/lectures/demos. Highly recommend checking him out if you like that kind of thing :)

Yeah, I probably should since I want to add glob or regex support later.

It's a little out of the scope of stringzilla but I should still be able to use it for the heavy lifting

Post reply on HN