Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

291–300 of 311 posts

Re: Do you even need a database?

#291

Post by direwolf20 »

pseudosavant wrote: post #115
koliber wrote: post #59

I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications." One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that exec…

SQLite has become my new go-to when starting any project that needs a DB. The performance is very fast, and if anything is ever successful enough to outgrow SQLite, it wouldn't be that hard to switch it out for Postgres. Not having to maintain/backup/manage a separate database server is cheaper and easier.

If it's a server app, I almost always have to switch to Postgres eventually so now I start with Postgres.

Re: Do you even need a database?

#292

Post by wmanley »

briandw wrote: post #232

The article is fine, but I wanted to call this out. "Every database you have ever used reads and writes to the filesystem, exactly like your code does when it calls open()." Technically not true. Applications like SQLite use mmap to map the file into a locally addressable memory space. This lets you skip the syscalls when reading and writing. The kernel can map that data in dynamically much faster than a userland pro…

SQLite in mmap mode (not the default) will use mmap for reading. It will still use write using `pwrite` so it can detect and recover from write failures.

See https://www.sqlite.org/mmap.html :

> The default mechanism by which SQLite accesses and updates database disk files is the xRead() and xWrite() methods of the sqlite3_io_methods VFS object. These methods are typically implemented as "read()" and "write()" system calls which cause the operating system to copy disk content between the kernel buffer cache and user space.

> Beginning with version 3.7.17 (2013-05-20), SQLite has the option of accessing disk content directly using memory-mapped I/O and the new xFetch() and xUnfetch() methods on sqlite3_io_methods.

The principal advantage to mmap in my mind is that the cache is shared among your SQLite connections.

Re: Do you even need a database?

#293

Post by econ »

Back when Ajax was a hip new thing someone asked me why xml? We later answered that question with JSON of course but his hypothetical solution was much more hilarious. I'm calling it the html database. He made two mockups. One where everything was a richly formatted static html file and one that used a blogging platform. It had things like archives by month each with their own url. A paginated front page with chunk from each posting.

It was pretty funny and I couldn't really see something wrong with it.

In his demo he pulled in a document with a table in it and inserted the table into a div. One version used the domparser but he also made one with and and chopped out a substring.

In the 80's, because of resource constraints, a standard thing to ask was "why is my data not formatted the way I need it?"

IOW when to format it properly? when it is created? When it is consumed? Both? At some other time?

If you are going to create it one time and consume it 100 000 times html is a great "DB"

Re: Do you even need a database?

#294

Post by econ »

graemep wrote: post #282
koliber wrote: post #59

I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications." One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that exec…

If you look at those "When do you actually need a database?" constraints I think its missing consistency which prevents bugs and makes debugging easier. When you combine all those a database is a better alternative for all but the simplest cases.

At some point you end up writing a lot of code to say something that is a single word on the db query.

Perhaps (besides simple things) if you have many millions of users and no money. Or if you need something a DB is truly bad at.

Re: Do you even need a database?

#295

Post by tnelsond4 »

pdyc wrote: post #289
tnelsond4 wrote: post #239

I love SQLite, I love the idea, I love having something mature and lightweight, but like the author I discovered it's overkill or dare I say insufficient for certain use cases. I was building a client side dictionary app with search functionality and thought that using sqlite's wasm port would be the perfect solution. I used SQLite for a couple years and it started to wear on me. The database files were bigger than t…

what was your data size? i am surprised 800kb made a difference? using stringzilla was smart approach,my guess is it being unusually faster made all the difference.

Usually I'm dealing with about 20mb of compressed data, almost 100mb uncompressed. Even with only a couple mb of data SQLite still has a startup time of a couple hundred milliseconds on my phone. But that's a couple hundred milliseconds when loading a database that's already decompressed. When loading 100mb SQLite usually took a second or so which I didn't really like for a pwa.

It took me quite a few attempts to get something faster than SQLite. My new format loads instantly because I'm just casting the data to a struct. The only thing that takes time is decompressing, but that's still faster than loading the uncompressed via SQLite. My phone loads 100mb from 20mb compressed in about 400ms.

But writing my own format gives other benefits like being able to extract all the HTML tags and capital letters beforehand for fast and sensible search and reconstructing it on render. It's also just way easier for me to edit tsvs with markers for what parts are indexed and have that transformed into an indexed format with 3 indexes.

Also, with SQLite I was just running one module, but with my new format I'm running about 20 instances of it because it keeps the data nicer, more manageable and makes everything very parallel. Though I keep the number of web workers to 2 because it doesn't seem to benefit much to increase it more.

https://github.com/tnelsond/peakslab

Re: Do you even need a database?

#296

Post by AlienRobot »

econ wrote: post #293

Back when Ajax was a hip new thing someone asked me why xml? We later answered that question with JSON of course but his hypothetical solution was much more hilarious. I'm calling it the html database. He made two mockups. One where everything was a richly formatted static html file and one that used a blogging platform. It had things like archives by month each with their own url. A paginated front page with chunk f…

He invented the RESTful DB.

Re: Do you even need a database?

#297

Post by locknitpicker »

vovanidze wrote: post #207
locknitpicker wrote: post #81

Earlier quoted context omitted.

> weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess. I don't think it makes any sense to presume everyone around you is brainwashed and you are the only soul cursed with reasoning powers. Might it be possible that "we" are actually able to analyse tradeoffs and understand the value of, say, h…

i think you missed the "on day 1" part of my comment. k8s, iac, and observability are incrdible tools when you actually have the scale and team to justifiy them. my point is strictly about premature optimizaton. ive seen teams spend their first month writing helm charts and terraform before they even have a single paying user. if you have product-market fit and need zero-downtime rollbacks, absolutly use k8s. but if…

> i think you missed the "on day 1" part of my comment. k8s, iac, and observability are incrdible tools when you actually have the scale and team to justifiy them.

No, not really. It's counterproductive and silly to go out of your way to setup your whole IaC in any tool you know doesn't fit your needs just because you have an irrational dislike for a tool that does. You need to be aware that nowadays Kubernetes is the interface, not the platform. You can easily use things like minikube, k3s, microk8s, etc, or even have sandbox environments in local servers or cloud providers. It doesn't matter if you target a box under your desk or AWS.

It's up to you to decide whether you want to waste your time to make your life harder. Those who you are accusing of being brainwashed seem to prefer getting stuff done without fundamentalisms.

Re: Do you even need a database?

#298

Post by chrisweekly »

mike_hearn wrote: post #176
chrisweekly wrote: post #168

Earlier quoted context omitted.

In the spirit of helpfulness (not pedantry) FYI "knocked up" means "impregnated". Maybe "put together"?

Ah, this must be a British vs American English thing, thanks for the info. Yes I meant it in this sense: "If you knock something up, you make it or build it very quickly, using whatever materials are available." https://www.collinsdictionary.com/dictionary/english/knock-u...

Heh, thanks, yes, your meaning was obvious from context alone. I'm really surprised not to have encountered the British usage before (or so rarely). Maybe owing to potential for huge misunderstandings. ("Jimmy knocked her up" -> woke in the night || impregnated and abandoned.)

TIL

Re: Do you even need a database?

#299

Post by Anamon »

DB-less is underrated. You can take it even further.

One of my favourite websites ever loads its entire main content index into the client as a JSON file. It's about 5 MBs I think, so maybe 10% of the ad and autoplay video crap most "modern" websites make you download in the first few seconds. Except that this initial download actually gives you an amazing UX in return, because all the searching, filtering, pagination etc. happens entirely client-side without further network requests. I don't know any other website that is as much of a joy to just browse through. You never wait for anything until the point that you found what you want and actually want to open a detail page. Just wonderful.

Re: Do you even need a database?

#300

Post by ghc »

evanelias wrote: post #246
ghc wrote: post #117

Earlier quoted context omitted.

> Oracle® Database Platform Guide 10g Release 2 (10.2) for Microsoft Windows Itanium (64-Bit) Well, I guess that at least confirms Oracle on Itanium (!?) still supported RAW 5 years ago. I'm guessing everyone's on ASM by now though, if they're still upgrading. I ran into a company not long ago with a huge oracle cluster that still employed physical database admins and logical database admins as separate roles...I wou…

> still supported RAW 5 years ago I seem to remember Oracle 10g was first released over 20 years ago? It has been EOL for much longer than 5 years...

Oh you're right! I was looking at the last documentation update timestamp, but the original release was 2006. That makes a lot more sense than Itanium support in 2021.
Post reply on HN