Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

301–310 of 311 posts

Re: Do you even need a database?

#301

Post by WatchDog »

jmull wrote: post #138

> Binary search beats SQLite... For a pure ID lookup, you're paying for machinery you're not using. You'll likely end up quite a chump if you follow this logic. sqlite has pretty strong durability and consistency mechanism that their toy disk binary search doesn't have. (And it is just a toy. It waves away the maintenance of the index, for god's sake, which is almost the entire issue with indexes!) Typically, people…

It's also just wrong, their SQLite benchmark is only using a single thread for the SQLite connection. It's much faster with multiple connections.

Re: Do you even need a database?

#302

Post by akdev1l »

trgn wrote: post #200
akdev1l wrote: post #183

Earlier quoted context omitted.

SQLite can do it

it's storage file is a csv? or do you mean import/export to csv?

You can import csv files into in memory tables and query them or you can use the csv extensions

$ sqlite3 :memory:

.import myfile.csv mytable

SELECT * FROM mytable;

$ sqlite3 :memory:

SELECT *

FROM csv_read('myfile.csv');

Re: Do you even need a database?

#303

Post by tnelsond4 »

Hendrikto wrote: post #284
tnelsond4 wrote: post #239

I love SQLite, I love the idea, I love having something mature and lightweight, but like the author I discovered it's overkill or dare I say insufficient for certain use cases. I was building a client side dictionary app with search functionality and thought that using sqlite's wasm port would be the perfect solution. I used SQLite for a couple years and it started to wear on me. The database files were bigger than t…

> The database files […] didn't compress super well That’s a good thing. If they were compressible that would indicate inefficiency in the SQLite format.

Well, I went and looked at it again and it seems like SQLite databases although larger than my custom format do compress to a 10% smaller size provided there's no indexes. But my custom format uses indexes so it's not really a good comparison. I guess I should see what it's like with indexes.

But at the end of the day SQLite just doesn't meet the needs of my project due to it's size and lack of free decompression support. Passing a decompression stage before giving it to SQLite is unwieldy and requires a separate decompression module and incurs a copy operation. I don't know enough about SQLite to attempt to compile and integrate the zstd decompressor into it.

Re: Do you even need a database?

#305

Post by koliber »

tstenner wrote: post #286
koliber wrote: post #167

Earlier quoted context omitted.

The reality is that things will be blazing fast in any language if you save things by PK in HashMaps.

In the benchmark Rust is more than 50% faster than the runner up

Correct. Order-of-magnitude-wise, it's roughly the same as the alternatives.

In the context of writing a new service for a new company, you should not spend one second thinking about whether your technical choices will allow you to serve 100,000 requests per second, or 150,000 requests per second. If you are, you are focusing on the wrong thing. If you get to 1,000 requests per second with a real paying client base you already achieved more than most dream of.

On the other hand, if you are optimizing a mature distributed low-latency equity trading system that is consuming ten's of thousands of market data ticks per second, a 50% improvement in performance on a 20 machine cluster might turn into some real $$$ savings. But that's not what this article is about.

Re: Do you even need a database?

#306

Post by leafarlua »

hunterpayne wrote: post #235
leafarlua wrote: post #42

Earlier quoted context omitted.

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

"NoSQL is the correct answer." No, no it isn't. It never is. Just as building your house on a rubber foundation isn't the correct answer either. This is just cope. Unless your use cases don't care about losing data or data corruption at all, NoSQL isn't the correct answer.

You are probably correct. I was just parroting what I have seem from the industry. Isn't it common for even big start ups to have decided to go with rubber for their foundation? It seems to not be a wise decision, yet many engineers do take this path. And that is why I end up confused on this type of discussion.

Re: Do you even need a database?

#307

Post by ngc248 »

leafarlua wrote: post #152
marcosdumay wrote: post #85

Earlier quoted context omitted.

No, when things change fast and unpredictably, NoSQL is worse than when they are well-known and stable. NoSQL gains you no speed at all in redesigning your system. Instead, you trade a few hard to do tasks in data migration into an unsurmountable mess of data inconsistency bugs that you'll never actually get into the end of. > is mostly bad design decisions and poor domain knowledge Yes, using NoSQL to avoid data mig…

Makes sense. But in this case, why NoSQL exists? What problems does it resolves and when should it be considered? I'm being naive, but fast changing environment has been one of the main advantages that I was taught from devs when it comes to NoSQL vs SQL (nosql being the choice for flexible schemas). So it is more about BASE vs ACID?

NOSQL works out really well when you know the access patterns you are working with, if not it will be like pegging a round hole with a square peg

Re: Do you even need a database?

#308

Post by nchmy »

tnelsond4 wrote: post #295
pdyc wrote: post #289

Earlier quoted context omitted.

what was your data size? i am surprised 800kb made a difference? using stringzilla was smart approach,my guess is it being unusually faster made all the difference.

Usually I'm dealing with about 20mb of compressed data, almost 100mb uncompressed. Even with only a couple mb of data SQLite still has a startup time of a couple hundred milliseconds on my phone. But that's a couple hundred milliseconds when loading a database that's already decompressed. When loading 100mb SQLite usually took a second or so which I didn't really like for a pwa. It took me quite a few attempts to get…

This is really cool. I'm working on stuff that is somewhat aligned with this - offline knowledge base/educational platform focused on things like appropriate technologies for rural people in the developing world. Storing in the browser and, more importantly, searching it is definitely one of the major challenges. (it's also just a much more dynamic app)

My main question about this is whether it can be dynamically/incrementally updated within the browser? Eg new material is available or edits have been made, so sync it from backend and it gets merged in.

I've been working on using rxdb to sync and store in browser - it can use its own indexeddb abstraction, sqlite or it's own OPFS-based DB. It can also load any of these into memory in its memory-mapped mechanism. I've also made a mechanism to load everything into flexsearch in a sharedworker, so that you can do full text search fairly performantly.

It's a lot of complexity though. I'd be curious to hear any of your thoughts. Or even to chat if you're open to it!

Re: Do you even need a database?

#309

Post by tnelsond4 »

nchmy wrote: post #308
tnelsond4 wrote: post #295

Earlier quoted context omitted.

Usually I'm dealing with about 20mb of compressed data, almost 100mb uncompressed. Even with only a couple mb of data SQLite still has a startup time of a couple hundred milliseconds on my phone. But that's a couple hundred milliseconds when loading a database that's already decompressed. When loading 100mb SQLite usually took a second or so which I didn't really like for a pwa. It took me quite a few attempts to get…

This is really cool. I'm working on stuff that is somewhat aligned with this - offline knowledge base/educational platform focused on things like appropriate technologies for rural people in the developing world. Storing in the browser and, more importantly, searching it is definitely one of the major challenges. (it's also just a much more dynamic app) My main question about this is whether it can be dynamically/inc…

I'm not sure I follow exactly, but if I understand you mean that when the database file is updated that the app updates? Right now on app load it updates the service worker and shows the files in cache first. If there's a newer file it fetches it in the background, it then sends a message to the client that there is a new file. I haven't implemented the next part yet but it should be able to invalidate the current file and load the new file without refreshing the page. Right now the new files will load the refresh after the new service worker is activated.

But the page still had to be refreshed to load the new service worker. I'm looking into ways to cut the time to loading the new files down because right now you have to refresh the page 3 times for the new files to take over.

The .peak files aren't designed to be a database that you can just add to during runtime, they're rather static and highly efficient in that context. But it's easy to edit the source files and generate a new .peak file from that.

You can take a folder of any kind of files and run peakgen on it and it will create a compressed .slab file that you can search and fetch results from just like the .peak files. I first saw that done with SQLite and I really liked it, so I knew I could do it too.

If you want to chat you can shoot me an email.

Post reply on HN