Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database ruins all good ideas

squarism.com

Post reply on HN
Print view View original

151–160 of 165 posts

Re: The database ruins all good ideas

#151

Post by JoshuaDavid »

sonthonax wrote: post #141
nesarkvechnep wrote: post #132

Earlier quoted context omitted.

Clearly you are not a Node.is dev.

Or a Django dev, or a Rails dev, or any other ‘framework developer’.

I'm a "framework developer" (and primarily a PHP framework developer at that) and I definitely got those kinds of questions the last time I was interviewing, and the company I work for still asks schema design questions.

Designing a good schema is, in my opinion, the single most important thing for creating useful software. I would much rather work on janky spaghetti code that operates on a clean and well-mapped schema than beautiful SOLID, commented, well-documented code that runs against a terrible, fragmented datastore.

Re: The database ruins all good ideas

#152

Post by jmchuster »

heurisko wrote: post #87
noisy_boy wrote: post #83

Earlier quoted context omitted.

> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD…

I always go further and use the same database for my tests e.g. use postgres rather than H2. It is easy enough to setup and tear down using docker.

and now as easy as adding `@AutoConfigureEmbeddedDatabase`

https://github.com/zonkyio/embedded-database-spring-test

also helps with having it verify all your database migrations as well

Re: The database ruins all good ideas

#153

Post by sonthonax »

cornel_io wrote: post #144
sonthonax wrote: post #141

Earlier quoted context omitted.

Or a Django dev, or a Rails dev, or any other ‘framework developer’.

As a matter of fact, I have done both of those a lot, and I'd argue that knowing how model design interacts with the database is arguably more important because if what footguns the ORMs can be. But you are correct, interviews for those types of jobs for whatever reason don't tend to focus as much on DB stuff, though they really really should given that a quarter of your job will end up being unfucking the system awa…

> end up being unfucking the system away from the poor choices

The amount of times I've had to fix UUID4s stored as a PK string in my career... It's bloody infuriating, it's like your typical 'ORM developer' just doesn't understand that: a UUID is a 128bit number; doesn't understand that in Postgres there's an extension for it (which Django doesn't use); how much slower a string lookup is compared to a numeric lookup; and how that indexes are physically stored in order, meaning that a UUID4 will be inserted at random points on disk utterly killing insert performance.

Generally these kind of performance fuckups happen because you get developers who treat the database as a magical performance box that needs an occasional index. One of the worst systems I've seen was an accounting system that used triggers to update the balance, so you'd insert a ledger that would update a single row. Guess what that'd do? It'd block all other inserts until the update was committed. The previous developers kept on trying to improve the performance by adding more servers, but it was just burning money because they were inadvertently locking a row (rather than calculating balances on the fly).

Re: The database ruins all good ideas

#154

Post by yowlingcat »

sonthonax wrote: post #92
simonw wrote: post #85

Earlier quoted context omitted.

I find using libraries like factory_boy and pytest's fixtures mechanism makes that horrific domino stack pretty manageable.

Man I fucking hate factory boy, by default it fuzzes the input meaning your tests are non deterministic. Honesty, I don’t understand why people use it; how many models do people have where meta programming factory classes makes sense?

I think it comes down to whether you want to maintain a fixture or a factory. In my experience, it can be easy to set up a fixture but over time it increasingly becomes a pain -- sometimes enough that setting up a factory is worth it.

Re: The database ruins all good ideas

#155

Post by majewsky »

> Or maybe you use sqlite in dev/tests and something bigger in prod.

I had that for a while. SQLite for the test suite and Postgres for actual runs. It was a shitshow, having to support two separate SQL dialects with separate feature sets. Then I realized that you can actually bootstrap and start a Postgres in tmpfs in about one second, so that's what my tests do nowadays.

Re: The database ruins all good ideas

#156

Post by majewsky »

AdrianB1 wrote: post #16

A dockerized database server is something I cannot understand; I understand bare metal, I can accept virtualized, but I cannot find a good used case for a mid sized or large server (dozens of gigabytes to dozens of terabytes) dockerized and I don't know why a smaller server is a problem.

For our team, the reason is standardization. We have our entire baremetal infrastructure running Kubernetes. There is no process for running anything in not-a-Docker-container. This does come with some caveats for stateful databases, but it's still easier to manage than a special-snowflake environment for database deployments.

Re: The database ruins all good ideas

#157

Post by sebazzz »

majewsky wrote: post #155

> Or maybe you use sqlite in dev/tests and something bigger in prod. I had that for a while. SQLite for the test suite and Postgres for actual runs. It was a shitshow, having to support two separate SQL dialects with separate feature sets. Then I realized that you can actually bootstrap and start a Postgres in tmpfs in about one second, so that's what my tests do nowadays.

Nice, does MSSQL have something similar? I know there is localDB, but that has its quirks and databases need to be cleaned up manually.

Re: The database ruins all good ideas

#158

Post by ric2b »

KronisLV wrote: post #107
ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

But what about when there will be problems with your single point of failure? Sometimes you'll run into situations where an instance of any software has degraded performance for reasons beyond either your control or understanding. That's one of the situations where horizontal scaling let's you deal with such circumstances better, instead of outright impacting everything and everyone connected to it. A front end web s…

> A front end web server container crumbles under the load? No worries, just redirect the traffic to others.

You can do basically the same thing with read replicas, promote one of them to primary and shut down the machine causing issues.

Re: The database ruins all good ideas

#159

Post by erichocean »

tremon wrote: post #35
simonw wrote: post #29

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…

horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas There's one feature that I'd like to see in that area: partitioned writable replicas. In the same vein that you can partition the table storage across an index, I'd like it to be possible to assign different writers to different parts of a table/database. Of course, you'd still need a single primary r…

> I'd like it to be possible to assign different writers to different parts of a table/database

If you're happy letting the database do that for you, that's how current CockroachDB works.

Re: The database ruins all good ideas

#160

Post by xemdetia »

HWR_14 wrote: post #149
xemdetia wrote: post #79

Earlier quoted context omitted.

I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern envi…

Thanks for giving me some thoughts to turn over in my head. I have to admit, having been able to support most of my projects on a simple LAMP stack, I may simply be overly worried about scaling. But my hope is always to have a sideproject skyrocket. Did you read the article a little while back about Discord's challenges with storing tons of messages and how they upgraded their tech stack? Did it make sense to you?

The discord post made sense because they understood what their workload was and did the type of 'classifying' queries that I previously described. They also decided to move off Mongo when they were hitting the 'real limits' of the underlying system from not being able to fit critical things in RAM. So they had a solid understanding of what was going on and they moved to rectify it. I don't want to presume to know anything about your sideproject but there's always going to be an limit you hit in any system, and you just need to build monitoring to understand when you are approaching that cliff.

They also had some clear characteristics of their system for their critical query that they could take advantage of: a new message is written to generally once, the messages table scales linearly, and updates and deletes are relatively rare. Using those things they were able to seek out a datastore that helped maximize the thing that's happening the most which is the write, and they were able to engineer a system accordingly. They also know that the messages would grow linearly without bound at a variable rate, so they could plan for that capacity. To me, that makes a lot of sense.

Especially if it is a side project you can't over-engineer capacity but you can do at least the measured math of what actions are actually happening in your system and what are your expectations for those queries. I would suggest that often times you'd want to focus on your critical queries that if slowed down would have the biggest net negative, but you'd have to find those. Every user action in a system should be attributable to some combination of reads or writes that can be reasoned about to try and figure out where there might be problems of contention, linear growth, exponential growth or growth up to a limit for a table, or something else. Either way, once you start classifying some of the critical paths in your system often either an answer or better questions start to come to the top. Without some of this information you can't back of the envelope whether or not your system can take a perceived load.

Post reply on HN