Live data from Hacker News

Mistakes Beginners Make When Working with Databases

craigkerstiens.com

201–209 of 209 posts

Re: Mistakes Beginners Make When Working with Databases

#201
post #188
post #175

Earlier quoted context omitted.

Oh wow, that's definitely good to know. I do quite a number of migrations and so far this hasn't bitten me, but it could.

Migrating from Mysql to Postgres might be a good migration :-)

Sadly that's not always an option for me...

Re: Mistakes Beginners Make When Working with Databases

#202

Earlier quoted context omitted.

Denial isn't the same as argument. The definition of a UUID pretty much means they are. Enough bits and the chances of collision are less than that of a superintelligence spontaneously evolving in your morning coffee and hacking your computer. And its past time amateurs stopped thinking they've come up with the perfect UUID (timestamp + server id! Perfect! Until I set my clock back, or repurpose the server) and just…

Timestamp + server id can be uniquely guaranteed per server without coordination, so yes, it works fine :P Also, modern computers have monotonic clocks available and can use things like PTP.

Well you need to ensure that two servers never re-use the same ID so they can't come up with their own ID and there must be some form of co-ordination there. You also must ensure that you never enter two things into the database from the same server within one tick of the clock. Monotonic doesn't mean you'll never get the same number twice, just that it always changes in the same direction (in this case, the timestamp never gets lower, but nothing stops you getting the same number twice).

Re: Mistakes Beginners Make When Working with Databases

#203
post #158

Earlier quoted context omitted.

The problem is there's very few examples of real-world data that actually matches a primary key - i.e. is guaranteed to be unique and never changes. I've been burned by this so many times. In reality, everything needs to at least have some capability to change.

Unique and never changes can be two things concatenated, not one thing. If that concatenation is too long you can ram anything thru a hash to get a constant length smoothly distributed key. Its also fun to use "weak" hashes for this because it trolls wanna be security types who don't understand the application. Given the above, some important things concatenated and hashed works. Can always add an application ID or i…

> So something that's unique is NOW() (assuming low enough sample rate LOL)

We can all invent algorithms for nice unique values when we include the caveat "apart from the edge cases". It's those edge cases that bugger everything up.

Re: Mistakes Beginners Make When Working with Databases

#204

Earlier quoted context omitted.

The thing is, for most cases relevant to web apps, you don't even actually want binary data. Your application isn't going to touch the file itself anyway - a client somewhere is going to be the one processing and downloading the file. There's never a circumstance where dealing with hundreds of kilobytes or more of data is going to be faster than a 50-character file name, regardless of where it's stored.

But if you replicate your DB, now you also have to replicate your filesystem. Easier to just segregate BLOBs in their own tables and let DB replication do all the work.

It depends on why you're replicating but I would say in most cases you don't need to.

If you're forking your DB (i.e. there will now be two copies that will diverge from each other) than sure, you would need to replicate your filesystem. But that's not complicated (copying directories is a solved problem).

If you're talking about replication for performance reasons, then DB replication and file replication aren't necessarily going to go hand-in-hand, and the solutions are going to look fairly different (for web apps, "file replication" probably looks like a CDN).

Re: Mistakes Beginners Make When Working with Databases

#205

Earlier quoted context omitted.

Does Varnish preload your images from cache? What if 100 people hit the page at the same time for an uncached image? I am not saying it can't be done, just having trouble figuring out the benefits.. its easier to store an image in cloud storage over in a database... and cheaper.. and less bug prone...

Varnish will actually hold those 100 connections open while it dispatches 1 single request to the backend. When that request returns (assuming the cache control headers allow it), it will serve that one response to all 100 requestors. It's a very powerfull tool for anything that's even slightly cachable.

If your site has 1 varnish, if your site has 20 varnishes it would do 20 connections.

Re: Mistakes Beginners Make When Working with Databases

#206
post #158

Earlier quoted context omitted.

The problem is there's very few examples of real-world data that actually matches a primary key - i.e. is guaranteed to be unique and never changes. I've been burned by this so many times. In reality, everything needs to at least have some capability to change.

Unique and never changes can be two things concatenated, not one thing. If that concatenation is too long you can ram anything thru a hash to get a constant length smoothly distributed key. Its also fun to use "weak" hashes for this because it trolls wanna be security types who don't understand the application. Given the above, some important things concatenated and hashed works. Can always add an application ID or i…

> Unique and never changes can be two things concatenated,

No, for a good primary key, the "unique" parts can be due to concatenation, but the whole key (and thus all the elements) needs to be unchanging. [0]

[0] Modern databases can actually deal with changeable PKs, though with a potentially serious performance hit, but in many use cases where data travels outside the database for some kind of interaction where the results need to get reentered into the database, this is still a problem, since you can't (in any general way) cascade updates to things (which may not even be online systems, e.g., paper records feeding human processes) outside of the DB.

Re: Mistakes Beginners Make When Working with Databases

#207
post #202

Earlier quoted context omitted.

Timestamp + server id can be uniquely guaranteed per server without coordination, so yes, it works fine :P Also, modern computers have monotonic clocks available and can use things like PTP.

Well you need to ensure that two servers never re-use the same ID so they can't come up with their own ID and there must be some form of co-ordination there. You also must ensure that you never enter two things into the database from the same server within one tick of the clock. Monotonic doesn't mean you'll never get the same number twice, just that it always changes in the same direction (in this case, the timestam…

I'm aware. Step 1 is generally still considered under the "eventual consistency" umbrella (fork-join specifically). Step 2 doesn't really matter because all you actually need is monotonicity from the clock; locally it's really easy to guarantee uniqueness (and remember, it's fine to delay transactions as long as they commit eventually, which is one reason CAP is kind of imprecisely stated; you're only unavailable if you never respond to a request, when in practice it's really trying to convey latency requirement).

Re: Mistakes Beginners Make When Working with Databases

#208

Earlier quoted context omitted.

Shouldn't this risk be mitigated with authorization rules? Or do we assume we are delivering pages without any type of auth first?

You should allow to reset password to the users without authentication (and therefore without authorization). That's the nature of password reset link.

Oh of course. Good point.

Re: Mistakes Beginners Make When Working with Databases

#209

If only developers learned more about the DB engines they use and their capabilities. I've seen so many bone headed decisions (EAV "schemas" being a recurring one) because it was easier to program around the DB than learn how to use it proper. It's not because a few classes insulate you from the bone headed SQL that it's a good idea.

EAV isn't all bad, there are use-cases for it. If you have a highly dynamic data (user configurable) then EAV might be a good way to STORE that data. Is it ugly, it sure is. Moving read out to a cache layer, and search out a purpose built system, the DB becomes a persistence engine. You have solved the other issue you don't mention with EAV, and thats performance. Edit (my ability to post should be forbidden till I h…

Most EAV implementations I've seen had under 10 distinct "schema-less" virtual tables (sorry about lack of terminology) with perhaps 40-60 distinct "columns" and the different variations of a record types usually had consistent attributes.

Basically the whole thing could've been replaced by 5-10 tables. Maybe taking advantage of Postgres OORDBMS capabilities for the common columns.

Queries tended to group complex CTEs and multiple self-joins. You know it's an anti-pattern when devs start complaining about postgres join performance and you see they got two tables being joined 15 times...

Post reply on HN