Live data from Hacker News

Things I wished more developers knew about databases

medium.com

381–390 of 464 posts

Re: Things I wished more developers knew about databases

#381
post #200

Earlier quoted context omitted.

Go sql/database uses its own connection pool. But still that shouldn't create any problems. I have seen the reverse where apps that assumed temporary tables stick around from statement to statement without an explict `txn` (which regular postgres connections don't need) clearly failed. But I have not seen the issue you talk about. My wild guess would be that the Go code never closed the result/rows which caused eithe…

The database/sql package isn’t very magical. The comment about magical library makes me think it was some other package.

Any sufficiently advanced technology is indistinguishable from magic. If your source of knowledge is stack overflow or a youtube video posted on linkedin you already live in a world filled with magic.

Re: Things I wished more developers knew about databases

#382
post #318
post #292

Earlier quoted context omitted.

Why not judge people by the result of their work, instead of their age?

Because then it’s harder to deal with our own imposter syndromes if we can’t blame it on the youth and hold their heads in the toilet while giving them the professional-development equivalent of a wedgie. This was discussed at length in last week’s “Grey Beard Weekly” newsletter.

Now that you have advertised the newsletter. Maybe share a link too.

Re: Things I wished more developers knew about databases

#383
post #276

Earlier quoted context omitted.

Go code often reinvents/reimplements a lot of things from scratch, reintroducing problems that have been addressed long ago in other systems. It's like this new trend, let's rewrite everything in Go to be cool. Financially makes little to no sense.

Let’s not just target Go with that sentiment, it applies almost universally, just in varying degree. Counterpoint: how is anyone supposed to learn, if not from their mistakes? We might worry about the blast radius, but there’s no compression algorithm for experience.

OK. Rust, too.

Re: Things I wished more developers knew about databases

#384
post #329

Earlier quoted context omitted.

Let’s not just target Go with that sentiment, it applies almost universally, just in varying degree. Counterpoint: how is anyone supposed to learn, if not from their mistakes? We might worry about the blast radius, but there’s no compression algorithm for experience.

Another option is to learn from the mistakes of others.

People rarely post their mistakes to stack overflow.

Re: Things I wished more developers knew about databases

#385
post #276

Earlier quoted context omitted.

Go code often reinvents/reimplements a lot of things from scratch, reintroducing problems that have been addressed long ago in other systems. It's like this new trend, let's rewrite everything in Go to be cool. Financially makes little to no sense.

guess what other shiny new language is doing that, too...

Zig?

Re: Things I wished more developers knew about databases

#386

Earlier quoted context omitted.

> Rails' implementation of enums is a good example of this. The advantage of ActiveRecord enums (vs db-native) is that you can change the list of valid values without having to run a whole ALTER TYPE - and all the overhead that would entail in doing without downtime in a production db.

And the disadvantage is that you need to look at the code to understand the data. Some things change infrequently enough for DB native to be the better solution.

A nice "trick" is to declare your enum column on the database as a string, and your enum in the code as a hash with string values. So you can have self-explained data saved onto the database, and all the niceties Rails gives you from the enum.

  enum object_type: { review: "review", purchase: "purchase", offer: "offer", reward: "reward" }
  # OR
  enum action: TYPES.map { |type| [type.to_sym, type.to_s] }.to_h

Re: Things I wished more developers knew about databases

#387
post #327
post #261

Earlier quoted context omitted.

Just because you saw a few exception does not mean the rule does not hold in general. Or are you saying most people don't learn with time (a corollary of your theory that age is not a big factor)?

Most people don’t magically grow wiser with time. They need to be in an environment where they can grow, otherwise they’ll emerge just as stupid as before.

If you start out with a cohort of individuals across the spectrum of ability, over time many of those of lesser ability will self-select out of the pool. It is my experience (oops, see what I did there) that people rarely spend a career doing what they're not very good at. The exceptions inevitably stand out.

So, combining the winnowing of the not-very-apt with the gaining of knowledge through experience, the end result is that you have a preponderance of wise old experienced contributors.

I you are one of the younger inexperienced ones who believe they know better it's likely you'll self-select at some point, secure in your Dunning-Kreuger knowledge, and move to some career in which your high level of competence is valued more.

Re: Things I wished more developers knew about databases

#388
post #176

"The fastest way to access to a row in a database is by its primary key. If you have better ways to identify records, sequential IDs may make the most significant column in tables a meaningless value. Please pick a globally unique natural primary key (e.g. a username) where possible." Has anyone had a problem due to surrogate keys?

I can agree with everything in the article except this one. >Has anyone had a problem due to surrogate keys? There's one problem with surrogate keys: they are not convenient to users (too long and not meaningful). There are two problems with natural primary keys, and you are guaranteed to hit one of them at some point. 1. It turns out your key isn't actually unique. To resolve the collision you have to replace natura…

The article mentions auto-incrementing keys, not surrogates. Not the same thing at all. Not all incrementing keys are surrogates and not all surrogates are incrementing keys.

Also, your problem number 1 is a problem whether the natural key in question is the "primary" one or not. Certainly if you choose the wrong natural key then you'll have to fix that - that's why you should take care to make a wise choice of natural key regardless of whether you are also using a surrogate.

Re: Things I wished more developers knew about databases

#389

Earlier quoted context omitted.

How about the following: - When to use JOIN vs a subquery? - When is a subquery actually a correlated subquery? Will this destroy your performance? Or is it a critical feature? - Should you put constraints in the JOIN or in the WHERE? Will the distinction drastically affect performance? - When do you use WHERE vs HAVING? - Is the NULL from the join because no joined row was found, or because the joined row had a NULL…

I will be very glad if you actually answer these questions, in a separate comment. I'm driven to write this by nothing but the desire to know.

> - When to use JOIN vs a subquery?

Normally, subqueries return a single value whereas joins can result in n rows of output for 1 row being joined on, and you can access all the columns of those n rows. There are ways to make use of more than one value (e.g. (tuple) IN (subquery)) but if you want to SELECT more than one value, you need to join.

Depending on the database, it might be slower to do a correlated subquery than a join though (MySQL especially).

> - When is a subquery actually a correlated subquery? Will this destroy your performance? Or is it a critical feature?

A subquery is a correlated subquery when it references symbols from the outer query. That means it needs to be evaluated once per row, and can't be evaluated once at the start of query execution. It can destroy performance if it needs to be evaluated too often - if it's in your 'where' clause and is evaluated over too many rows, e.g. it's mixed in with a boolean expression that can't be short cut.

> - Should you put constraints in the JOIN or in the WHERE? Will the distinction drastically affect performance?

Conventionally, you should put equi-join constraints (equality expressions with foreign / primary keys in other tables) in the JOIN clause and other constraints in the WHERE clause. For inner joins, it doesn't make a difference where you put the predicate, semantically. There is a semantic difference for outer joins though (left join, right join, full outer join): failure to join results in a tuple worth of null values from one or both sides (left/right vs full), rather than eliminating the row.

Where the semantics aren't different, performance should not be affected. Of course the database engine might be stupid, but a fundamental requirement of a reasonable query planner is in determining (a) join order and (b) which indexes to use for the combination of join predicate and where predicate. No query planner worth its salt won't consider using the where clause along with the ON clause on the JOIN when fetching rows in the joined table.

> - When do you use WHERE vs HAVING?

WHERE is before GROUP BY and filters the rows that enter aggregation (if any), HAVING comes after GROUP BY and filters the aggregated rows. If you use a derived table (a nested query with a table alias), then you can use WHERE instead of HAVING for no semantic difference, but derived tables may execute differently (MySQL will generally materialize them, PostgreSQL will see through them).

> - Is the NULL from the join because no joined row was found, or because the joined row had a NULL value itself?

If the column is nullable, and you used an outer join, you can't tell. Normally you check for the primary key or some other non-nullable column to discover if a join failed (most often used in anti-join, when you want to find all rows that don't have corresponding rows in the join).

Re: Things I wished more developers knew about databases

#390
post #326
post #225

Earlier quoted context omitted.

Briefly, 1. Let me try with a simple example. Suppose you have a fact table A with fields (ItemID, Item, Amt) where Amt is in USD. Rule of thumb is: don't expose A to the consumer; instead write a SQL View V_A and expose that instead: CREATE VIEW V_A AS SELECT ItemID, Item, Amt FROM A Then suppose a European counterpart wants to use the same API but needs the amounts to be in Euros. You can write another view: (in pr…

There's another step that could be added there, too: After the ALTER VIEW, V could be slowly incrementally updated over however long you need to back-populate AmtGBP, and the views will continue to just work the whole time. Once done, V_A can be simplified to remove the ISNULL and Amt, then Amt dropped from V. That way you don't get build-up of cruft over the years, and the experience isn't interrupted for the migrat…

Is there anything you recommend for handling SQL definitions in version control, development and production envs?

For production, I created a command on the app that loads the stored procedures into the DB idempotently on each deployment/configuration. This won’t work if the app server scales but allowed us to store stored procs in VC.

For development, we ran the command on each page load as a sort of hacky “live reload”. It didn’t work well (which highlighted the issue with scalability in production) because Postgres, fairly, doesn’t like parallel redefinitions of the same stored proc.

I’m not sure how best to automate this. For production, seems like a case of running a command once per DB server.

And in development, using a fs watcher that loads changes in.

But I don’t know, this is new territory for us and I couldn’t find anything out there to manage it within the context of a web framework. Perhaps I’m searching for the wrong thing.

Post reply on HN