Live data from Hacker News

Things I wished more developers knew about databases

medium.com

161–170 of 464 posts

Re: Things I wished more developers knew about databases

#161
post #16

I never realized this before but many excellent developers struggle with SQL beyond simple SELECT statements. I have a colleague who is by all accounts a deeply technical person but one day he confessed to me that he didn't really grok SQL and that he'd rather work with a "real" procedural programming language to just store and retrieve data. Part of it may be due to the fact SQL isn't really a programming language b…

I'm pretty sure I would be good at SQL if I devoted time to it, because I've worked in other declarative DSLs and generally outperformed other programmers using them, but basically every place I've ever worked at that used SQL extensively had already employed SQL experts that just handled the problems on that level so that it didn't seem worthwhile bothering with it.

Re: Things I wished more developers knew about databases

#162
post #136

Earlier quoted context omitted.

How about the following: - When to use JOIN vs a subquery? - When is a subquery actually a correlated subquery? Will this destroy your performance? Or is it a critical feature? - Should you put constraints in the JOIN or in the WHERE? Will the distinction drastically affect performance? - When do you use WHERE vs HAVING? - Is the NULL from the join because no joined row was found, or because the joined row had a NULL…

> The basic concepts are simple, but the implementation details quickly become very complex True, but those kinds of questions come up in every language: Should I use an array or a dictionary? Should people have references to projects or should projects have references to people or both? Is money a float, an int, a decimal or should I write my own money class? Should I memoize the results? Is it thread-safe? As you c…

They do. And language designers seek to smooth the edges and develop ways to encourage devs to write clear and intuitive code. When a language says "well, these are hairy questions that you should just figure out" we tend to criticize those languages unless they have clear reasons for that decision. It should be obvious that something is thread-compatible. "Well, if you model it in pi-calculus it is easy" is a crappy way of handling criticism.

"Its easy if you think about it mathematically" is not enough.

Re: Things I wished more developers knew about databases

#163

Earlier quoted context omitted.

I firmly believe that every developer should spend 2-3 weeks early in their career working with nothing but SQL. It will pay huge dividends for the rest of it. IMO a lot of the issue is that developers for many years using Java or PHP, were using SQL to handle everything. The application language was a pass through later between the client and the database. Your goal was to accomplish as much as possible in a single…

> The application language was a pass through later between the client and the database. This style of doing things resulted in spaghetti style unmanageable databases, filled with an unknowable number of triggers and procedures, all written in PL/SQL (which is much, much worse than either Java or PHP). The reason why ORMs started to become popular is that you can write your application without filling your DB with ar…

But now you're filling your application with arcane and inscrutable logic, with an extra layer of abstraction via the ORM to make it even less scrutable.

I think one should view a SQL DB like a microservice. Instead of REST endpoints (or gRPC or whatever), create stored procedures. These define a strong contract with your DB, the capabilities that it provides to your app(s). Now you know what the query and insert patterns are, and can tweak the table layout under the covers without screwing up your application code.

Of course you can abuse this into a spaghetti monolith, just like you can evolve a microservice into a spaghetti monolith, but you shouldn't. There's no technology that will prevent you from making poor architectural decisions, you just have to not go down those dark paths.

Re: Things I wished more developers knew about databases

#164
post #96

Earlier quoted context omitted.

I'm disappointed that there isn't more criticism of the SQL language . The whole NoSQL buzz got me excited, then turned out to actually mean NoRelational. It is wild that we are still using a language that looks and feels like COBOL, and any criticism is met with drive-by disapproval (downvotes and no comments) or an argument about why relational databases are important. SQL is a deeply flawed language by standards t…

In regards to 'select' before 'from', there could be a similar argument to be made for declaring imports at the top of a file. When you write a program, you may not know what libraries you need. Variable declaration at the beginning of a function is also a common pattern. The SQL language stood the test of time where as COBOL did not. I think it says something about how well it was designed. I strongly suspect that d…

>Variable declaration at the beginning of a function is also a common pattern

What language requires you to declare variables at the beginning? C even stopped doing that. People choose to do that, but ime that's after the writing phase, to make it more readable

>When you write a program, you may not know what libraries you need

Sort of -- the difference is that when I'm writing code in my IDE, I know all database objects available to me. It's in the schema. A library I import once (to the project itself), and the IDE can always assist from then-on. But SQL is designed such that despite the library (schema) being imported, the IDE can't actually assist, unless I write the code out of order (eg start with SELECT * FROM table and then start working)

>However, once it clicks, it's fits naturally with how a relational db works

I think you've misunderstood the complaint -- the relational language is a very strong concept, which has stood the test of time, and is difficult to complain about -- it does its job well, and fits naturally with how a relational db works.

SQL the language however is:

A hodgepodge of random keywords tossed about in a totally inconsistent fashion (eg postgres overlay: OVERLAY('Txxxxas' PLACING 'hom' FROM 2 FOR 4) -- postgres at least generally offers a consistent comma-separated syntax for every functions

has weird and technically unnecessary limitations (like SELECT being evaluated after the WHERE clause, so you can't use the aliases defined in select clause)

putting SELECT before the FROM (disabling IDE auto-complete support)

the stuffing of a 3-value logic system into a 2-value logic interface, so boolean operations break silently and produce nonsense in the face of a database with NULL values, because no mapping of NOT (TRUE, FALSE, NULL) AND NULL makes sense. [0]

It generally composes very poorly, leading to redundant, long and convoluted queries

The MODEL is fantastic -- the language is not.

[0] https://sigmodrecord.org/publications/sigmodRecord/1709/pdfs...

Re: Things I wished more developers knew about databases

#165
post #149

Here's a fun bug I had a few years ago - Had a postgres database which was using pgbouncer for connection pooling. The most senior developer (24yo or so) we had on the project was using Go to connect to the database to write some simple reports, but each report took hours to run, and often had to sleep for 30+ minutes. So, after a while, pgbouncer would kill their connection, and their report would die. No other appl…

was the go library fixed?

Re: Things I wished more developers knew about databases

#166

Earlier quoted context omitted.

how would you generate hole-free sequences for use cases like that of parents (invoicing)?

If it was a business requirement that you have perfectly sequential invoice numbers with no gaps, do it at the application level, not at the storage level. Let the database do what it's great at doing: efficiently store and retrieve data.

I don't think it is not possible to enforce perfectly sequential numbers at the application level.

This is the same problem as multiple threads trying to increment the same global variable. Unless there is mutual exclusion while the variable is being read/incremented there will be race conditions.

I think the only way to enforce mutual exclusion for applications would be at the database layer (or any other layer where there is 1 resource that is shared between each application peer).

Re: Things I wished more developers knew about databases

#167
post #103

Earlier quoted context omitted.

There is an entire world between ORM and PL/SQL. Programmatically constructing SQL statements is also a thing. Just because someone writes SQL does not mean SQL needs to be spread through out code or that we need to have lot of logic in PL/SQL. Of course, there will be cases where a store procedure is desired (any kind of validation that cannot be expressed as fkeys, canonicalization of some core data components etc)…

How did you minimize roundtrips between server and DB, or did you find that they were not a big concern? I'm working on a project with a Postgres database, and as it gets more complex I'm moving more stuff into stored procedures, pretty much wherever a single action requires multiple statements in series (e.g. check if this thing exists, check a value, get the id of some other thing, on success update another table).…

I think they're saying that you profile your changes on an individual query.

Yeah, if all of your queries are slow, you've got more work to do, but that doesn't change the overall process of optimization.

Re: Things I wished more developers knew about databases

#168
post #47

Earlier quoted context omitted.

Part of the issue is that a complicated database can handle the same SQL query many different ways based on indexes and other configurations. This kind of "magic" isn't always clear when programmers are mostly used to working with data structures and procedural code. The other problem, IMO, is that programming languages are very poor at bridging the difference between the SQL domain and the language domain. We really…

>programming languages are very poor at bridging the difference between the SQL domain and the language domain Depends a lot on the language. I've lost countless hours to things like JOOQ trying to figure out how to get it to do what I want, or express the query in its quirky not-quite-right DSL, plus dealing with mappings, pojos, auto-generation, and so on. However, on the other hand, in a dynamic language with just…

jOOQ can be used in a less-type-safe way. For example, `fetchMaps` [1] does more-or-less what you describe.

However, I have found it worthwhile to learn to use the more advanced features you mention. Extending type safety to queries is incredibly useful. Consider cases when developers are making code and schema changes concurrently that overlap.

[1] https://www.jooq.org/javadoc/latest/org.jooq/org/jooq/Result...

Re: Things I wished more developers knew about databases

#169
post #144

Earlier quoted context omitted.

SQL is very much like CSS to me. It's declarative, the primitives seem entirely non-intuitive, it often takes a lot of fiddling to get what you want, the behind-the-scenes execution is mostly a black box, and while it's supposed to work the same on different implementations (of browsers/databases), there are tons of little gotcha quirks. All in all, they're both entirely different skill sets from traditional programm…

This is... one of the best comparisons I've seen and sums up the reason why I dislike SQL as well (although I know how to use it). Sometimes it really feels like you're trying to give instructions to someone via chat which gets Google translated to chinese, japanese and russian on the way - it's this very lossy communication channel where you need to tweak the language "just so" to get maximum performance. I think it…

Databases are quite transparent about what they do. The impression of "just so"-ness is a property of the observer, not the system.

Re: Things I wished more developers knew about databases

#170
post #89

Earlier quoted context omitted.

When you look at SQL from a logical/set-based perspective, it is by no means unintuitive. Basically, all you do is join all the tables you need and then filter out everything you don't need and maybe do an aggregation here and there.

How about the following: - When to use JOIN vs a subquery? - When is a subquery actually a correlated subquery? Will this destroy your performance? Or is it a critical feature? - Should you put constraints in the JOIN or in the WHERE? Will the distinction drastically affect performance? - When do you use WHERE vs HAVING? - Is the NULL from the join because no joined row was found, or because the joined row had a NULL…

You need to spend time working with the language and understanding how the set based operations used in SQL work. Declarative languages means you can express things in multiple ways and get the same results. This is an incredibly important aspect of SQL

>the basic concepts are simple, but the implementation details quickly become very complex

This is no different than programming. Assuming that SQL doesn't have complexity because you can only SELECT, INSERT, UPDATE or DELETE is going to have you banging your head against the wall. Tackle the complexity in SQL like you'd tackle the complexity in your programming language of choice; read the docs, work through examples, and read how other people solve the problem. There's ton out there for SQL

>When do you use WHERE vs HAVING?

HAVINGs allow you to add a condition to an aggregate function. So SUM(myColumn) > 5 would be something you put in a HAVING clause. Honestly, this is pretty clear cut.

>Should you put constraints in the JOIN or in the WHERE? Will the distinction drastically affect performance?

The first thing to understand is a condition in a join versus a where might return a different result set, specifically on anything other than an INNER join. The impact to performance will depend on the rest of your query, your data, and your index coverage. For simple cases, there is likely no difference. For complex ones, there may be an impact

> Is the NULL from the join because no joined row was found, or because the joined row had a NULL value itself?

An inner join shouldn't produce a null. That's why it's an inner join, as the data needs to exist in both places. If you want to "test" whether a join found a row, look at the field you were joining to and see if it's a non-null value. Nulls won't join to Nulls unless you've change some settings in most RDBMS. If you're looking at other fields to determine the presence of a row from a join, make sure you're looking at a non-nullable field.

>When to use JOIN vs a subquery?

A better way to phrase this would be when to just join the table, vs writing a sub query and joining to that. When is a question of the complexity of the query and performance characteristics, and that can't be answered in the abstract. The most important thing is that in a large number of cases you can do both, and knowing how to express things in both ways is powerful.

>And all of the questions I pose above have clear answers

No they don't. Any time you're wondering about how different SQL impacts performance, there's absolutely a huge "it depends" angle on it, because how you've structured the tables, index coverage, and the volume of data, can have a significant impact. This is why DBAs still have jobs, because the database is an incredibly complex system. You seem to be complaining that SQL shouldn't be complex, yet are not willing to accept that it is more complex that you've assumed it to be. It's complex. You don't need to know everything if your just a dev, but don't just assume it's simple.

>while JOIN seems like it ought to be intuitive

I'd check the diagram here -> https://stackoverflow.com/questions/13997365/sql-joins-as-ve... Half those joins aren't needed as you can re-order a right join into a left join. For 95% of development Inner joins and left joins are all you need. The other 5% is an outer join and that's mainly needed in report writing, not app development.

Post reply on HN