Live data from Hacker News

Things I wished more developers knew about databases

medium.com

311–320 of 464 posts

Re: Things I wished more developers knew about databases

#311
post #16

I never realized this before but many excellent developers struggle with SQL beyond simple SELECT statements. I have a colleague who is by all accounts a deeply technical person but one day he confessed to me that he didn't really grok SQL and that he'd rather work with a "real" procedural programming language to just store and retrieve data. Part of it may be due to the fact SQL isn't really a programming language b…

Does it mean that my years of job of analyst writing SQL queries might be a valuable treasure if I were to apply for some dev job?

Re: Things I wished more developers knew about databases

#312
post #98
post #69

Earlier quoted context omitted.

I find thinking about analytic queries like these as map/filter/reduce easier than thinking in terms of the SQL!

SQL and functional programming constructs are actually quite similar: SELECT -> map FROM -> stream JOIN -> flatMap WHERE -> filter GROUP BY -> collect ORDER BY -> sorted https://blog.jooq.org/2015/08/13/common-sql-clauses-and-thei...

I don't see how JOIN is like flatMap. It's a subset of a cross-product. It's more like do-notation:

  x 

Re: Things I wished more developers knew about databases

#313
post #276
post #149

Here's a fun bug I had a few years ago - Had a postgres database which was using pgbouncer for connection pooling. The most senior developer (24yo or so) we had on the project was using Go to connect to the database to write some simple reports, but each report took hours to run, and often had to sleep for 30+ minutes. So, after a while, pgbouncer would kill their connection, and their report would die. No other appl…

Go code often reinvents/reimplements a lot of things from scratch, reintroducing problems that have been addressed long ago in other systems. It's like this new trend, let's rewrite everything in Go to be cool. Financially makes little to no sense.

I.e. https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild...

Re: Things I wished more developers knew about databases

#314
post #146
post #132

Earlier quoted context omitted.

Like everything else, write it the simple/elegant way then profile it and tweak if you have to. Once you're at the point where you have to worry about these things, tuning the SQL is still probably much less complex than writing the query in your app language or figuring out how a NOSQL db can do these joins.

In reality it rarely works this way - there's plenty of systems which are falling apart due to "death of thousand cuts" type issues. You run a profiler and most of the queries are slow and there's no one obvious part to optimize - because developers over the years ignored basic optimizations and there are inefficiencies everywhere. E.g., for a quick practice run, try optimizing Wordpress without making it a static pa…

This is my experience especially with ORMs. It's really hard to optimize when your "top query time" list is just a list of your most frequently called methods, where there's no single obvious pathological case, but lots of little inefficiencies.

Re: Things I wished more developers knew about databases

#315
Many years ago I've read some chapters from Itzik Ben-Gan's "Inside Microsoft SQL Server 2008 T-SQL Querying." It's an excellent book to anyone who wants to know how things work under the hood. While the title is in fact about SQL Server there so many things explained out there that apply to almost any SQL engine that is a worth the purchase.

Re: Things I wished more developers knew about databases

#316
post #21

Earlier quoted context omitted.

I sometimes feel in the minority. I love databases. When I work in the Ruby on Rails ORM ActiveRecord I can actually visualize the SQL it is generating in my head and also do all sorts of tricks when needed.

That's the key. ORMs get a bad name but most of the time you just want to display a list of things, or one thing in more depth or maybe create a new thing. ORMs unfortunately, have a habit of getting in the way when you want to do something they don't natively support. When they just ignore things the database provides people just end up reinventing the wheel. Rails' implementation of enums is a good example of this.

> That's the key. ORMs get a bad name but most of the time you just want to display a list of things, or one thing in more depth or maybe create a new thing.

If you just need to display a list of things then there's nothing much simpler than:

  var result = exec_query("select * from things")
  foreach (var row in result) {
    //output html or something here
  }
The problem is we decided this was bad and had to add more layers, abstractions, translations and complications in between the database and the output, then we needed tools like ORMs to help with that.

Re: Things I wished more developers knew about databases

#317
post #96
post #20

Learn about modelling. Database is more than just storing data. Drink less koolaid of NoSQL, any NoSQL. It is trading initial result with future development time. SQL has been battlefield tested. No amount of "convenience" is more convenient than learning the fundamentals.

I'm disappointed that there isn't more criticism of the SQL language . The whole NoSQL buzz got me excited, then turned out to actually mean NoRelational. It is wild that we are still using a language that looks and feels like COBOL, and any criticism is met with drive-by disapproval (downvotes and no comments) or an argument about why relational databases are important. SQL is a deeply flawed language by standards t…

I agree. The SQL language is in desperate need of an "upgrade" to a proper functional language. It's missing so many basic features that it's just painful.

For example, why do I have to repeat expressions in the SELECT, GROUP BY and ORDER BY clauses!?

E.g:

    SELECT 
        LEFT(Foo,4) as Prefix,
        COUNT(1) as N
    FROM MyTable
    GROUP BY LEFT(Foo,4)
    ORDER BY LEFT(Foo,4)
This gets really obnoxious for complex expressions. I mean sure, I can break things out into functions, or use the WITH clause, but both of those often end up being more verbose, not less. This is solved in functional languages such as Haskell with the "where" or "let" clauses (not to be confused with the SQL WHERE filter).

I wish I could do something like:

    SELECT 
        Prefix,
        COUNT(1) as N
    FROM MyTable
    GROUP BY Prefix
    ORDER BY Prefix
    LET Prefix = LEFT(Foo,4)
Similarly, it's crazy to me that most SQL dialects don't allow relational functions to be passed around as proper data types. This kind of thing comes up a lot when implementing row-level security.

I'd like to be able to write "view functions" that can take any relation as an input, as long as it contains some column, and then add on some joins or filters based on that column. Imagine you have a bunch of tables that link to the "Customers" table via a "CustomerID" column or whatever. It would be great to be able to natively write a query that says "AddCustomerName" or "FilterDisabledCustomers". These would take an existing relation (table/view/function) as an input, and return a new relation as the output with an extra column or filtered rows.

This kind of modular style made up of small chunks of code that can be elegantly composed is just not possible or very messy in most SQL platforms. It's typical of most modern functional languages, and I think the "next gen" query language that replaces SQL will look more like Haskell and less like COBOL.

Re: Things I wished more developers knew about databases

#318
post #292

Earlier quoted context omitted.

Every good rule has an exception.

Why not judge people by the result of their work, instead of their age?

Because then it’s harder to deal with our own imposter syndromes if we can’t blame it on the youth and hold their heads in the toilet while giving them the professional-development equivalent of a wedgie.

This was discussed at length in last week’s “Grey Beard Weekly” newsletter.

Re: Things I wished more developers knew about databases

#320
post #149

Here's a fun bug I had a few years ago - Had a postgres database which was using pgbouncer for connection pooling. The most senior developer (24yo or so) we had on the project was using Go to connect to the database to write some simple reports, but each report took hours to run, and often had to sleep for 30+ minutes. So, after a while, pgbouncer would kill their connection, and their report would die. No other appl…

Nodejs Sequelize is transparently doing this when ping connection. That aside, I wonder why you need to keep the connection alive for > 30 mins, while usually sql con is short lived. Why can't you just close and reopen them, is it temporary table?

Not the GP, but I assume transactional consistency was important for the report, hence the need to keep the connection alive. That's a pretty common situation.
Post reply on HN