Live data from Hacker News

Things I wished more developers knew about databases

medium.com

441–450 of 464 posts

Re: Things I wished more developers knew about databases

#441
post #437

Earlier quoted context omitted.

Web frameworks like Rails/Django use the idea of migrations to make changes to the database. The idea is that you have a set of migration scripts like: migrations/1765_create_table_users.sql migrations/2891_store_procedure_x.sql migrations/5892_change_store_procedure_x.sql (.sql/.rb/.py, it doesn't matter). And you have a "migrations" table in your database that contains the numbers of the migrations that have been r…

Yeah, I’m aware of that, thank you. I was wondering if there was a way with a faster feedback loop and allowed for bug fixes without creating a new migration.

You don't need to write the migration until you're done. It's possible to have a very tight feedback loop in any case.

I'm doing a lot of work in a Rails codebase where I edit views/functions/procedures all the time. My setup is quite usable.

My current setup: I edit those .sql files and run them with psql in my local while developing (without writing any migration yet).

I have some like this running on one screen to make sure the modified files are executed by psql immediately as I change them (you could use `guard` too):

  find ~/projectx/db/functions -type f -name "*.sql" | entr -d -p psql db_name -f /_
and I edit the db/functions/*.sql files freely, adding things, changing behaviour of functions and they are updated on the fly. (I can run tests -or try things in the browser- to verify my changes work as I expect).

--

Once I finish and I know everything is great, I just add the migration. The migration is simply an indicator of which files I've modified and to specify the right order to run them (which is useful if they are dependencies), like:

  # migration
  def up
    execute File.read(function1_sql_file)
    execute File.read(function2_sql_file)
  end
I could have an alias that automates generating that migration but it's just 4 lines...

[ I'm also using pgTAP to write tests for functions, it's quite nice :) ]

Re: Things I wished more developers knew about databases

#442
post #63

Earlier quoted context omitted.

As a developer: I do care, but it's hard for me to focus on building software if I also need to think about DBA tasks, DevOps tasks, and so on. These things all take time, patience, and energy. I've just spent most of today running and re-running a CloudFormation template to create a SQL database. Most. Of. A. Day. It's partly because I'm not a DevOps expert and even if I wanted to be one, that would also take time,…

> It's partly because I'm not a DevOps expert and even if I wanted to be one, that would also take time, patience and energy. It's this exact divergence that creates the disconnect. If someone doesn't understand and doesn't have to care about the whole experience, they're going to focus on their side and stop when their side is good enough. On the other hand, if that same someone is going to be regularly developing t…

> Nitpicking over "not my specialization" is the antithesis of a smooth engineering process.

Counter argument: jack of all trades, master of none.

I'm already a full stack developer. I work on a complex front end application, a GraphQL server, a .net core platform split into multiple microservices, and a MSSQL database. I know my way around these components fairly well now but it's taken a good couple of years to get to this point.

I could also invest a bunch of my time learning all the intricacies of cloudformation templates and how IAM roles work too, sure. But is it the best use of my time as a developer, when I'm much more productive writing code?

You can just end up stretched too thin.

Re: Things I wished more developers knew about databases

#443
post #401

Earlier quoted context omitted.

Because we got rid of DBAs in favor of “big data” developers that never learned much about SQL in the first place.

Because 64gb of ram is really cheap these days. It no longer makes sense to tune your queries, or to wait weeks\months for the vendor to tune their queries, when you can just slap a few sticks in and call it a day.

But we aren't doing something simple like that, we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes. It takes hours to get a development environment put together to try and reproduce / debug a problem. We are adding complexity instead of keeping things simple.

And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

Re: Things I wished more developers knew about databases

#444
post #49

Earlier quoted context omitted.

If you're writing a CRUD application, an ORM saves a lot of headaches. If you're doing complex reporting queries, an ORM is strictly worse. And yes, I've seen developers, architects, and authors of ORMs that believed otherwise. They are wrong. As an example, very, very few ORMs can make the distinction between SELECT ... FROM foo LEFT JOIN bar ON foo.id = bar.foo_id AND bar.category_id = 5 LEFT JOIN baz ON bar.id = b…

Curious, what is the difference here?

In the first case, if for a given row bar.category_id 5 then there's no match and hence null will be returned for any value from bar. Thus there will be no match in baz either (assuming the id columns are non-null).

In the second case, the first join will always return a row from bar for a given foo_id if it exists, but the second join does not return values from baz for rows where bar.category_id 5.

Re: Things I wished more developers knew about databases

#445
post #231

Earlier quoted context omitted.

I find SQL INSERT statement not intuitive. I can understand why SQL requires me to declare the field names and then the values of a new row that I am inserting; but it would've been a huge time saver if SQL had a key-value dictionary-like syntax: INSERT INTO "my_table" "col1": value1, "col2": value2, ...

And when you need to insert more than one row, you repeat the column name over and over again for each row - not a good idea either.

[deleted]

Re: Things I wished more developers knew about databases

#446

Earlier quoted context omitted.

And when you need to insert more than one row, you repeat the column name over and over again for each row - not a good idea either.

Single-row inserts are super common in both application code and in interactive usage of SQL, so I think that it is worth it to have a syntax for them that reduces this common error. Especially when a table has many columns of the same type (like booleans). E.g. insert into Permissions(UserId, Create, Read, Update, Delete, Share, ForceUnlock, LaunchNukes) values (12345, 1, 1, 0, 1, 0, 0, 1); When I wrote my (now unma…

The DB we use at work (SQLAnywhere) supports inserting from select, and using auto column name matching, so for a table with columns (name, en1, en2) you could have

    INSERT INTO tbl WITH AUTO NAME
    SELECT 'foo' AS name, 1 AS en1, 0 AS en2;
Not nearly as neat as the direct key=value syntax but...

Re: Things I wished more developers knew about databases

#447
post #443
post #401

Earlier quoted context omitted.

Because 64gb of ram is really cheap these days. It no longer makes sense to tune your queries, or to wait weeks\months for the vendor to tune their queries, when you can just slap a few sticks in and call it a day.

But we aren't doing something simple like that, we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes. It takes hours to get a development environment put together to try and reproduce / debug a problem. We are adding complexity instead of keeping things simple. And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out…

> we are building monstrosities based around the theory of micro-services in the cloud. Kubernetes

This isn't something I'm doing and none of the people I personally know are doing this (disclaimer: im not in SV or the "startup scene")

> And query tuning isn't that difficult. Spend a few hours on this site and you will be better than 90% of devs out there: https://use-the-index-luke.com/

It used to be that companies had DBAs and could call out their developers\vendors on their sloppy queries. They have been replaced by 64gb of ram

Re: Things I wished more developers knew about databases

#448
post #441

Earlier quoted context omitted.

Yeah, I’m aware of that, thank you. I was wondering if there was a way with a faster feedback loop and allowed for bug fixes without creating a new migration.

You don't need to write the migration until you're done. It's possible to have a very tight feedback loop in any case. I'm doing a lot of work in a Rails codebase where I edit views/functions/procedures all the time. My setup is quite usable. My current setup: I edit those .sql files and run them with psql in my local while developing (without writing any migration yet). I have some like this running on one screen to…

Oh wow, now I see what you mean. Thank you! That’s work great. I wasn’t aware of ‘entr’ either, that’s exactly what I had in mind!

I’ll have a look at pgTAP too. Naturally we want to test in CI, I can see this working really well. I did look at myTAP too, since we have a few MySQL instances.

Re: Things I wished more developers knew about databases

#449
post #384
post #329

Earlier quoted context omitted.

Another option is to learn from the mistakes of others.

People rarely post their mistakes to stack overflow.

They surely write papers about it.

I bet anyone at Google is able to access ACM, SIGPLAN, HOPL, IEEE, USENIX papers.

Re: Things I wished more developers knew about databases

#450
post #7

(The 80/20 rule applies below, some developers do care) Developers... just don't care. They want to spin up an ORM, point it at a URI, and forget about it. I've fought this for over a decade now as a DBA, SRE, DevOps, and architect. Most of the developers don't want to deal with anything infrastructure-wise; they want to spend all the time they can just focusing on the problem they're writing software to solve. Obser…

Depends on the organization.

Thankfully in most places I have worked so far you wouldn't go very far being an 'X Developer'.

Everyone has always been required to be an expert in at least two domains, those T and Pi shaped concepts.

So I have very seldom met those kind of devs that only want to spin up an ORM.

Post reply on HN