Things I wished more developers knew about databases • Hacker News Board

Re: Things I wished more developers knew about databases

#201

Post by vsareto » Wed, Apr 22, 2020, 6:44 PM UTC

AndrewKemendo wrote: post #179

vsareto wrote: post #12

Earlier quoted context omitted.

Many interests are pulling developers' attention in several different areas all the time. Database, security, accessibility, performance, infrastructure, tooling and productivity, business concerns, workflow processes (agile), language concerns, new things All of these like to say "if only the developer could do $MY_AREA better, they'd be better developers and we'd have better software". Each of them wants to pile on…

10, 11 year timeline to what? You should be constantly learning everything about the whole stack so that you can actually build functional, reliable, manageable and maintainable systems. I expect a competent developer to be able to build a modern multi-page web application, with a HTML/JS front end, relational database back end, appropriately configured certificates and DNS/CNAME/URL, build basic uptime and applicati…

> 10, 11 year timeline to what?

To be decent at each of those areas I listed.

Why those 10 specific things?

Those are all areas I've seen "Developers should learn X" calls-to-action.

Sometimes it's outright loathing over developers that don't know specific topics. Other times it's wishful thinking, or maybe a nice-to-have.

If you, as a developer, read all of those posts, they probably all make fair points about the importance of knowing each of those things as they relate to development. You decide there is some merit to learning it.

So you decide to make a to-do list to go learn each of those topics, because you want to listen to the blog posts and be a good developer, and do some real work to prove you know it. That's what's going to take you a while.

Re: Things I wished more developers knew about databases

#202

Post by gedy » Wed, Apr 22, 2020, 6:46 PM UTC

falcolas wrote: post #64

arvinsim wrote: post #10

Earlier quoted context omitted.

Perhaps development work has reached complexity where it demands full-time attention? Perhaps there is a reason why DBA, SRE, Devops and Architects are separate roles?

A DBA can't fix a developer's code. A good 3/4 of the list linked in TFA are things a DBA or Architect can't affect. At some point, a developer needs to be responsible for how they use a tool that's been provided for them. Abstractions break.

Well likewise, many of the DBAs I've worked with don't try and understand the software needs, and want the world to revolve around what is optimal for them.

Re: Things I wished more developers knew about databases

#203

Post by jatone » Wed, Apr 22, 2020, 6:47 PM UTC

StandardFuture wrote: post #192

AndrewKemendo wrote: post #179

Earlier quoted context omitted.

10, 11 year timeline to what? You should be constantly learning everything about the whole stack so that you can actually build functional, reliable, manageable and maintainable systems. I expect a competent developer to be able to build a modern multi-page web application, with a HTML/JS front end, relational database back end, appropriately configured certificates and DNS/CNAME/URL, build basic uptime and applicati…

You seem to be contradicting yourself without realizing it. > You should be constantly learning everything This is an in-progress action i.e. the developer is still learning. > I expect a competent developer to be able to build This is now considering a "learned" developer. I am not sure you are making the point you think you are making. The point I think you are trying to make is your expectations of what an experie…

hes not contradicting himself, the fact is there are a large percentage of developers who just don't care. its not a matter of still learning. they just don't care to learn.

Re: Things I wished more developers knew about databases

#204

Post by msla » Wed, Apr 22, 2020, 6:52 PM UTC

cookiecaper wrote: post #174

arvinsim wrote: post #10

Earlier quoted context omitted.

Perhaps development work has reached complexity where it demands full-time attention? Perhaps there is a reason why DBA, SRE, Devops and Architects are separate roles?

> Perhaps there is a reason why DBA, SRE, Devops and Architects are separate roles? It's mainly an artifact of the way we've broken up degree tracks, and the boundaries that each group is taught to stay inside, lest any particular group actually ends up culpable for a failure. Make fun of the "full-stack rockstar ninja" all you want, but the reality is that it is possible to have a functional understanding of all of…

The phrases "functional understanding" and "working knowledge" are gigantic sucking tarpits.

Before I got my current job, I felt confident in my knowledge of computer networking at the LAN level. I knew I wasn't going into the telecom world and I knew I didn't have the knowledge to debug BGP or ensure a CO was doing everything right, but LANs? Sure. No problem. I knew DHCP, Ethernet, TCP/IP, even stuff like PPP which is more niche now. Heck, I'd even passed a college course on the subject. OK... set up an ICMP server and make it useful. That's LAN, right? Certainly gonna be used on a LAN.

I'm not saying it was hard. I'm saying that I'd never touched ICMP before except for ping and didn't know what the more advanced stuff even was. Did I have a "working knowledge" of basic networking? Did I have a "functional understanding" of how to get a building full of computers to talk to each other?

It's always something. I thought I had a good, working knowledge of networking. Someone who'd actually done networking in a corporate setting would have disagreed, and pointed to a list of things I'd never touched because those things aren't useful in SoHo LANs and aren't theoretical enough for a classroom. Multiply that by a few dozen topic headings and watch people sink under the load.

Re: Things I wished more developers knew about databases

#205

Post by StandardFuture » Wed, Apr 22, 2020, 6:53 PM UTC

jatone wrote: post #203

StandardFuture wrote: post #192

Earlier quoted context omitted.

You seem to be contradicting yourself without realizing it. > You should be constantly learning everything This is an in-progress action i.e. the developer is still learning. > I expect a competent developer to be able to build This is now considering a "learned" developer. I am not sure you are making the point you think you are making. The point I think you are trying to make is your expectations of what an experie…

hes not contradicting himself, the fact is there are a large percentage of developers who just don't care. its not a matter of still learning. they just don't care to learn.

To be clear, I was not trying to say his points were void of validity. I was trying to append some clarity to properly differentiating between a developers general knowledge base, a developers temporary knowledge base, and the actual day-to-day doings of the developer. They are not equivalent sets of things even if they intersect.

As a personal anecdote, I know I have learned many things on a project that really helped improve the code, that today I am not able to recall and would have to go back and re-learn with a minimum of a refresher. This happens a lot too. :)

Re: Things I wished more developers knew about databases

#206

Post by randomdata » Wed, Apr 22, 2020, 6:54 PM UTC

harikb wrote: post #200

chaps wrote: post #149

Here's a fun bug I had a few years ago - Had a postgres database which was using pgbouncer for connection pooling. The most senior developer (24yo or so) we had on the project was using Go to connect to the database to write some simple reports, but each report took hours to run, and often had to sleep for 30+ minutes. So, after a while, pgbouncer would kill their connection, and their report would die. No other appl…

Go sql/database uses its own connection pool. But still that shouldn't create any problems. I have seen the reverse where apps that assumed temporary tables stick around from statement to statement without an explict `txn` (which regular postgres connections don't need) clearly failed. But I have not seen the issue you talk about. My wild guess would be that the Go code never closed the result/rows which caused eithe…

The database/sql package isn’t very magical. The comment about magical library makes me think it was some other package.

Re: Things I wished more developers knew about databases

#207

Post by jl6 » Wed, Apr 22, 2020, 6:57 PM UTC

falcolas wrote: post #7

(The 80/20 rule applies below, some developers do care) Developers... just don't care. They want to spin up an ORM, point it at a URI, and forget about it. I've fought this for over a decade now as a DBA, SRE, DevOps, and architect. Most of the developers don't want to deal with anything infrastructure-wise; they want to spend all the time they can just focusing on the problem they're writing software to solve. Obser…

Application developers don’t care about data - they want the database to be a dumb storage unit because they only care about their application functionality.

If you can convince your application developers of the prime importance of data, they will start caring about their database.

Applications come and go but data lives forever.

Re: Things I wished more developers knew about databases

#208

Post by wwweston » Wed, Apr 22, 2020, 6:59 PM UTC

bdamm wrote: post #34

mgkimsal wrote: post #25

Earlier quoted context omitted.

> 2 years working on a non-trivial backend should expose one to these problems. You can be exposed to them, but without understanding them, and experiencing both good, bad, and really bad 'solutions' to them, and understand the impact (on the business, on the code, on security, on maintainability, etc)... you just can't really get all that in 2 years. I know plenty of people who've been 'exposed' to certain type of t…

I wish I could slap anyone who gives a hoot about tabs vs spaces. Fortunately modern languages like go are removing the version control problem that not caring about style and using auto-formatting IDEs produces.

> using auto-formatting IDEs produces.

I strongly suspect a lot of the remainder of the tabs vs spaces reminder is actually about how it either constrains editor / IDE choices OR requires and investment of time to deal with whichever choice someone else made.

"Why do you care, my IDE just handles this" is pretty close to saying "use my IDE," on top of "use my convention."

Re: Things I wished more developers knew about databases

#209

Post by wenc » Wed, Apr 22, 2020, 7:05 PM UTC

breischl wrote: post #163

TremendousJudge wrote: post #82

Earlier quoted context omitted.

> The application language was a pass through later between the client and the database. This style of doing things resulted in spaghetti style unmanageable databases, filled with an unknowable number of triggers and procedures, all written in PL/SQL (which is much, much worse than either Java or PHP). The reason why ORMs started to become popular is that you can write your application without filling your DB with ar…

But now you're filling your application with arcane and inscrutable logic, with an extra layer of abstraction via the ORM to make it even less scrutable. I think one should view a SQL DB like a microservice. Instead of REST endpoints (or gRPC or whatever), create stored procedures. These define a strong contract with your DB, the capabilities that it provides to your app(s). Now you know what the query and insert pat…

> Instead of REST endpoints (or gRPC or whatever), create

> stored procedures. These define a strong contract with your DB

Exactly. It took me years to grasp this, but when I did my code became much simpler.

A REST API (that returns JSON) has to contort a tabular data structure into a loosely-typed hierarchical data structure (JSON) which has to be read back, reconstructed and in many instances type-checked (e.g. DATETIMEs are not native to JSON, nuanced datatypes like DECIMAL(18,3) are lost).

Whereas a SQL interface returns data in its native tabular format with all the correct types.

Re: Things I wished more developers knew about databases

#210

Post by jatone » Wed, Apr 22, 2020, 7:08 PM UTC

izacus wrote: post #146

jayd16 wrote: post #132

Earlier quoted context omitted.

Like everything else, write it the simple/elegant way then profile it and tweak if you have to. Once you're at the point where you have to worry about these things, tuning the SQL is still probably much less complex than writing the query in your app language or figuring out how a NOSQL db can do these joins.

In reality it rarely works this way - there's plenty of systems which are falling apart due to "death of thousand cuts" type issues. You run a profiler and most of the queries are slow and there's no one obvious part to optimize - because developers over the years ignored basic optimizations and there are inefficiencies everywhere. E.g., for a quick practice run, try optimizing Wordpress without making it a static pa…

in reality it does work that way.

the problem you describe is completely different and its more a measure of system health in its entirety of the DB. it can be easier to fix or harder depending on the precise cause. but that's a standard debugging skill which isn't strictly in the DBA skillset.