The Database Administrator is dead • Hacker News Board

Re: The Database Administrator is dead

#21

Post by jbooth » Fri, Dec 13, 2013, 2:19 PM UTC

ghshephard wrote: post #15

mattmanser wrote: post #7

Earlier quoted context omitted.

Most people don't need dedicated DBAs and worse the DBA often never has sufficient domain knowledge of the problem to make an intelligent suggestion anyway. DBs performance is complicated, yes, but the vast majority of it is extremely simple. It's just none of this simple stuff has to be learnt until it's too late and the cost of fixing it has dramatically increased. There's a certain level where you need a DBA and t…

"Most people don't need dedicated DBAs" That's a pretty easy statement to defend. But, I'll respond by saying that most Companies running Oracle 11g with more than a couple terabytes of databases, require a competent DBA, particularly if Disaster Recovery/Transaction Rollback is important. " the DBA often never has sufficient domain knowledge of the problem " Of the half dozen or so truly high level DBAs I've worked…

Why would someone run Oracle in this day and age? "We have 100k lines of established PL/SQL running our business" is a reason, of course, but you could make a similar argument that COBOL development is alive. PL/SQL pretty much always leads to an unmaintainable mess, and if you need BIG DATA, you can go much bigger without Oracle than you can with it, and cheaper to boot.

Re: The Database Administrator is dead

#22

Post by falcolas » Fri, Dec 13, 2013, 2:21 PM UTC

mattmanser wrote: post #7

static_typed wrote: post #2

The DBA is not dead! They just look and smell that way! But in all seriousness, if your app uses a database, you are incompetent not to employ an expert to help with the database, whether advising on the query plan of those non-performant queries, or what is the best setup for the current stage of the business and app, they are very useful.

Most people don't need dedicated DBAs and worse the DBA often never has sufficient domain knowledge of the problem to make an intelligent suggestion anyway. DBs performance is complicated, yes, but the vast majority of it is extremely simple. It's just none of this simple stuff has to be learnt until it's too late and the cost of fixing it has dramatically increased. There's a certain level where you need a DBA and t…

> What we really need is to demystify DB performance, which for the most part is fairly simple.

Without getting into the rest of your argument, I'd like to quickly address this.

No, it really isn't.

In your typical MySQL database, your performance for a simple "select * from x where y" is going to go through a lot of complicated machinery (most of which can be tuned for performance), a few points of which I will enumerate below.

    1) Acquire a query cache lock & see if this query is there
    2) Run the query through the optimizer
      2a) Perform multiple shallow dives into a table to look at the cardinality of the filtered columns
      2b) Identify the best indexes based on the shallow dives
      2c) Create a query plan
    3) Push the query plan down into InnoDB
    4) Load the index into memory, if its not already there
    5) Load the potential rows into memory, if they are not already there
      5a) If there's not enough memory, load a few into memory, and be ready to push those rows out of memory in favor of more rows when needed
      5b) Load rows that are still in the insert tree but not yet part of the regular buffer pool or pages on disk
    6) Loop through the candidate rows for matches to the filter
    7) Return the data to MySQL
    8) Acquire the query cache lock & update it
    9) Return the data to the client

Any and all of these can (and often should) be tuned. There are 600+ page books and very old (and oft updated) blogs dedicated to this topic... it's not something you can teach a developer in a couple of days.

As an example, I attended an introductory course to being a MySQL DBA; it lasted 5 days of 8-5 teaching & running examples. And it only scratched the surface of what I do on a daily basis.

Re: The Database Administrator is dead

#23

Post by antoko » Fri, Dec 13, 2013, 2:23 PM UTC

falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Apparently you didn't make it to the conclusion.

  So perhaps the role of the DBA isn’t necessarily dead, it’s just moved
  to its new home at the datastore-as-a-service provider. The successful
  DBA will understand that this new world means handling petabytes of
  data and billions of operations on thousands of logical databases.
  They will cope with less mature database technologies in increasingly
  difficult workload environments. They will automate or die.

  Long live the DBA.

The whole post was making the exact same point as you in the end, the author actually runs a datastore-as-a-service business.

Re: The Database Administrator is dead

#24

Post by sourc3 » Fri, Dec 13, 2013, 2:24 PM UTC

Persistence ignorance introduced by mainstream ORMs about 5-7 years ago have turned the new generation of programmers blind to the fact database IS part of your application. Records you save don't get sucked into a magic black box that just responds to your queries as your app needs grow.

DBAs are not going to go anywhere. Sure, you can scale the DB in the cloud quite a bit in and perform well but it's not free :) In the cloud, you would be literally paying for your bad design decisions in terms of hard dollars rather than performance issues.

Most start-ups or in house apps are fine while the salary(dba) => cost of cloud. However, that magic condition starts returning false pretty quickly if you are doing any non-trivial data management.

Re: The Database Administrator is dead

#25

Post by shoeless » Fri, Dec 13, 2013, 2:24 PM UTC

falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Exactly. You could extrapolate this into any "The [whatever] is dead" statement that we see so often. And it's not getting any easier - e.g. are we capturing and processing less data these days? Is the data we store less valuable?

> You can't get a DBA from a college, from a technical school, or from any other form of formal education. Most DBAs these days are grown internally from developers or system administrators who decide to (or are forced to) specialize while on the job.

I can't say how much truth is in this statement. This stuff if learned organically by doing it on real world projects. It's scary at times to think that you just can't teach this stuff.

Re: The Database Administrator is dead

#26

Post by bluedino » Fri, Dec 13, 2013, 2:24 PM UTC

The cloud is killing all the low-level admins. Mail administrators? Backup operators?

Moving to BYOD will continue to reduce the numbers of administrators as well. Eventually you'll have two groups of IT staff, a very small group of high-level engineers who build and implement everything, and then a very large group of low-skill helpdesk type people who reset your accounts and fill in your login information on your device.

Amazon's Mayday support service shows this already. Soon, a form of this will be on every product you can think of. Office, Windows, every tablet and computer in your office.

Re: The Database Administrator is dead

#27

Post by cognivore » Fri, Dec 13, 2013, 2:25 PM UTC

So, a thought experiment.

Pick any data storage system you like - MongoDb, Redis, Riak, whatever.

Now, you get to place a bet. In the future (let's say 25 years) what will still exist: your choice of NoSQL, or the standard database with schema, SQL, access, and so on? If you choose wrong, you die.

Now you get to see the future and see if you die. Which are you betting on?

Re: The Database Administrator is dead

#28

Post by csmuk » Fri, Dec 13, 2013, 2:26 PM UTC

falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

I disagree. dedicated DBAs are on their way out.

Databases are becoming pretty good at managing themselves and the marginal performance gains from tuning usually are easily offset by throwing bigger kit at the problem or throwing more cash at the plan you are on.

Re: The Database Administrator is dead

#29

Post by lmm » Fri, Dec 13, 2013, 2:26 PM UTC

static_typed wrote: post #2

The DBA is not dead! They just look and smell that way! But in all seriousness, if your app uses a database, you are incompetent not to employ an expert to help with the database, whether advising on the query plan of those non-performant queries, or what is the best setup for the current stage of the business and app, they are very useful.

> if your app uses a database, you are incompetent not to employ an expert to help with the database, whether advising on the query plan of those non-performant queries, or what is the best setup for the current stage of the business and app, they are very useful

Given modern databases and modern hardware, I think the vast majority of applications never reach the point where query performance is an issue. For many small software companies hiring an expert in databases makes no more sense than hiring an expert in operating systems or networking - unless your needs are very specialized, these things work well enough out of the box.

Re: The Database Administrator is dead

#30

Post by cygwin98 » Fri, Dec 13, 2013, 2:27 PM UTC

falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Most DBAs these days are grown internally from developers or system administrators who decide to (or are forced to) specialize while on the job.

I agree with you on above statement. Though my conclusion is different from yours. I think in the future the line among DBA/SysAdmin/Developers will become even more blurred , developers will be trained/required to take over more and more work from DBA and sysadmin (DevOps anyone?); consequently, the demand for dedicated roles such as DBA and sysadmins will diminish. Hope I am wrong though.