Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Database Administrator is dead

thenextweb.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: The Database Administrator is dead

#61

Post by falcolas »

mdm_ wrote: post #31
falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Could you provide any tips/advice/reading recommendations to a programmer who is interested in learning more dba stuff? It would be appreciated!

Sure - here's a list of what's good to know for MySQL. Other DBs are going to have different needs, though the indexing data is good to know regardless.

Start with the book High Performance MySQL. [1]

Follow up with the whitepaper "Causes of Downtime". [2]

Then find a copy of the IMDB dataset, put that in a database, and write an app against it. Make that app perform well, then simulate load against the app (pretend it hit the top of Reddit and Hacker News simultaneously), and keep it performing well.

After that, it's a matter of practical practice.

[1] http://www.amazon.com/High-Performance-MySQL-Optimization-Re...

[2] http://www.percona.com/redir/files/white-papers/causes-of-do...

Re: The Database Administrator is dead

#62

Post by jbooth »

mattzito wrote: post #59
jbooth wrote: post #21

Earlier quoted context omitted.

Why would someone run Oracle in this day and age? "We have 100k lines of established PL/SQL running our business" is a reason, of course, but you could make a similar argument that COBOL development is alive. PL/SQL pretty much always leads to an unmaintainable mess, and if you need BIG DATA, you can go much bigger without Oracle than you can with it, and cheaper to boot.

Guh - every few months this comes up, and it's fairly hilarious to me how much anti-oracle dogma there is here. Oracle is the most feature-rich database in the industry, and the most expensive, and the most complicated. This comes with benefits and curses. It is freakishly powerful. If you want, you can roll back just your view of the database to three hours ago. You can show different versions of the same schemas to…

Well, I'll admit that I haven't used Oracle in 4 years, but back then, the automatic-sharding thing (goldengate IIRC?) just plain didn't work.

I guess I'm just biased towards solving scalability problems at the application level. It seems like an uphill battle to take a declarative/descriptive language like SQL and tune it to execute a query the way you want it executed -- it seems a lot more straightforward to just write code that does what you tell it to.

Re: The Database Administrator is dead

#63

Post by falcolas »

csmuk wrote: post #57
falcolas wrote: post #52

Earlier quoted context omitted.

> All of our development staff are capable of diagnosing these issues and providing suggestions. I hope you're paying them well, because if they're all crack DBAs on top of being crack developers, they're the proverbial unicorn, and are probably worth $2-3 hundred thousand apiece.

No they just know how not to fuck up their queries to start with so we don't need a DBA to optimise 300 join statements...

Knowing how to write good queries and indexes is admittedly 80% of having a good performing database. However, as with any other 80/20 split, the remaining 20% is the hard part, and you'll end up paying a specialist to take care of it for you.

Or your dataset and query volume will not grow to the point where you actually need a performant database, but that's a business problem, not a DB solution.

Re: The Database Administrator is dead

#64

Post by falcolas »

cygwin98 wrote: post #30
falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Most DBAs these days are grown internally from developers or system administrators who decide to (or are forced to) specialize while on the job. I agree with you on above statement. Though my conclusion is different from yours. I think in the future the line among DBA/SysAdmin/Developers will become even more blurred , developers will be trained/required to take over more and more work from DBA and sysadmin (DevOps a…

My role within our DBA group is that of devops exactly (after spending a year as a line DBA).

Given that background I still think that the topic of databases is just too deep for a generalist. I know a lot about the MySQL database (and a little about PostgreSQL), enough to write failover software, automate deployments, write guardian crons which slap down problematic queries & pre-emptively, automate backups, do vip failover and haproxy configuration... and I still have to go to my boss for most of the hard questions.

His knowledge encapsulates 12 years of working with and around MySQL, and it's proven invaluable to our customers. Knowing when to force certain optimizations, how to make subqueries run O(1) vs O(n), how to rebuild a complete database from binary logs, how to configure MySQL to work with SSD caches... these problems don't come up often, but when they do, not having a DBA available to you means contracting out to one at exorbitant rates.

It's the difference between a few minutes of downtime when the proverbial dung hits the fan, versus a few hours or days.

Re: The Database Administrator is dead

#65

Post by falcolas »

Shorel wrote: post #40
falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

> He needs someone to figure out how to make the cartesian product queries he's written into efficient queries. I wish our DBAs were like that. I am a developer with Sublime Text. I have to make the queries fast. I have to design good indexes. They can complain if a query is slow, but they never actually help to fix it. They only have to make and restore backups when disks die.

Sounds like you have a sysadmin with mad interview skills. ;)

Sorry to hear it, either way.

Re: The Database Administrator is dead

#66

Post by falcolas »

shoeless wrote: post #25
falcolas wrote: post #16

I guess I had better tell my colleagues that our jobs are all dead! Wait, it's an absolute statement for a headline, so it's actually "absolute crap". > These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome. And when he is successful and gets clients, and a few thousand rows in his database, he realizes that he needs someone to keep that database alive. He…

Exactly. You could extrapolate this into any "The [whatever] is dead" statement that we see so often. And it's not getting any easier - e.g. are we capturing and processing less data these days? Is the data we store less valuable? > You can't get a DBA from a college, from a technical school, or from any other form of formal education. Most DBAs these days are grown internally from developers or system administrators…

> This stuff if learned organically by doing it on real world projects. It's scary at times to think that you just can't teach this stuff.

Yup. And with the supply being so low, it's hard to get a DBA (you'll probably have to steal one from another business), so people are flocking more and more to DBaaS, and DBaaS providers are more than happy to propagate the fiction that "you don't need a DBA, you have us!".

It honestly doesn't bother me much that they make these statements; it's marketing.

On the other hand, believing those statements harm our customers; they spend time and money to migrate to these providers and find out the hard way that they still need someone who can handle their DBs for them. That does bother me.

Re: The Database Administrator is dead

#67

Post by jrarredondo »

cognivore wrote: post #27

So, a thought experiment. Pick any data storage system you like - MongoDb, Redis, Riak, whatever. Now, you get to place a bet. In the future (let's say 25 years) what will still exist: your choice of NoSQL, or the standard database with schema, SQL, access, and so on? If you choose wrong, you die. Now you get to see the future and see if you die. Which are you betting on?

Sometimes you just bet on red AND on black to stay safe while keeping the wheel rolling and the drinks coming.

Re: The Database Administrator is dead

#68

Post by mattzito »

jbooth wrote: post #62
mattzito wrote: post #59

Earlier quoted context omitted.

Guh - every few months this comes up, and it's fairly hilarious to me how much anti-oracle dogma there is here. Oracle is the most feature-rich database in the industry, and the most expensive, and the most complicated. This comes with benefits and curses. It is freakishly powerful. If you want, you can roll back just your view of the database to three hours ago. You can show different versions of the same schemas to…

Well, I'll admit that I haven't used Oracle in 4 years, but back then, the automatic-sharding thing (goldengate IIRC?) just plain didn't work. I guess I'm just biased towards solving scalability problems at the application level. It seems like an uphill battle to take a declarative/descriptive language like SQL and tune it to execute a query the way you want it executed -- it seems a lot more straightforward to just…

GoldenGate is for multi-master replication and shares a lot of the challenges you'll find with any multi-master replication solution.

No, I was referring to Oracle RAC, which does away the need for sharding. All of your nodes see a complete and comprehensive picture of the data on a shared set of disks.

> I guess I'm just biased towards solving scalability problems at the application level.

That's a totally valid strategy, and indeed, an option that many companies go for. But you're just shifting complexity from one place to another. Either you're going to be building in complexity to make your application aware of data distribution, movement, sharding, and so on - or you're going to use a more complex data storage platform like Oracle.

And SQL tuning is a skillset, much like writing good code. If you are good at SQL tuning, it's not that hard.

My point is just that it depends on your business requirements. I personally think Oracle's price point is so high that I would never use it. But if money were no object, and I was designing an application, why would I want to have my application have to think about where the data lives?

Wouldn't it be a lot simpler to just say, "go here for your data", and let the dedicated application deal with that?

Or put another way, in the same way it seems insane to shove 100k lines of business logic into the database layer, why doesn't it seem insane to shove 100k lines of data management logic into the application tier?

Re: The Database Administrator is dead

#69

Post by islon »

> These days, the technology decision maker is the dude with Sublime Text open and a cloud control panel up in Chrome.

My sublime text is open and I have a cloud control panel open on chrome, am I the technology decision maker? Nobody told me that here on the company.

Wait... maybe nobody told me because... I'm the decision maker D=

Re: The Database Administrator is dead

#70

Post by falcolas »

mattmanser wrote: post #49
falcolas wrote: post #22

Earlier quoted context omitted.

> What we really need is to demystify DB performance, which for the most part is fairly simple. Without getting into the rest of your argument, I'd like to quickly address this. No, it really isn't. In your typical MySQL database, your performance for a simple "select * from x where y" is going to go through a lot of complicated machinery (most of which can be tuned for performance), a few points of which I will enum…

This sort of nonsense is exactly the sort of mysticism I'm talking about. My, for example, do you think every query needs to be tweaked for this one: Push the query plan down into InnoDB So why did you include it on the list? 5a + 5b, Is MySQL so bad at memory management that you tweak it for every query? So why did you include it. Virtually everything on that list are implementation details that the vast majority of…

> Virtually everything on that list are implementation details that the vast majority of applications and developers don't need to even think about.

Particularly if you hire a DBA.

But that's not what you meant... and you're right. If your business doesn't generate enough query traffic and data to stress your database, then no, you don't need more than the indexing and smart query development which your average developer can learn.

The moment your business hits that wall, however, you'll be scrambling to get someone who can dig into those "implementation details" and wring every ounce of performance out of your database.

> [...] do you think every query needs to be tweaked for this one: Push the query plan down into InnoDB

Nope. However, for that one table that is better served by using Tokudb, or MyISAM, or the Archive engine it does matter (yes, there are actually use cases for using MyISAM tables instead of InnoDB).

> 5a + 5b, Is MySQL so bad at memory management that you tweak it for every query?

Again, for every query? No. Also, 5a and 5b have nothing to do with memory management, and everything to do with the size of the dataset you have to pull into memory to identify and return the results. Knowing how this affects the buffer pool LRUs, disk usage patterns, and how to optimize the interaction between the two can be vital.

Post reply on HN