Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQL Databases Don't Scale

adam.blog.heroku.com

Post reply on HN
Print view View original

71–80 of 112 posts

Re: SQL Databases Don't Scale

#71

Post by pj »

derefr wrote: post #58
trezor wrote: post #53

Earlier quoted context omitted.

Being ideologically for open software and an open operating system is not the same as "hating Microsoft with a passion". Mixing these two different concepts into the same bag is misleading, as the first one represents an ideological conviction and the other merely childish spite. And if you insist on being cheap, don't be surprised when it turns out that your free database was indeed some cheap stuff which doesn't co…

It's a causal relationship. If you believe that all software should be free (the GNU folks—those "first kernel developers"), then you must at least dislike any company which tries to profit from the creation of artificial scarcity of software. The two groups (the idealists and the "haters"), which now have very little overlap, originally started much the same. > Don't be surprised when it turns out that your free dat…

I've seen the term artificial scarcity used a few times in this context. Could you elaborate on what you mean by that?

When I think artifical scarcity, I think something like DeBeers who has warehouses full of diamonds they are hoarding. Whereas with software, there actually is a scarcity of great programmers. There are only so many of them.

With software, you know... it takes time and energy to support it. It costs money to write help documentation, pay for servers where people can download it. All that stuff costs money.

Is open source anti-capitalism?

Is the artificial scarcity coming from the thought that because software is just 1's and 0's that it can be copied almost infinitely? It takes a lot of time and energy to build something that is worth copying. What about all that research and development and risk that was taken to build it. How are those risk takers and R&D staffs going to generate a return on that investment without charging for the product they create?

Re: SQL Databases Don't Scale

#73

Post by dflock »

gaius wrote: post #54
dflock wrote: post #6

Earlier quoted context omitted.

I think one of the points made at the end was that there kinda isn't one - people have been trying to come up with the perfect solution to scaling RDBMS's for ever and the current state of the art (pretty much) is as described in the article - i.e, not great. People with really massive MySQL setups (eBay, for example) basically just do a huge amount of the stuff described here, but with lots of message-queue type stu…

the current state of the art (pretty much) is as described in the article - i.e, not great That simply isn't true. For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how…

What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.

Re: SQL Databases Don't Scale

#74

Post by DanielBMarkham »

dflock wrote: post #73
gaius wrote: post #54

Earlier quoted context omitted.

the current state of the art (pretty much) is as described in the article - i.e, not great That simply isn't true. For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how…

What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.

If that were the case, then I think the concluding sentences should be changed.

When hundreds of companies and thousands of the brightest programmers and sysadmins have been trying to solve a problem for twenty years and still haven’t managed to come up with an obvious solution that everyone adopts, that says to me the problem is unsolvable.

Re: SQL Databases Don't Scale

#75

Post by tybris »

alexgartrell wrote: post #12

Crazy idea: Write your data-handling code in it's own tight little module. Use good abstractions so that the rest of your app doesn't give a crap HOW it's happening. When it's time to update to millions of clustered servers running bigTable, you rewrite the module, and you're done. Software Engineering saves the day!

Well, you're right in the sense that Newton was right about gravity. It works perfectly down here on earth, but breaks down out in the universe. In a distributed system you'll want to take into account caching, replication, atomicity, security, etc. In other words, that's either going to be a hell of a complex API or an API that gives you too little control to build a high-performance system. There's no telling what works best, every computer system is a set of trade-offs and it always depends on the application.

Re: SQL Databases Don't Scale

#76

Post by antirez »

I'm the author of Redis so I should be biased the other way around, but the Redis and KV experience taught me that to be exposed to the KV parading was for me a similar experience as learning the Scheme language, I started to write code in imperative languages in a new way. In the same way once you start thinking at scalability of your data in a new way, from the point of view of partitioning the data, organizing this data in away that is easy to access for your usage pattern, make judicious use of serialization, SQL databases can be a good pick. Simply you need to abandon the paradigm of let's design our nice tables and run our multiple joins and group by against.

But of course designing DBs in this new way makes most of the SQL features not needed in most scenarios... and you still have the overhead. And there are a number of other limits now that complex designs are not a good idea, for instance there is no way to get back the data in the natural ordering (the order you pushed this data, or the reverse)...

This is where KV stores start to be interesting as real world alternative, not just in order to learn the paradigm of scalability. But again, if you don't trust KV stores it is truly possible to use an SQL database in a more conservative and scalable way.

Re: SQL Databases Don't Scale

#77

Post by gaius »

dflock wrote: post #73
gaius wrote: post #54

Earlier quoted context omitted.

the current state of the art (pretty much) is as described in the article - i.e, not great That simply isn't true. For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how…

What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.

That's a very curious statement, especially for HN. Why would you consider needing good, experienced people to be a disadvantage?

Re: SQL Databases Don't Scale

#78

Post by derefr »

pj wrote: post #71
derefr wrote: post #58

Earlier quoted context omitted.

It's a causal relationship. If you believe that all software should be free (the GNU folks—those "first kernel developers"), then you must at least dislike any company which tries to profit from the creation of artificial scarcity of software. The two groups (the idealists and the "haters"), which now have very little overlap, originally started much the same. > Don't be surprised when it turns out that your free dat…

I've seen the term artificial scarcity used a few times in this context. Could you elaborate on what you mean by that? When I think artifical scarcity, I think something like DeBeers who has warehouses full of diamonds they are hoarding. Whereas with software, there actually is a scarcity of great programmers. There are only so many of them. With software, you know... it takes time and energy to support it. It costs…

Artificial scarcity is not an ethical or legal principle, but an economic one. Because 1s and 0s can be "copied almost infinitely", in order to make money on it one has to enforce an artifical constraint on the number of copies that are allowed to be made. The distinction is simply that with its opposite, natural scarcity, no effort needs to be made to ensure that each object has value on its own.

To put it another way, the "default state" of a naturally scarce commodity, when it is simply produced and then discarded, is still high-value. It can be resold at auction, or traded in a market, and will be fought over even in a state of legal and moral anarchy. The default state of an artificially scarce commodity, however, is value asymptotically approaching zero; without some sort of agency to "prop up" its value, it is worthless, and no secondary party will assign it value unless it is forced to as part of a larger deal (e.g. accepting the legal code to be a citizen of a country.)

Your question ("is open source anti-capitalism?") is just a matter of equivocation. Open Source fits just fine with capitalism, but it is not the capitalism you would first imagine (i.e. that of the US); it is instead "true", or lassiez-faire Capitalism.

In Lassiez-faire Capitalism, there can be no artificially scarce commodities; they are simply a market inefficiency to be eliminated, along with those producing them. A true Capitalism would destroy the value in all "information products"—movies, books, music, games, and, yes, software. In a true Capitalism, value is just "what people are willing to pay"; you don't deserve an ROI just because you worked hard, you only earn money if people feel that your commodity has value to them. In such a market, the only software that could exist is that which was produced for other motives than profit—open source software—or produced as a means to an end of profit, e.g. software that makes a business process more efficient, rather than software that is a product in-and-of-itself.

And now you see why we do not use such a capitalism ;) Though some "creatives" would survive, whether as on-the-whole consultants or by donations from fans (see http://pc.ign.com/articles/967/967564p1.html), the majority would not. In reality, a majority of people desire to keep these creatives around and producing, even at the cost of large market inefficiencies. Realize thus that copyright, more than anything, is a form of socialism, in that it redistributes wealth to those we think deserve a "fair share" for their efforts. Sort of mixes up the arguments most people have on the subject: Open Source is anti-socialist ;)

Re: SQL Databases Don't Scale

#79

Post by pj »

derefr wrote: post #78
pj wrote: post #71

Earlier quoted context omitted.

I've seen the term artificial scarcity used a few times in this context. Could you elaborate on what you mean by that? When I think artifical scarcity, I think something like DeBeers who has warehouses full of diamonds they are hoarding. Whereas with software, there actually is a scarcity of great programmers. There are only so many of them. With software, you know... it takes time and energy to support it. It costs…

Artificial scarcity is not an ethical or legal principle, but an economic one. Because 1s and 0s can be "copied almost infinitely", in order to make money on it one has to enforce an artifical constraint on the number of copies that are allowed to be made. The distinction is simply that with its opposite, natural scarcity, no effort needs to be made to ensure that each object has value on its own. To put it another w…

Okay, I think I understand better now. There's only physically so much gold on the planet. Barring alchemy, we can't make more of it, so it has a natural scarcity.

From that, I gather that we can't use the traditional economics of Supply and Demand to allow a market to decide the value of software. Because the denominator in the equation demand divided by supply is infinite, essentially, it doesn't matter what value demand has in the numerator, because anything divided by infinity is zero.

But my initial impression of this model is that it is doomed to failure. It is doomed to failure, because the artificially scarce product being produced itself requires the consumption of naturally scarce products that are subject to the laws of supply and demand. The developers that produce the software for example need to eat, the need to live in a house, they need to consume energy to drive to work, etc...

So how do we find a balance? Do we find a new way to value software or will software just go away? Will software be relegated to a charitable organization like in the article you linked to, which was quite interesting by the way, so thanks for that. Will the only valuable software be that software which is paid for in advance to be developed at the risk of the consumer? "If you pay me, I'll build it, otherwise, I won't" sort of scenario?

It's like the software only has value if it doesn't exist. If the mechanism to produce the software exists only in the minds of a scarce few who can implement the solution or who have the ability to control access to it, perhaps through SaaS or some other monthly subscription mechanism like WoW or battle.net.

Perhaps we can value software based on how much additional revenue it helps you generate or how much savings it generates through optimizations or automations.

How are we going to strike that balance? Is the gold rush for software over? If there is no carrot, who will run those wheels and invest in our future? Who will take the time to solve the problems before the problems arise? Take Oracle vs. MySQL. Oracle has solved a lot of the problems MySQL has. Over time, as people contribute to the MySQL code base, those problems can be resolved, but consumers of MySQL have to wait for someone else to implement the solution, or they have to pay for a computer programmer to find a work around or implement some solution outside MySQL.

It feels like it will slow us down. Corporations like Oracle and Microsoft will survive a little while longer, but if MySQL ever actually does become as good as Oracle, then people will stop paying for Oracle. If people stop paying for Oracle, Oracle can't hire the best and the brightest.

Well, I appreciate the answer, I do understand better, but that just leaves me with more questions, so I'm thinking out loud.

All I can think of right now is that software will all move to aaS models or be embedded in physical devices so that we can attach a natural scarcity to them. There are only so many factories that can produce microchips. The article you linked to suggested selling plastic figurines with the software, which is a similar idea. It harkens back to the age of dongles.

Re: SQL Databases Don't Scale

#80

Post by [deleted] »

gaius wrote: post #77
dflock wrote: post #73

Earlier quoted context omitted.

What hardware is this running on, roughly? What the author was getting at, I think, was not that you can't do it, just that it's hard/expensive; you need fairly beefy hardware and experienced DBA's to manage horizontal scaleout manually.

That's a very curious statement, especially for HN. Why would you consider needing good, experienced people to be a disadvantage ?

[deleted]
Post reply on HN