Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MySQL is bazillion times faster than MemSQL

dom.as

Post reply on HN
Print view View original

31–40 of 148 posts

Re: MySQL is bazillion times faster than MemSQL

#31

Post by arihant »

When debating a fresh new database project - benchmarks aren't really the way to go, because those can be improved. What should be debated is what underlying fundamental has the new project gotten right. Almost every successful database project got something fundamentally right - MySQL, Redis (in-memory), MongoDB(page based storage), Riak(Distributed).

On that note, here's my take on MemSQL - People who love MySQL, almost never want to migrate. People who hate MySQL, usually hate it because of SQL. People who are far along in the projects enough that decision of database is no longer a matter of taste, don't use a brand new untested DB.

Therefore, I am unsure how a brand new, in-memory, SQL based solution fits.

Re: MySQL is bazillion times faster than MemSQL

#32

Post by midom »

ericfrenkiel wrote: post #5

When someone writes on their intro "... and now in return I want to waste your time a bit." it's a sign that someone isn't being serious. Domas' main criticism is that our video, http://vimeo.com/44087431 , uses a MySQL with standard defaults. Since tuning a database is by definition a custom process, we wanted to demonstrate what performance you'd get "out of the box." The video speaks for itself. MemSQL can push 80…

No, that is not the highest praise you can get from me. It definitely is very interesting technology, and what it does well, it does really well - though I didn't get to benchmark parallel performance, as I had work to do :-)

I wouldn't have written that post the way it is if your marketing would be honest and with lots of actual technical merit. There're plenty other companies in the community that have invested much more into decent technical communication.

I agree that in some cases pure durability isn't needed and the best-effort with enforced maximum slippage is good enough, but still, pure durability implementation is extremely naive, and I can show that it hits benchmarks really badly :-)

Your optimizer does not optimize very simple things, and though you can claim that in-memory performance hides that, still, it is nowhere close a drop-in replacement of any kind.

I understand what you've done from technology perspective, and it is an interesting approach (albeit I'm not trying to evaluate applicability).

I just cannot approve the way you did initial publicity - unfair comparisons are unfair comparisons, and if you're using them that way, I can come up with my unfair comparison :)

Re: MySQL is bazillion times faster than MemSQL

#33

Post by nikita »

MemSQL CTO here. Great article- Domas has done a good job of digging into the internals of MemSQL! A few questions/comments:

1.) The range query issue you pointed out can be explained by a well known limitation of skip lists. Unlike B-Trees, skip lists are unidirectional. By default, our indexes are ascending, so indeed you have to skip to the end to run a MAX() or "ORDER BY id DESC" query. To fix this, just change the primary key to be descending in your schema:

    CREATE TABLE x (id int, ..., PRIMARY KEY id (id) DESC, ...)
This is explained here http://developers.memsql.com/docs/1b/indexes.html#skip-list-.... If you want both behaviors in your app, you'll have to create two skip list indexes (for now).

2.) The transaction-buffer > 0 setting is really where we shine. Synchronous durability is something we have for flexibility, but I'll be the first to admit that it's not really optimized to perform well. The customers that we're working with are okay with this. And it's inspired by what modern companies do. Maybe it's changed in the time since I've left (Domas?) but Facebook certainly does not write comments synchronously to disk.

3.) Our synchronous durability does indeed flush in 50 MS cycles, so you'll see poor performance on single threaded inserts. However, as you pointed out, we're optimized for the case with multiple concurrent writers. Since MemSQL implements group commit on high parallel load throuput picks up. Sure, writing your own very specific benchmark you can show us writing poorly, but we've never worked with a customer that's needed single threaded, synchronous writes to shine. If this is your use case, unless you need substantial read performance, MemSQL is not the database for you.

Re: MySQL is bazillion times faster than MemSQL

#34

Post by maratd »

cs702 wrote: post #15

Whenever I read an article like this, I often end up feeling that on the 'truthiness' spectrum the scale goes like this: lies, damn lies, statistics, benchmarks ;-)

Every public benchmark is a horrible blasphemous lie. The only benchmarks that are honest are the ones you conduct in private, with your very specific use case. There is no point in publishing them, because they're useless to everybody but you. Public benchmarks, on the other hand, are crafted as advertising for your favorite [fill in the blank].

Re: MySQL is bazillion times faster than MemSQL

#35

Post by ChikkaChiChi »

aphyr wrote: post #30
ChikkaChiChi wrote: post #26

Simple solution: [1] Someone create a project on GitHub [2] Upload an amply sized CSV full of data [3] Publish list of queries needed to "gauge" speed [4] Allow anyone to post their metrics to see how well tuned any database can be in comparison The only other criteria would be to ask them to all use the same form factor. EC2 makes the most sense.

Shared environments with virtualized IO are not good places to run benchmarks.

I couldn't agree more, but its the closest to standardized testing we can get that everyone can use.

I HATE the idea personally, but I cannot think of a better way to start comparing apples to apples.

Re: MySQL is bazillion times faster than MemSQL

#36

Post by bsg75 »

arihant wrote: post #31

When debating a fresh new database project - benchmarks aren't really the way to go, because those can be improved. What should be debated is what underlying fundamental has the new project gotten right. Almost every successful database project got something fundamentally right - MySQL, Redis (in-memory), MongoDB(page based storage), Riak(Distributed). On that note, here's my take on MemSQL - People who love MySQL, a…

> People who hate MySQL, usually hate it because of SQL.

Or because of the shortcuts it has taken to avoid the "hard stuff", which results in developers taking shortcuts by using MySQL to avoid the "hard stuff".

Love or hate SQL for what it is, not the MySQL implementation of it.

Re: MySQL is bazillion times faster than MemSQL

#37

Post by btilly »

ChikkaChiChi wrote: post #26

Simple solution: [1] Someone create a project on GitHub [2] Upload an amply sized CSV full of data [3] Publish list of queries needed to "gauge" speed [4] Allow anyone to post their metrics to see how well tuned any database can be in comparison The only other criteria would be to ask them to all use the same form factor. EC2 makes the most sense.

http://www.tpc.org/ already does something like this, but much more rigorously.

Re: MySQL is bazillion times faster than MemSQL

#38

Post by merlincorey »

ericfrenkiel wrote: post #13
mikeash wrote: post #10

Earlier quoted context omitted.

I thought his main criticism was that MemSQL isn't durable by default, and that when you enable that it becomes incredibly slow. That's a pretty big deal, and IMO running database benchmarks on an unsafe configuration like this is extremely dishonest.

That's a bit heavy-handed; additional benchmarks will be published against a variety of configurations and other databases. Companies use MemSQL because it uses memory as the primary locus of data. If you have a fast data problem, you couldn't use a disk-based system. At high speeds, you'd want to deploy any database in an active-active mode.

I just want to echo mike above with a slightly different tact that perhaps will get across my perception of our feelings.

It's dishonest, at the very least, to claim ACID compliance and then do benchmarks against a known ACID compliant competitor without having the same level of ACIDity enabled in your program.

If you want to prove how fast MemSQL is against MongoDB, that's one thing (I understand it also can be "fully durable" but it is not by default due to speed concerns). To prove how fast MemSQL is against MySQL, you have to prove it on the same playing field, which means it has to, necessarily, be just as ACIDic and respond to "real" queries on large datasets with high concurrency.

tl;dr Do some benchmarks of apples-to-apples ACID, and do something about the common case of a head of a table, then you will be honestly faster (or not) than MySQL

Re: MySQL is bazillion times faster than MemSQL

#39

Post by [deleted] »

nikita wrote: post #33

MemSQL CTO here. Great article- Domas has done a good job of digging into the internals of MemSQL! A few questions/comments: 1.) The range query issue you pointed out can be explained by a well known limitation of skip lists. Unlike B-Trees, skip lists are unidirectional. By default, our indexes are ascending, so indeed you have to skip to the end to run a MAX() or "ORDER BY id DESC" query. To fix this, just change t…

[deleted]

Re: MySQL is bazillion times faster than MemSQL

#40

Post by timaelliott »

nikita wrote: post #33

MemSQL CTO here. Great article- Domas has done a good job of digging into the internals of MemSQL! A few questions/comments: 1.) The range query issue you pointed out can be explained by a well known limitation of skip lists. Unlike B-Trees, skip lists are unidirectional. By default, our indexes are ascending, so indeed you have to skip to the end to run a MAX() or "ORDER BY id DESC" query. To fix this, just change t…

You seem to be missing the main reason why people have an issue with MemSQL. The issue are quotes like this on your website and video:

"MEMSQL IS 30 TIMES FASTER THAN MYSQL."

It's an extremely invalid and biased comparison. You're quite literally comparing the speed of writing to RAM versus the speed of writing to disk. If you didn't make such ridiculous assertions, people would accept your product for the actual awesome things it does and not focus on refuting your bullshit claims.

Post reply on HN