Tokumx would solve all of these issues. 2TB goes a long way in tokumx with lzma compression.
Migrating from MongoDB to Cassandra
41–50 of 65 posts
Re: Migrating from MongoDB to Cassandra
#42Earlier quoted context omitted.
Knowing some of the FC people first hand, MongoDB did actually serve them fairly well for a substantial amount of time. Until they started hitting max limits, it didn't really make sense to move to c . Going straight to C or something like it would have been almost cargo-cultish ( eg If we build industrial strength, we will get industrial levels of traffic ).
Exactly. Can you imagine all of the Java/C developers telling Ruby developers that they are idiots and don't know anything about programming ? Simply because they choose a technology that is designed for developer productivity at the expense of scalability. Because that what seems to happen for every database discussion.
>Can you imagine all of the Java/C developers telling Ruby developers that they are idiots and don't know anything about programming ?
Unfortunately this seems to happen all the time.
Re: Migrating from MongoDB to Cassandra
#43When I read posts like this all but confirming MongoDB isn't the great product 10Gen make it out to be, I wonder how the heck MongoDB are still even relevant and then I remind myself of the fact that 10Gen have one of the best marketing and sales teams in the game at the moment. While MongoDB has improved greatly over previous versions, I can't help but feel if 10Gen put as much effort into improving their product as…
I'm not disagreeing with you, but this is not what the article is about:
"MongoDB was not a mistake. It let us iterate rapidly and scaled reasonably well. Our mistake was when we decided to put off fixing MongoDB’s deployment, instead vertically scaling (...) By the time we had cycles to spend, it was too late to shard effectively It would have been terribly painful and unacceptably slowed our cluster for days."
It seems they weren't sharding their data. The advantage of the popular NoSQL databases like MongoDB is that they allow easier horizontal scaling than general purpose RDMS (though this is debatable - Postgres and Oracle allow you to make the same trade offs as NoSQL databases, they just don't force you to)
When you read the rest of the article, they explain that when they had to make a painful transition anyway, they chose Cassandra, to a different set of advantages and disadvantages that suited their needs.
MongoDB is still relevant because it is well supported. The product is well documented, excellent tooling is available, it is widely adopted, so that when you have a question, you can often google the answer.
It has an elegant query language; especially the built in aggregation framework is far more convenient than having to write map reduce functions for every query. It is easy to deploy and use. All these things make it a product that is pleasant to use from the point of view of a developer or DBA. I think you underestimate the importance of non-technical advantages and disadvantages
I am just not convinced it is the best database because I just don't see a use case for a 'general purpose' NoSQL database. For general purpose storage, RDMS like Postgres and Oracle are great. They support sharding if you really want it, and even allow indexing of unstructured data these days. They don't force you to use joins and transactions if you don't need them, but at least they support these features when you do.
Re: Migrating from MongoDB to Cassandra
#44Earlier quoted context omitted.
> Cassandra and MongoDB are completely fine as general purpose databases. Whenever I hear this, I feel that the person saying it either doesn't have enough experience or that every problem to them looks like a nail for their particular flavor of NoSQL. That's like saying a street bike is a general purpose bicycle. I'm sure someone out there can make it work on an outdoor offroad trail, but is it ideal? Cassandra is r…
> Cassandra is really awesome at writes but reads are really expensive This is getting old, and is just plain untrue. Reads are more expensive than the extremely cheap writes, but they really aren't expensive. > Atomicity at the row level only Partition level, really, in modern terminology (apologies if that's what you meant).
> This is getting old, and is just plain untrue. Reads are more expensive than the extremely cheap writes, but they really aren't expensive.
It has improved a lot over time but it isn't a myth due to the fact that it's not good out of the box. You have to spend some time with configuration and mulling over the schema to mitigate it. It's something you don't have to really think about as much on other data stores.
Re: Migrating from MongoDB to Cassandra
#45When I read posts like this all but confirming MongoDB isn't the great product 10Gen make it out to be, I wonder how the heck MongoDB are still even relevant and then I remind myself of the fact that 10Gen have one of the best marketing and sales teams in the game at the moment. While MongoDB has improved greatly over previous versions, I can't help but feel if 10Gen put as much effort into improving their product as…
> 10Gen have one of the best marketing and sales teams in the game at the moment. No doubt. I have 2 MongoDB mugs even though I vowed to never touch a product produced by them again (it was, after the whole -- we'll throw your data write requests over the wall and pray fiasco).
I got one from SCALE x10. Southern California Linux Expo
Re: Migrating from MongoDB to Cassandra
#46"To buy us time, we ‘sharded’ our MongoDB cluster. At the application layer. We had two MongoDB clusters of hi1.4xlarges, sent all new writes to the new cluster, and read from both..."
I'm curious about this. Why were you doing the sharding manually in your application layer? Picking a MongoDB shard key - something like the id of the user record - would produce some fairly consistent write-load distribution across clusters. Regardless - it seems like write-load was a problem for you, yet you sent all the write load to the new cluster - why not split it?
Re: Migrating from MongoDB to Cassandra
#47When I read posts like this all but confirming MongoDB isn't the great product 10Gen make it out to be, I wonder how the heck MongoDB are still even relevant and then I remind myself of the fact that 10Gen have one of the best marketing and sales teams in the game at the moment. While MongoDB has improved greatly over previous versions, I can't help but feel if 10Gen put as much effort into improving their product as…
There is one thing MongoDB does spectacularly well - you can feed in arbitrary JSON and get the same JSON back out. (No need to define schemas or play any kind of system/db administrator.) Even the queries have the same "shape" as JSON, so no need for another arbitrary query language. It will eventually bite you, and bite you hard. But you'll be well into the millions of records before that happens. Developers and pr…
Yes. It does bite you quickly. As you add more models you start to duplicate a lot more information and by that time you'd think relational makes sense but you have to continue to use MongoDB. The option you got is either embed or reference. And still, there is no JOIN in mongo so you'd iterate many collections and do combine within your application code.
I think as PostgreSQL continue to improve its JSON data type people will look at SQL again even if they need to a basic model working. Because at the end working with constraints can help. Well, either side will bite you but one has to weigh...and sure that's a tough question.
Re: Migrating from MongoDB to Cassandra
#48Earlier quoted context omitted.
Cassandra 2.0 can handle 5TB per node easily, 10TB with some care. Best to scale out, not up. That said, if someone else has already made the hardware choice for you, you can always run multiple C* nodes on a single machine. I know several production clusters that fit this description.
PostgreSQL can handle petabytes easily. However if you need to query a petabyte of data, then you need to rethink your solution. PrestoDB + Hive + Hadoop may be what you need.
so can you put petabytes on 1 server? or can postgresql shard?
Re: Migrating from MongoDB to Cassandra
#49Earlier quoted context omitted.
It's telling that those jira tickets are both in the top five most voted on open server tickets and are from 2010 and before. (I know you know this roger, but) TokuMX completely resolves both of them.
But does TokuMX solve all the other issues like write concurrency, eye wateringly slow (and single threaded) index building, or there being no way to practically reduce the file sizes. (repairDB takes forever and requires the same amount of free space as already used, and compact also needs free space and doesn't remove any files).
I don't know know much about TokuMX besides it being MVCC, which would solve the write concurrency issue nicely.
Re: Migrating from MongoDB to Cassandra
#50Earlier quoted context omitted.
Knowing some of the FC people first hand, MongoDB did actually serve them fairly well for a substantial amount of time. Until they started hitting max limits, it didn't really make sense to move to c . Going straight to C or something like it would have been almost cargo-cultish ( eg If we build industrial strength, we will get industrial levels of traffic ).
Exactly. Can you imagine all of the Java/C developers telling Ruby developers that they are idiots and don't know anything about programming ? Simply because they choose a technology that is designed for developer productivity at the expense of scalability. Because that what seems to happen for every database discussion.
The better comparison would be to PHP, and indeed, no one should ever use tools of quality as poor as PHP or MongoDB.