Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“MongoDB is dead. Long live Postgresql”

github.com

Post reply on HN
Print view View original

111–120 of 160 posts

Re: “MongoDB is dead. Long live Postgresql”

#111

Post by lucian1900 »

remon wrote: post #107
lucian1900 wrote: post #100

Earlier quoted context omitted.

1. That's not necessarily the only possible implementation. It would be trivial to assign a number to each key and keep this map in the header of the db file. 2. That's not really the issue, I don't care about the size of small dbs. Large dbs have gigantic sizes. 3. That is absolutely abysmal, yes. It doesn't just use 2x as much space as other dbs, in practice that can be up to 20-30x as much in bad cases. It's commo…

1) Not trivial at all. As documented in a rather broad range of papers and whatnot. You're almost certainly oversimplifying the problem. 2) For gigantic DBs the preallocation overhead is almost non-existent. 3) Fair enough. Your last point is, again, is not fact. It's not slower but on average measurably faster than most commonly used RDBMs with equal "tuning" efforts if utilized for the same task. I can't help but f…

1) I don't think I am. The problem of (structural) subtyping is well known and researched and applying it to a database would not break new ground either.

2) The preallocation overhead is not what is the problem. The files are simply extremely large for the amount of data in them.

There has been plenty of measurement and I have done some of my own. With unacknowledged writes it's quite fast, but no other database client I know of does that because it's stupid. With the highest durability and safety mode it's slower than at least Postgres and Mongo still doesn't have transactions or generic indexes.

Re: “MongoDB is dead. Long live Postgresql”

#112

Post by cullenking »

rbranson wrote: post #66
cullenking wrote: post #44

Maybe I am just incredibly lucky, but mongodb has worked fine for ridewithgps.com - we are sitting at 670gb of data in mongo (actual DB size, indexes included) and haven't had a problem. Replica sets have been fantastic, I wish there was another DB out there that did auto-failover as cleanly/easily as mongo does. We've had a few server crashes of our primary, and aside from 1-2 seconds or so of errors as requests com…

How many reads do you do on that server? 192GB + 8 SSDs is a pretty serious setup. Just the disks themselves should be able to push 500K+ random IOPS.

It's actually very much over built. I had the budget, so I built for growth. This setup should easily last us through this year, assuming 4x growth over last year. I am actually putting together another similarly spec'd machine with 384gb of ram (why not, it's cheap) because the current secondary is on 15k disks with only 24gb of ram. Still more than enough to handle the load through this last year, but probably not this coming year, at least without a bunch more ram.

In regards to actual iops, not sure what this thing can peak at off the top of my head, but we'll easily be doing 100 queries a second this year, with a considerable portion of those queries pulling out ~1mb documents.

Playing it conservative, so I am moving towards gzipping those large documents (never need to access anything but the full data, > 90% of accesses are directly served to clients that can handle inflating the data). For now they will stay in mongo, but I am building out an evaluation of using a flat file structure and just letting nginx pass them out.

Re: “MongoDB is dead. Long live Postgresql”

#113

Post by cullenking »

jlouis wrote: post #110
cullenking wrote: post #44

Maybe I am just incredibly lucky, but mongodb has worked fine for ridewithgps.com - we are sitting at 670gb of data in mongo (actual DB size, indexes included) and haven't had a problem. Replica sets have been fantastic, I wish there was another DB out there that did auto-failover as cleanly/easily as mongo does. We've had a few server crashes of our primary, and aside from 1-2 seconds or so of errors as requests com…

670 gigabytes is a puny database size. You should be able to press so much power through a system with a disk system like the one you have. I would seriously consider a Postgres setup on a data set of that size. Additionally, I would probably just store the JSON data directly inside postgresql.

See comment below. Definitely a small database size. You hit the nail on the head. It's going to grow fast this next year, and I'd like to put off sharding as long as possible, hence gzipped storage of the data.

postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc). However I'll be storing a point reduced version of the full track in postgis, so i can move to using actual intersection queries for matching tracks, instead of the current brute force approach (check everypoint in every track sharing a bounding box) that works now. All bets are out the window though with 2-4x the traffic and data we currently have, using that brute force approach.

I already run another beefy postgis setup (192gb ram, though spinning disks not SSDs) for serving OSM maps, and eventually OSM routing hence the ram.

Re: “MongoDB is dead. Long live Postgresql”

#114

Post by cullenking »

blablabla123 wrote: post #53
cullenking wrote: post #44

Maybe I am just incredibly lucky, but mongodb has worked fine for ridewithgps.com - we are sitting at 670gb of data in mongo (actual DB size, indexes included) and haven't had a problem. Replica sets have been fantastic, I wish there was another DB out there that did auto-failover as cleanly/easily as mongo does. We've had a few server crashes of our primary, and aside from 1-2 seconds or so of errors as requests com…

>data that isn't updated frequently How often did you update your data then? In my current project I am seeing locking issues in my way soon...

I think I peaked at 5% lock utilization this year, so I haven't seen any real issues.

Our actual track data isn't updated frequently. Mostly it serves as an archive for a user, and is only seen by 1-2 other people. Most people use our service to store all their activities, which for the most part are really boring. They are interested in aggregate metrics like "I've ridden 200 miles this month".

A smaller portion of our data is from planning a route using google maps, which has much more modest storage requirements, since it's optimized data (one point every mile if it's a straight line) instead of 1hz logging from a GPS unit. This stuff is edited, but I'd say only 10% of planned routes are ever modified, so actual updates on the track data are small.

Re: “MongoDB is dead. Long live Postgresql”

#115

Post by rdtsc »

functional_test wrote: post #19

Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one or more of: * didn't read the manual * poor schema * didn't maintain the database (compactions, etc.) In this case, they hit several: " Its volume on disk is growing 3-4 times faster than the real volume of data it store;" They should be doing compactions and…

> Seriously, another case of using Mongo incorrectly?

If everyone uses Mongo incorrectly, the problem is not Mongo. It is like the person crying out how everyone in the world is crazy.

Re: “MongoDB is dead. Long live Postgresql”

#116

Post by weddpros »

cullenking wrote: post #112
rbranson wrote: post #66

Earlier quoted context omitted.

How many reads do you do on that server? 192GB + 8 SSDs is a pretty serious setup. Just the disks themselves should be able to push 500K+ random IOPS.

It's actually very much over built. I had the budget, so I built for growth. This setup should easily last us through this year, assuming 4x growth over last year. I am actually putting together another similarly spec'd machine with 384gb of ram (why not, it's cheap) because the current secondary is on 15k disks with only 24gb of ram. Still more than enough to handle the load through this last year, but probably not…

"just letting nginx pass them out": proxy cache could work as well... Volume of data is much less here, but I let nginx cache gridFS stored photos...

Re: “MongoDB is dead. Long live Postgresql”

#117

Post by rdtsc »

remon wrote: post #63
lucian1900 wrote: post #25

Earlier quoted context omitted.

Mongo's disk format is extremely wasteful, the database files are gigantic. That is a real problem and there is no way to compact this to anywhere near the size something like Postgres would have for the same data. Mongo is very bad at managing used memory. In fact it doesn't actually manage memory since it just mmaps its database file. It also touches disk much more often than would be reasonable, especially for how…

Although it is true that MongoDB uses a lot of disk compared to your average RDMS there are reasons for that. 1) MongoDB (and various other NoSQL solution) are schemaless and thus have to store document fields along with the values for each document. This alone usually results in roughly twice as much actual disk space being used compared to an RDBMS. 2) MongoDB preallocates fairly large chunks of disk for their mmap…

> "it's terrible booo!" nonsense has to stop.

Nope. Pedal to the metal with that I say. They marketed the shit out of their database. Up until 2 years ago shipped with unacknowledged writes as a default. There is no excuse for that.

Don't want negative publicity -- stop making unrealistic marketing statements. That is the problem with MongoDB -- vis-a-vis their marketing, their underlying technology sucks. It is rather relative you see.

All those things you highlighted + compaction not being a background process, just mmapping a 2GB file for each DB. A global write lock. The unacknowledged writes problem, sorry, to me that screams "we don't know what we are doing, please don't let us near your data"

Re: “MongoDB is dead. Long live Postgresql”

#118

Post by functional_test »

jeffdavis wrote: post #43
functional_test wrote: post #19

Seriously, another case of using Mongo incorrectly? I want to believe all the Mongo hate, but I can't because I always find out that the actual problem was one or more of: * didn't read the manual * poor schema * didn't maintain the database (compactions, etc.) In this case, they hit several: " Its volume on disk is growing 3-4 times faster than the real volume of data it store;" They should be doing compactions and…

> They should be doing compactions and are not. https://jira.mongodb.org/browse/SERVER-11763 It looks like compaction is an offline process. That really puts the user between a rock and a hard place.

In a proper production environment, you just compact each slave one at a time because you have a replica set rather than a single instance.

Of course, if you aren't replicating your business's production database, you have a whole world of problems.

Re: “MongoDB is dead. Long live Postgresql”

#119

Post by raverbashing »

macspoofing wrote: post #103
threeseed wrote: post #86

Earlier quoted context omitted.

It has nothing to do with the defaults. It is all about people forgetting that MongoDB is a document database and not a relational one. I can write apps that will be 10x faster with MongoDB and 10x faster with PostgreSQL. It's all about matching your domain model to your database.

>t is all about people forgetting that MongoDB is a document database and not a relational one. It's been a while since I looked into Mongo, but that was a Mongo marketing problem. They used to (still?) advertised themselves as a RDBMS replacement, literally.

Yes, but it's more complex than for example, replacing MySQL with PostgreSQL, where the basic structure is going to be the same.

You have to think and adapt to convert from a RDBS to MongoDB

Re: “MongoDB is dead. Long live Postgresql”

#120

Post by cullenking »

rbranson wrote: post #66
cullenking wrote: post #44

Maybe I am just incredibly lucky, but mongodb has worked fine for ridewithgps.com - we are sitting at 670gb of data in mongo (actual DB size, indexes included) and haven't had a problem. Replica sets have been fantastic, I wish there was another DB out there that did auto-failover as cleanly/easily as mongo does. We've had a few server crashes of our primary, and aside from 1-2 seconds or so of errors as requests com…

How many reads do you do on that server? 192GB + 8 SSDs is a pretty serious setup. Just the disks themselves should be able to push 500K+ random IOPS.

One thing to note that I left out above, I also use the same severs for mysql. Our mysql working set is something like 30gb now, so I have a decent chunk of that ram apportioned to mysql.

Additionally our mysql db sees many more queries than mongo, so the overbuilt hardware is a bit less overbuilt when taking that into consideration :)

Post reply on HN