Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bye Bye Mongo, Hello Postgres

theguardian.com

Post reply on HN
Print view View original

381–390 of 427 posts

Re: Bye Bye Mongo, Hello Postgres

#381

Post by kstrauser »

dsirola wrote: post #331
kstrauser wrote: post #264

Earlier quoted context omitted.

I couldn't disagree more, at least in the context of this article. If you have an abstraction layer so high level that app developers can't tell whether they're using MongoDB or PostgreSQL, then they're not able to use any of the advantages of either of those systems. Sure, use an ORM to abstract away the differences between PostgreSQL and MySQL (up until you need to care about them). That's reasonable. But maintaini…

Why wouldn't you be able to take advantage of those systems? I think it's quite the opposite where you can take the advantages without even knowing about it / affecting other parts of the system.

FedEx needs both big, slow, high-volume trucks for moving stuff between cities, and small, nimble tricks for delivering to the doorstep. Someone decides that it’s inefficient to maintain two separate standards: any driver should be able to get into any available vehicle and have it Just Work for whatever job they have at hand, right? So they decide to make a single vehicle that can fulfill all roles.

Well, a vehicle that can squeeze down an alley won’t have the cargo room of a giant highway truck. The inter-city drivers will hate its poor capacity. Likewise, one that has a big 20-speed transmission for for hauling heavy loads is going to drive the city drivers nuts. They’re going to end up with one single interface to all possible roadways that everyone can come together and agree to hate.

If the database API is so free-form that you can store anything in it, you won’t get the advantages of PostgreSQL’s strict typing and lightning fast joins. If you make it so regimented that your data model ends up looking like a set of tables with foreign keys, then it won’t be able to make full use of MongoDB’s... whatever it does well.

They’re different animals. Choosing one highly affects the rest of your system design, from how you arrange your data to how you add new data to how you search for it. PostgreSQL and MongoDB have fundamentally different strengths and weaknesses, and if you make something that works equally well with both, it’s inherently going to suck equally on either.

Re: Bye Bye Mongo, Hello Postgres

#382

Post by bcoughlan »

csytan wrote: post #68
nemild wrote: post #30

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineer…

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also…

> Not only that, their marketing/outreach efforts were also aimed at younger developers.

I do remember a lot of MongoDB t-shirts, cups and pens around every office I was in around 2011-2013. When I would ask they would tell me that a MongoDB developer flew halfway across the world to give them all a workshop on it.

Re: Bye Bye Mongo, Hello Postgres

#383

Post by Operyl »

latch wrote: post #265
Thaxll wrote: post #225

It's not a great article tbh, it's well written but it shows the clear lack of knowledge running a backend. The title should be "we didn't know what we were doing so we switched to a managed DB" I mean yeah who knew that blocking NTP therefore time drifting would break everything... For those criticizing MongoDB, Fortnite generates $3B/year and runs on MongoDB, you should tell them it's a mistake and that they should…

I don't usually bite for these "X uses Y, so Y must be good", but I didn't know about Fortnite and MongoDB. A quick google suggest they've had downtime due to issues with Mongo and have had problems scaling it though.

Their level of scale is .. amazingly higher up there than most companies using mongo I imagine, and they seem to be doing a good job at it now.

Re: Bye Bye Mongo, Hello Postgres

#384

Post by hu3 »

jasondc wrote: post #232

If you think MongoDB isn't doing well, just look at their financials and stock price, they're on fire: https://finance.yahoo.com/quote/MDB?p=MDB Anecdotes like this are just that, anecdotes, there are no numbers in this article to show a trend away from Mongo, Mongo is actually continuing to gain adoption. See https://db-engines.com/en/ranking

Maybe I'm wrong but your link tells me stock price took a hit today:

> -8.82%% As of 11:56AM EST. Market open.

1 month chart still looks green so this might just be a correction.

Re: Bye Bye Mongo, Hello Postgres

#385

Post by ris »

quietbritishjim wrote: post #262
aidos wrote: post #226

Earlier quoted context omitted.

I’ll try to avoid a flame war, but since you’re using python, SQLAlchemy allows for composing sql strings.

For it to replace MongoDB's aggregation pipeline, it would need to play nicely with JSONB. Does it do that? This is the thing I'm really missing. For example, if documents in the JSONB column all look roughly like this: { "someArrayField": [ { "key": "steve", "value": 7 }, { "key": "bob", "value": 15 }, ], "someOtherField": [ "whatever" ] } * Can I count the number of entries in someArrayField, summed across all reco…

It absolutely can, but in my experience, 99% of the time, choosing to make a data field JSON/JSONB ends up being a mistake.

Re: Bye Bye Mongo, Hello Postgres

#386

Post by ris »

kibwen wrote: post #200
mulmen wrote: post #183

Earlier quoted context omitted.

Saying "I don't know SQL so I will just use JSON" really misses the point though. SQL is easy. Data is hard. NoSQL products offer to get rid of SQL which includes an implication that SQL itself was the challenge in the first place. The problem then is that you have lost one of the best tools for working with data.

I dunno that SQL is exactly easy, though. It's one thing to say "select statements are essentially identical to Python list comprehensions", but in practice I still have to look up the Venn diagram chart every time I need to join anything, and performance optimization is still a dark art. I'd say SQL is easy in the same way that Git is easy: you can get away with using just 5% of it, but you'll still need to consult…

> performance optimization is still a dark art

The idea is, in relational databases, that the vast majority of the time you shouldn't have to do it. Because you're writing your queries in a higher level (nay, functional) language, the query planner can understand a lot more about what you're trying to do and actually choose algorithms and implementations that are appropriate for the shape and size of your data. And in 6 months time when your tables are ten times the size, it is able to automatically make new decisions.

More explicit forms of expressing queries have no hope of being able to do this and any performance optimization you do is appropriate only for now and this current dataset.

Re: Bye Bye Mongo, Hello Postgres

#387

Post by eecc »

wftglf wrote: post #261
konschubert wrote: post #257

Earlier quoted context omitted.

I agree, but it looked them a year if I am reading the article right. In most early stage startups, that would be an unacceptable loss of time. So I don't judge them for doing a one-shot migration even if it causes an hour of downtime. It all depends on the business.

Yeah it did take a long time! Part of this though was due to people moving on/off the project a fair bit as other more pressing business needs took priority. We sort of justified the cost due to the expected cost savings from not paying for OpsManager/Mongo support (as in the RDS world support became 'free' as we were already paying for AWS support) - which took the pressure off a bit. Another team at the guardian di…

The article mentions several corner cases that weren’t well covered by testing and caused issues later. What sort of test tooling did you use, Scalacheck?

Re: Bye Bye Mongo, Hello Postgres

#388

Post by riku_iki »

user5994461 wrote: post #356
riku_iki wrote: post #307

Earlier quoted context omitted.

in sharded setup each shard contains subset of data, each node doesn't contain all data: https://docs.mongodb.com/manual/core/sharded-cluster-shards/ You are talking about replica sets, which is different concept.

In a sharded setup, every shard must be on a 3 nodes cluster for redundancy (a replica set), the 3 nodes keep the exact same data with only one node accepting reads and writes. You have to add capacity (shards) 3 nodes at a time, two third of which sit unused. It's not scalable at all.

Its scalable in terms that if your total data doesn't fit into single machine, you can distribute it between shards in mongo, but can't in pgsql.

replica set is for redundancy and availability, but you can use replicas for reads, so scale your reads traffic. pgsql works absolutely the same way, you have one master which accepts writes and read-only slaves/replicas.

Re: Bye Bye Mongo, Hello Postgres

#389

Post by jstarfish »

StreamBright wrote: post #122
mattbillenstein wrote: post #119

Earlier quoted context omitted.

Your architectural choices are puzzling to me - Rails/Django -- I've rescued more bad Django apps than I can count.

Exactly, I have moved companies from random web framework + random database to static site generators + CDN with high rate of success too. No point of using Rails/Django like stuff unless you have an extremely good case to, which is certainly not the Guardian use case.

> No point of using Rails/Django like stuff unless you have an extremely good case to, which is certainly not the Guardian use case.

Django itself was literally developed to suit the use cases of a newspaper.

Re: Bye Bye Mongo, Hello Postgres

#390

Post by darkarmani »

pas wrote: post #371
michelpp wrote: post #76

Earlier quoted context omitted.

You cannot be old-school and hype. That's the reason why it is old school. And the people who can't make good database design choices are exactly the kind of people who should be using SQL. Postgres knows how to optimize and plan queries efficiently based on the actual distributions of values in your dataset. These poor choosers should be doing that... by hand? https://www.postgresql.org/docs/11/planner-optimizer.htm…

hipsters are allegedly the cool old-schoolers. the whole promise of mongo is/was distributed (HA+LB), which was all the rage back then, when AWS AZs dropped like flies every few weeks and scaling was seen as the problem. go fast, break things was the mantra. and it's still not trivial to do pgsql maintenance without downtime, whereas in a clustered/distributed "solution", you can enjoy certain additional freedoms. th…

> the whole promise of mongo is/was distributed (HA+LB)

Yes. It is so easy to bring up a mongo cluster and feel like you have HA. Don't worry it won't be proven wrong until you have writes during your cluster degradation and are unlucky.

Post reply on HN