Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bye Bye Mongo, Hello Postgres

theguardian.com

Post reply on HN
Print view View original

261–270 of 427 posts

Re: Bye Bye Mongo, Hello Postgres

#261

Post by wftglf »

konschubert wrote: post #257
revel wrote: post #177

Earlier quoted context omitted.

Totally agreed. This is pretty much the definitive guide on how to perform a high stakes migration where downtime is absolutely unacceptable. It's extremely tempting, particularly for startups, to simply have a big-bang migration where an old system gets replaced by something else in one shot. I have never, ever seen that approach work out well. The Guardian approach is certainly conservative but it's hard to read th…

I agree, but it looked them a year if I am reading the article right. In most early stage startups, that would be an unacceptable loss of time. So I don't judge them for doing a one-shot migration even if it causes an hour of downtime. It all depends on the business.

Yeah it did take a long time! Part of this though was due to people moving on/off the project a fair bit as other more pressing business needs took priority. We sort of justified the cost due to the expected cost savings from not paying for OpsManager/Mongo support (as in the RDS world support became 'free' as we were already paying for AWS support) - which took the pressure off a bit.

Another team at the guardian did a similar migration but went for a 'bit by bit' approach - so migrating a few bits of the API at a time - which worked out faster, in part because stuff was tested in production more quickly, rather than our approach with the proxy which, whilst imitating production traffic, didn't actually serve Postgres data to the users until 'the big switch' - so not really a continuous delivery migration!

Re: Bye Bye Mongo, Hello Postgres

#262

Post by quietbritishjim »

aidos wrote: post #226
quietbritishjim wrote: post #210

Earlier quoted context omitted.

> IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. The thing I dislike about this type of comment – although I now notice yours doesn't explicitly say this – is the implication that devs don't like SQL because they're lazy or stupid. Well, sometimes that is probably true! But there are some tasks where you need to build the query dynamically…

I’ll try to avoid a flame war, but since you’re using python, SQLAlchemy allows for composing sql strings.

For it to replace MongoDB's aggregation pipeline, it would need to play nicely with JSONB. Does it do that? This is the thing I'm really missing.

For example, if documents in the JSONB column all look roughly like this:

    {
        "someArrayField": [
            { "key": "steve", "value": 7 },
            { "key": "bob", "value": 15 },
        ],
        "someOtherField": [ "whatever" ]
    }
* Can I count the number of entries in someArrayField, summed across all records?

* Can I get the per-record mean of the "value" sub-field, summed across all records?

* Can I filter by records that have a "someArrayField" entry where "key" is "steve" and "value" is at least 10 (the above record should NOT match)?

Re: Bye Bye Mongo, Hello Postgres

#263

Post by wftglf »

lykr0n wrote: post #45
flatiron wrote: post #9

Earlier quoted context omitted.

they talked about NTP. which to me means Mongo on EC2 maybe?

That's what I understood they were doing and they didn't want to do that anymore.

Yep that's what we were doing, and the management software (OpsManager) was also running on EC2 instances. We messed up the VPC configuration so that NTP didn't work on some of the instances - which unsurprisingly broke authentication between OpsManager and the db instances

Re: Bye Bye Mongo, Hello Postgres

#264

Post by kstrauser »

dsirola wrote: post #214
indigodaddy wrote: post #170

Earlier quoted context omitted.

Point 5. I'm not a software/application person... can you expound on how else you would interact with a DB if not directly?

I'm pretty sure he wants to say that you should abstract your database away so that your business code / domain model doesn't depend on it. It becomes a "plugin" to your application and you can easily switch it just by writing another implementation.

I couldn't disagree more, at least in the context of this article. If you have an abstraction layer so high level that app developers can't tell whether they're using MongoDB or PostgreSQL, then they're not able to use any of the advantages of either of those systems.

Sure, use an ORM to abstract away the differences between PostgreSQL and MySQL (up until you need to care about them). That's reasonable. But maintaining a magical MaybeSQL layer that's powerful enough to not totally suck is going to totally suck.

Re: Bye Bye Mongo, Hello Postgres

#265

Post by latch »

Thaxll wrote: post #225

It's not a great article tbh, it's well written but it shows the clear lack of knowledge running a backend. The title should be "we didn't know what we were doing so we switched to a managed DB" I mean yeah who knew that blocking NTP therefore time drifting would break everything... For those criticizing MongoDB, Fortnite generates $3B/year and runs on MongoDB, you should tell them it's a mistake and that they should…

I don't usually bite for these "X uses Y, so Y must be good", but I didn't know about Fortnite and MongoDB. A quick google suggest they've had downtime due to issues with Mongo and have had problems scaling it though.

Re: Bye Bye Mongo, Hello Postgres

#267

Post by rodelrod »

kibwen wrote: post #200
mulmen wrote: post #183

Earlier quoted context omitted.

Saying "I don't know SQL so I will just use JSON" really misses the point though. SQL is easy. Data is hard. NoSQL products offer to get rid of SQL which includes an implication that SQL itself was the challenge in the first place. The problem then is that you have lost one of the best tools for working with data.

I dunno that SQL is exactly easy, though. It's one thing to say "select statements are essentially identical to Python list comprehensions", but in practice I still have to look up the Venn diagram chart every time I need to join anything, and performance optimization is still a dark art. I'd say SQL is easy in the same way that Git is easy: you can get away with using just 5% of it, but you'll still need to consult…

You could solve that by altogether dropping the Venn diagram metaphor when reasoning about joins. This is the number one problem I see with junior devs who have a hard time grokking SQL. If you think about a join as a cartesian product with a filter, where the type of join defines the type of filter, the reasoning is extremely easy.

Here's a good article about that: https://blog.jooq.org/2016/07/05/say-no-to-venn-diagrams-whe...

Re: Bye Bye Mongo, Hello Postgres

#268

Post by kstrauser »

StreamBright wrote: post #204
bdcravens wrote: post #181

Earlier quoted context omitted.

> It is kind of weird why engineers are so obsessed with tools though. Would you say the same thing about programming languages? I was actually listening to the Full Stack Radio podcast, and the latest episode talked about the power of moving more into the database. What struck me as a strong idea was that we can't treat databases as equivalent - different database servers have different strengths (for example, some…

Look, how I see these things is that we have tools. Do you think that people working on fields would fight over which tractor brand is better? Probably not. Do you thinks that builders would fight over which hammer brand is better. I don't think so. These fights over minor differences are happening mostly in the software industry. Freud wrote a pretty good thesis about it. https://en.wikipedia.org/wiki/Narcissism_of_…

> Do you think that people working on fields would fight over which tractor brand is better?

OMG yes. Having worked in the Midwest, I promise you that farmers absolutely do trash talk each other about John Deere vs New Holland vs International Harvester. Tell a Ford driver that you like his Chevy pickup and prepare to hear about it for the next hour.

> Do you thinks that builders would fight over which hammer brand is better.

You seriously haven't spent much time around blue collar workers, have you.

Re: Bye Bye Mongo, Hello Postgres

#269

Post by AlexCoventry »

wftglf wrote: post #242
viraptor wrote: post #103

Earlier quoted context omitted.

It's not their main platform though: > Digital Blog > A blog by the Guardian's internal Digital team. We build the Guardian website, mobile apps, Editorial tools, revenue products, support our infrastructure and manage all things tech around the Guardian

Hi! Thanks for your comments. I'm one of the authors of this post. It is the same platform at the moment (just not tagged with editorial tags so it stays away from the fronts), though sometimes the team that approves non-editorial posts to the site can be concerned about us writing about outages and things as it might carry a 'reputational risk', so we may end up migrating to a different platform in the future so we…

Does SecureDrop run on your AWS infrastructure?

Re: Bye Bye Mongo, Hello Postgres

#270

Post by inferiorhuman »

csytan wrote: post #68
nemild wrote: post #30

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineer…

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also…

> I think you're asking the wrong question. The question should be: How did MongoDB become so successful?

Marketing, marketing, and more marketing. Mongo was written by a couple of adtech guys.

> Not only that, their marketing/outreach efforts were also aimed at younger developers. When was the last time you saw a Postgres rep at a college tech event?

I remember being underwhelmed by two things at the one MongoConf I went to earlier this decade:

1.) My immediate boss was an unfathomable creep who was there mostly to pick up women

2.) Mongo was focused on how to work around the problems (e.g. aggregate framework) rather than how to solve them.

I can't recall ever seeing a Postgres rep, but I can recall having worked out a PostGIS bug with a fantastically tight feedback loop. The Postgres documentation and community are nothing short of amazing.

Meanwhile with Mongo I watched as jawdropping bugs languished. IDGAF what the reps say, anyone with even a few years experience should've been able to see through the bullshit that Mongo/10gen was/is selling.

Post reply on HN