Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bye Bye Mongo, Hello Postgres

theguardian.com

Post reply on HN
Print view View original

411–420 of 427 posts

Re: Bye Bye Mongo, Hello Postgres

#411

Post by magduf »

zozbot123 wrote: post #410
magduf wrote: post #408

Earlier quoted context omitted.

Yes, you're missing something pretty big. It's not just media companies rewriting something significant every few years; everyone in the tech industry does it. There's a very good reason for it: it provides interesting and high-paying work for tech employees. Who wants to go to a company, set up a great system that doesn't need any more work except maintenance, and then just settle down into maintenance mode? No one…

> interesting and high-paying work Busy-work is never interesting, and if it's "high-paying" someone is getting fleeced (and will do what they can so as not to get fleeced in the future). At some point, the constant rewriting will have to stop. In contrast, maintenance work on a well-engineered critical system can be quite interesting in its own right (and it's not like the "maintenance" doesn't involve writing some…

Taking part in a big new project to switch to something else can certainly be interesting, if it's a new technology, and even if it's really no better than the previous technology. If nothing else, it's really good for your resume.

Just to expand on this: if I were to take part in a project to move a critical service from Postgres to MongoDB (which I believe to be a worse solution), this would still be a better learning experience for me than just maintaining an existing installation. I'd have to learn about both DBs, and write a lot of new code to work with the new DB. That wouldn't happen with a Postgres system that's already set up and working fine. And my resume wouldn't look like someone that can come in and get to work on a big project and see it through by working on a maintenance project.

>At some point, the constant rewriting will have to stop.

At that point, the developers and managers who pushed for the big new project will have moved on to another company. Remember, moving companies every few years will generally yield a much higher salary than staying at the same company for your entire career.

>In contrast, maintenance work on a well-engineered critical system can be quite interesting in its own right

Sure, but you're not going to get a great salary doing that.

Re: Bye Bye Mongo, Hello Postgres

#412

Post by SllX »

chrisoverzero wrote: post #299
SllX wrote: post #70

Earlier quoted context omitted.

This actually resonates with me, maybe not in the way Ellison intended as I’m not familiar with the context he said it in. A bit off the main topic but the more I revert to just using emacs for some task I previously used a .app bundle or web page for, the more I question how much we the computer industry has just been spinning its wheels for the last 30+ years. I honestly can’t really tell what value WIMP-centric GU…

> I honestly can’t really tell what value WIMP-centric GUIs have brought to the table besides fashion […] Literal billions of users.

Billions of users of which things that are worthwhile?

Re: Bye Bye Mongo, Hello Postgres

#413

Post by pfriend »

wftglf wrote: post #242
viraptor wrote: post #103

Earlier quoted context omitted.

It's not their main platform though: > Digital Blog > A blog by the Guardian's internal Digital team. We build the Guardian website, mobile apps, Editorial tools, revenue products, support our infrastructure and manage all things tech around the Guardian

Hi! Thanks for your comments. I'm one of the authors of this post. It is the same platform at the moment (just not tagged with editorial tags so it stays away from the fronts), though sometimes the team that approves non-editorial posts to the site can be concerned about us writing about outages and things as it might carry a 'reputational risk', so we may end up migrating to a different platform in the future so we…

Thanks for writing the blog post. Insights like these and of such high quality are rare.

Can I ask what was the total cost of the migration?

If there was software that could do this database migration without downtime, how much would you/Guardian be willing to pay?

Re: Bye Bye Mongo, Hello Postgres

#414

Post by TimFogarty »

mukundmr wrote: post #360
idbentley wrote: post #161

I really don't get this as an indictment of MongoDB, or their OpsManager product really. They used the version of OpsManager that doesn't manage the deployment - is specifically not a deployment manager. Mongo does offer a managed version of this software, which the author mentions - with a justification for why they couldn't use that offering. However, I think this was the main mistake that The Guardian made. As the…

The pricing of hosted MongoDB solutions is high especially as volumes increase. The last I checked, disk space doesn't free up when you drop documents or collections and that adds to hosted cost unless you find time to fix the issue through manual intervention. This costing is moving us away from MongoDB in the future.

MongoDB will reuse free space when documents or collections are deleted (even though the storageSize stat will remain the same). You can compact the database to release free space to the OS. You can read more about how MongoDB's WiredTiger storage engine reclaims space here: https://docs.mongodb.com/manual/faq/storage/#how-do-i-reclai...

I work for MongoDB so if you have any questions about storage, feel free to reach out.

Re: Bye Bye Mongo, Hello Postgres

#415

Post by pas »

darkarmani wrote: post #390
pas wrote: post #371

Earlier quoted context omitted.

hipsters are allegedly the cool old-schoolers. the whole promise of mongo is/was distributed (HA+LB), which was all the rage back then, when AWS AZs dropped like flies every few weeks and scaling was seen as the problem. go fast, break things was the mantra. and it's still not trivial to do pgsql maintenance without downtime, whereas in a clustered/distributed "solution", you can enjoy certain additional freedoms. th…

> the whole promise of mongo is/was distributed (HA+LB) Yes. It is so easy to bring up a mongo cluster and feel like you have HA. Don't worry it won't be proven wrong until you have writes during your cluster degradation and are unlucky.

it works. 4.0 gives you transactions.

it's pretty great.

and, most importantly, as far as I know the default replica set config is safe. (durable, consistent, atomic) it handles a lot of failover scenarios for you automatically. and you have to manually intervene to get into a bad state - which might be okay for that particular business case (better than full downtime)

Re: Bye Bye Mongo, Hello Postgres

#416

Post by zorga »

int_19h wrote: post #406
zorga wrote: post #403

Earlier quoted context omitted.

That objection doesn't make sense. Queries are nothing more than a tree of predicates, how the backend end uses those predicates is not relevant to the API of specifying the predicates. Things like indexes whether in Mongo or in SQL are implementation details that can easily be hidden and not infect the API. You can interpret the tree of predicates into a SQL where clause or into a Mongo index search. The OP is corre…

This is all rosy in theory. In practice, the way you write the query matters quite a bit. Often even between different SQL implementations. And it's not just queries. Transactions often have important semantic differences that will be visible on application layer - again, even between different SQL implementations (e.g. MVCC vs locks).

> In practice, the way you write the query matters quite a bit.

Which is hidden in the query interpreter for said db implementation. Each implementation can break down that abstract query into whatever implementation specific query works best in that database.

There's always some abstract way to represent it that doesn't require vendor specific knowledge nor does it remove the ability to apply vendor specific abilities.

Look, I just don't agree with you, I agree with OP. Db specific stuff should be hidden from the domain layer by an abstract query representation and an abstract transaction representation to be plugged in at a later time.

Re: Bye Bye Mongo, Hello Postgres

#417

Post by fujimotos »

setquk wrote: post #203
KaiserPro wrote: post #169

Earlier quoted context omitted.

At "large financial news company" we had a "designed for the CV" tag that applied to stupid architectural decisions (of which there were many) One of the biggest and most expensive was using Cassandra to store membership details. Something like 4 years of work, by a team of 40, wasted by stupid decisions. They included: o Using Cassandra to store 6 million rows of highly structured, mostly readonly data o hosting it…

I think the worst place I ever worked was like that. Was going back quite a number of years now but it was a startup fired up by a one of the lesser MBAs to utilise a legal loophole to slice cash off the public via a web app and a metric ton of marketing to desperate people. Step one was hire everyone the guy had worked with at his previous company. They were all winforms / excel / SQL / sharepoint / office developer…

I once saw a CEO who wrote his own Web Framework and forced the entire company to use it.

At the time, under the influence of React, the idea was to "build web application sorely based on Functional Programming". Since after years of trying no one could figure out what that meant, the company ditched the CEO and ended up wasting a couple years of work.

Re: Bye Bye Mongo, Hello Postgres

#418

Post by int_19h »

zorga wrote: post #416
int_19h wrote: post #406

Earlier quoted context omitted.

This is all rosy in theory. In practice, the way you write the query matters quite a bit. Often even between different SQL implementations. And it's not just queries. Transactions often have important semantic differences that will be visible on application layer - again, even between different SQL implementations (e.g. MVCC vs locks).

> In practice, the way you write the query matters quite a bit. Which is hidden in the query interpreter for said db implementation. Each implementation can break down that abstract query into whatever implementation specific query works best in that database. There's always some abstract way to represent it that doesn't require vendor specific knowledge nor does it remove the ability to apply vendor specific abiliti…

Have you ever implemented an ORM? I did.

Stuff like "each implementation can break down that abstract query into whatever implementation specific query works best in that database" is wishful thinking. It's like saying that Java is faster than C++, in theory, because JIT can produce better code. And in theory, it can. In practice, we're not there yet. Same thing with high-level database abstractions - they're all either leaky in subtle ways, or they constrain you to extremely basic operations that can be automatically implemented efficiently on everything (but e.g. forget joins).

Re: Bye Bye Mongo, Hello Postgres

#419

Post by zorga »

int_19h wrote: post #418
zorga wrote: post #416

Earlier quoted context omitted.

> In practice, the way you write the query matters quite a bit. Which is hidden in the query interpreter for said db implementation. Each implementation can break down that abstract query into whatever implementation specific query works best in that database. There's always some abstract way to represent it that doesn't require vendor specific knowledge nor does it remove the ability to apply vendor specific abiliti…

Have you ever implemented an ORM? I did. Stuff like "each implementation can break down that abstract query into whatever implementation specific query works best in that database" is wishful thinking. It's like saying that Java is faster than C++, in theory, because JIT can produce better code. And in theory, it can. In practice, we're not there yet. Same thing with high-level database abstractions - they're all eit…

> Have you ever implemented an ORM? I did.

Several actually, which is why I know what I'm talking about; I've explored this area extensively. When Fowler first released PEAA I dug and went nuts and spent years coding up and exploring all the possible approaches and figuring out which ones I liked and why and which ones I didn't and why.

> Same thing with high-level database abstractions - they're all either leaky in subtle ways, or they constrain you to extremely basic operations that can be automatically implemented efficiently on everything (but e.g. forget joins).

If you're doing joins in your ORM, frankly, you're doing it wrong. Most ORM's do it wrong, they try and replace what a db does best; the right way to do it is to keep joins in the db. The role of an ORM when used properly is to map tables and views into objects and allow querying over those tables and views with an abstract query syntax. Joins belong in a view, not in code. It's called the object relational impedance mismatch for a reason, you have to draw a line in a reasonable place to get anything reasonable to work well and putting joins into the ORM is crossing that line and is why most ORM's utterly suck. Joins aren't queries, they're projections; put the queries in the code and the projections into the database, this works perfectly and lets each side do what it does best. Queries are easily abstracted, projections are not, projections don't belong in the ORM.

Re: Bye Bye Mongo, Hello Postgres

#420

Post by int_19h »

zorga wrote: post #419
int_19h wrote: post #418

Earlier quoted context omitted.

Have you ever implemented an ORM? I did. Stuff like "each implementation can break down that abstract query into whatever implementation specific query works best in that database" is wishful thinking. It's like saying that Java is faster than C++, in theory, because JIT can produce better code. And in theory, it can. In practice, we're not there yet. Same thing with high-level database abstractions - they're all eit…

> Have you ever implemented an ORM? I did. Several actually, which is why I know what I'm talking about; I've explored this area extensively. When Fowler first released PEAA I dug and went nuts and spent years coding up and exploring all the possible approaches and figuring out which ones I liked and why and which ones I didn't and why. > Same thing with high-level database abstractions - they're all either leaky in…

Any language with named tuples has a type system that is sufficiently expressive to handle joins without any sort of impedance mismatch. So, the only reason to avoid them is exactly the one that I cited earlier - the underlying implementation difference between databases.
Post reply on HN