Live data from Hacker News

Bye Bye Mongo, Hello Postgres

theguardian.com

151–160 of 427 posts

Re: Bye Bye Mongo, Hello Postgres

#151
post #68
post #30

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineer…

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also…

Not really, learning SQL is not harder than learning all the mongo query language, maybe even simpler. Sometimes you just want to try something new, especially in startups where the "best tool for the job" haven't revealed itself yet. Then you are just continuing with whatever you got because migration is too tiresome and it is kind of working. Then you know both SQL and mongo but you got a very extensive boilerplate ready for your next project so use mongo again. Until you got enough time to play with a relational again and you either migrate or build a new project based on it. Then you write a blog.

Re: Bye Bye Mongo, Hello Postgres

#152
post #124

Earlier quoted context omitted.

With a GIN index and a JSONB column you can index the entire json document. It's the same index type as Postgres uses for the full text search, but not super efficient. You can also project out specific subsets of the document to an index and Postgres will use them.

I take it this is Postgres specific and MySQL can't index JSON items.

I'm not familiar with MySQL but the features I outlined are Postgres specific.

Re: Bye Bye Mongo, Hello Postgres

#153
post #133
post #4

Earlier quoted context omitted.

It looks like they wanted to run Mongo inside their own Amazon account- under their direct control. AWS could do that with Postgres while MongoDB couldn't.

What's wrong with running a EC2 instance with Mongo in it?

Nothing, but they outline their problems in the article

Re: Bye Bye Mongo, Hello Postgres

#154

Their architectural choices are puzzling to me: 1) Why use Scala to write (a relatively simple) internal CMS? 2) Why use a clustered database for 2 million records? 3) Why write your own proxy? (in Akka, none the less) 4) Why would you migrate articles from Mongo to Postgres using a script that runs overnight in screen? The Guardian is, prima facie, a Wordpress blog. A simpler architecture would be: 1) Any CRUD web f…

yes and no.

I spent 4 years at a large financial news company, where we benefited from many ex graunaids who decided to migrate over the river.

They helped us create the new front end to the website, to much acclaim. However, it was hard for a number of reasons(this is from the financial news company, not the gruan.):

1) The journalists hated change, especially as they couldn't see any benefit. They just want to keep their same interface exactly as it is, bugs and all. They also had an active union.

2) There is 20 years of "micro services" moving data from the CMS, through various things to allow stuff like translations, syndications (very important source of money) data extraction, meta data processing, physical page layout, and many many more. Most of which is done by a legacy ETL framework pushing to and from a solaris FTP server that is old enough to join the army.

3) there is more than one way to enter data into the CMS.

4) The type of article, and the data in said article changed depending on where it came from, and what services nobbled it.

5) looking after the journalist's interface, curating the data, sorting the articles and adding meta data, looking after paying subscribers, and finally the front end, were all different departments that refused to talk to each other.

This meant that unlike a rational place, there was no source of truth for the CMS. It wasn't like you could call up article 342923 and display it. There was no guarantee that it would have all of the metadata (like were we allows to publish it) required. Add to that the inter department rivalry, which meant that for some reason the membership department were allowed to spend 4 years re-writing the same bit of functionality over and over again. (user management and payment gateways is a solved issue, but alas it took the best part of 25 million quid to find that out.)

To answer your questions:

1) because it scales maaaaan, looks good on my CV, I don't want to spend time doing boring work, I want to learn a new tool

2) see 1

3) see 1

4) Because I suspect that they've never seen a working ETL system

To answer your bonus questions:

1) Journalists have unions, changing the editor requires a _boat_ load of training, and is almost never worth it. Buy over build every time. But yes, its just text. However its the metadata that makes it. Whos in the article, whats the subject. Is it a lifestyle piece, does it have photos, who owns the copyright for the photos, is the article syndicatable, can we syndicate this article, who edited it. Etc, etc, etc. The text entry is the easy bit, its the parts that make it a real news paper that are hard.

2) Nope, almost certainly never done like that. The article will be given a UUID, and dumped into the CMS DB. The front page generator system will then dynamically pull out the articles based on parameters given first by the editors, (front page image, leading headline etc) then the related articles might be curated by hand, or by keyword/metadata or user's preference.

Then the advertising and tracking bits have to be injected, which account for 50-70% of the effort.

CDNs now allow a lot of logic to be pushed to the edge. (see https://labs.ft.com/2014/10/caching-user-agent-specific-resp...) which means that its not overly taxing to host a very large website.

Re: Bye Bye Mongo, Hello Postgres

#155
post #79

Earlier quoted context omitted.

MogoDB is proof that with the right strategy, marketing and luck, you really can fake it until you make it. Not that that's really a surprise or was unknown, it's just fairly new to see in the open source ecosystem instead of the enterprise one.

I'd posit its more a matter of maturing a new paradigm. There's a lot more edge cases you have to cover as NoSQL became more popular for production-at-scale. SQL has decades of production maturation, and has wider domain knowledge.

I'm sure there's some of that. But a lot of the early problems were a bit more weighted towards poor engineering in general, IIRC. For example, I seem to recall an early problem was truncating large amounts of data on crash occasionally.

Re: Bye Bye Mongo, Hello Postgres

#156

Earlier quoted context omitted.

You cannot be old-school and hype. That's the reason why it is old school. And the people who can't make good database design choices are exactly the kind of people who should be using SQL. Postgres knows how to optimize and plan queries efficiently based on the actual distributions of values in your dataset. These poor choosers should be doing that... by hand? https://www.postgresql.org/docs/11/planner-optimizer.htm…

This is what I'm talking about. You think that some feature that makes performance unpredictable is that important. While it's the last thing you should care about.

Unpredictable performance would mean, you can't predict how long a specific query will take. This is the opposite of how postgres works. If a query is slow, it's predictably slow if the data doesn't change. So what you must mean is, when the data changes, it doesn't behave like you expect.

So again, how do these poor choosers make better decisions in the face of changing data when they have to write all the algorithms by hand, and understand how to scale the data correctly? Seems to me that pretty soon they'd just be writing a very crappy database on top of some keystore.

Re: Bye Bye Mongo, Hello Postgres

#157
post #65
post #44

Earlier quoted context omitted.

> By what metrics can we say the product is one worthy of passing a HN smell test? Common sense and formal education? I'm sorry, I'm aware of how incredibly snarky and arrogant that sounds, but in this case I always struggled to comprehend how MongoDB, or most of "NoSQL" in general, was considered viable to begin with. "Schemaless" just immediately means that instead of the database keeping consistency, you now essen…

> "Schemaless" just immediately means that instead of the database keeping consistency, you now essentially have to do all your type and constraint checking in the application... This is so true. Some component has to maintain integrity of the data. Should it be the application developer? Or the database software team? I know which one I'd choose, and which one is focused on data integrity and not business logic.

> This is so true. Some component has to maintain integrity of the data. Should it be the application developer? Or the database software team?

if you're using partitioned (inherited) tables in Postgres, then it's the application developer, as foreign keys on inherited tables don't really work.

the workaround is triggers.

Re: Bye Bye Mongo, Hello Postgres

#158
post #112

I'm not sure why a database is required at all. Surely the main advantage is to support tables and joins. If you just want a key-value store surely S3 or similar would be simplest. Do they search the data? Surely Cassandra is designed for this. Any ideas?

Cassandra isn't designed for this. Been there, done that.

postgres all the way.

THe FT use(d) cassandra to store their membership details (6 million rows of largely static, read-only well structured data) The cluster was massive (12+nodes in at least two regions, from memory) slow and was impossible to upgrade reliably.

The support from datastax is shite. Backups are not reliable. imports even less so, and you are beta testing the whole system every time you do a point release.

CMSs have highly structured data. swallow your pride, map the data and build a proper schema. Its really not that hard.

Yes, cassandra has a graph layer, no its really not worth it. Yes it has gremlin, no you shouldn't need it if you've modelled your data correctly.

Re: Bye Bye Mongo, Hello Postgres

#159
post #143

Earlier quoted context omitted.

Absolutely - if you don't know SQL and you do know JSON, postgres looks scary and Mongo looks familiar.

You make it sound like learning SQL is like learning Assembler. It's not that hard. And ORMs exist in every language to abstract it all away. PostgreSQL looks scary because it is a swiss army knife. It has a million different features and data structures. MongoDB does only one thing.

> You make it sound like learning SQL is like learning Assembler

It's not that learning SQL is hard. It's that people are inherently lazy. "Learn another thing on top of the thing it already took me a couple of years to learn? No thanks."

You seem like the kind of person ready and willing to learn the right tool for the job. From my experience a few years ago on an accredit computing course that covered database admin and programming, this attitude is not representative of most of the software engineering students //unless// there's a specific assignment that requires particular knowledge.

Cs get degrees. And for plenty of developers out there, knowing one language (not even particularly well) gets jobs.

Re: Bye Bye Mongo, Hello Postgres

#160

It seems like the use case of a news agency would be better served by storing metadata in a RDMS and using something like HDFS or other file system for the actual articles.

the articles are trivially small. the 2 million articles could just about fit into RAM nowadays.
Post reply on HN