Bye Bye Mongo, Hello Postgres • Hacker News Board

Re: Bye Bye Mongo, Hello Postgres

#201

Post by rabidrat » Wed, Dec 19, 2018, 10:14 PM UTC

zapzupnz wrote: post #159

threeseed wrote: post #148

Earlier quoted context omitted.

You make it sound like learning SQL is like learning Assembler. It's not that hard. And ORMs exist in every language to abstract it all away. PostgreSQL looks scary because it is a swiss army knife. It has a million different features and data structures. MongoDB does only one thing.

> You make it sound like learning SQL is like learning Assembler It's not that learning SQL is hard. It's that people are inherently lazy. "Learn another thing on top of the thing it already took me a couple of years to learn? No thanks." You seem like the kind of person ready and willing to learn the right tool for the job. From my experience a few years ago on an accredit computing course that covered database admi…

I think what happens (and I have this attitude too) is that "learning" SQL takes a weekend...but then you know you'll wind up having to spend a lot longer learning the patterns of the language, and the nuances of the specific dialect, and which of the integration tools will work well with your workflow and pipeline. So while "sure I'll just learn SQL" is great for a personal or school project, when you've got to get something done next week, it's better to take maximal advantage of the tools/skills/workflow that you already have.

IOW, it's not just laziness, it's a kind of professional conservatism. which is partly what gets older engineers stuck in a particular mindset, but it's also a very effective learned skill. The opposite is being a magpie developer, which results in things like MongoDB taking off :)

Re: Bye Bye Mongo, Hello Postgres

#202

Post by WJW » Wed, Dec 19, 2018, 10:14 PM UTC

acjohnson55 wrote: post #121

csytan wrote: post #68

Earlier quoted context omitted.

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also…

> When was the last time you saw a Postgres rep at a college tech event? Is a Postgres rep a thing?

That's the point.

Re: Bye Bye Mongo, Hello Postgres

#203

Post by setquk » Wed, Dec 19, 2018, 10:16 PM UTC

KaiserPro wrote: post #169

pjmlp wrote: post #42

Earlier quoted context omitted.

Fashion driven development and not understanding how to actually make use of SQL.

At "large financial news company" we had a "designed for the CV" tag that applied to stupid architectural decisions (of which there were many) One of the biggest and most expensive was using Cassandra to store membership details. Something like 4 years of work, by a team of 40, wasted by stupid decisions. They included: o Using Cassandra to store 6 million rows of highly structured, mostly readonly data o hosting it…

I think the worst place I ever worked was like that. Was going back quite a number of years now but it was a startup fired up by a one of the lesser MBAs to utilise a legal loophole to slice cash off the public via a web app and a metric ton of marketing to desperate people.

Step one was hire everyone the guy had worked with at his previous company. They were all winforms / excel / SQL / sharepoint / office developers from big finance and had no idea where to go really. None of them had even touched asp.net.

Cue "what's popular". Well that was Ruby on Rails back then on top of MySQL and Linux. 4 people with zero experience pulled this stack in and basically wrote winforms on top of it. Page hits were 5-8 seconds each. Infrastructure was owned by SSH worms and they hadn't even noticed.

I think I lasted two days there before I said "I'm done".

Re: Bye Bye Mongo, Hello Postgres

#204

Post by StreamBright » Wed, Dec 19, 2018, 10:16 PM UTC

bdcravens wrote: post #181

StreamBright wrote: post #134

Earlier quoted context omitted.

Not sure, but there was Goodbye MySQL, Hello Postgres and Goodbye Postgres, Hello MySQL (same company did it actually) for sure. It is kind of weird why engineers are so obsessed with tools though.

> It is kind of weird why engineers are so obsessed with tools though. Would you say the same thing about programming languages? I was actually listening to the Full Stack Radio podcast, and the latest episode talked about the power of moving more into the database. What struck me as a strong idea was that we can't treat databases as equivalent - different database servers have different strengths (for example, some…

Look, how I see these things is that we have tools. Do you think that people working on fields would fight over which tractor brand is better? Probably not. Do you thinks that builders would fight over which hammer brand is better. I don't think so. These fights over minor differences are happening mostly in the software industry. Freud wrote a pretty good thesis about it.

https://en.wikipedia.org/wiki/Narcissism_of_small_difference...

Moving into databases is not something I understand. I worked the last 10 years as data engineer but I like to move out from databases a lot more than moving into them. :)

Re: Bye Bye Mongo, Hello Postgres

#205

Post by evanelias » Wed, Dec 19, 2018, 10:17 PM UTC

h1d wrote: post #124

sitharus wrote: post #88

Earlier quoted context omitted.

With a GIN index and a JSONB column you can index the entire json document. It's the same index type as Postgres uses for the full text search, but not super efficient. You can also project out specific subsets of the document to an index and Postgres will use them.

I take it this is Postgres specific and MySQL can't index JSON items.

The syntax and terminology are different, but MySQL has supported indexes on values inside of JSON (or any arbitrary expression/function) since MySQL 5.7, released about three years ago. In MySQL, you can define a "generated column" and index it in InnoDB.

https://dev.mysql.com/doc/refman/5.7/en/create-table-seconda... contains some JSON-specific examples.

Re: Bye Bye Mongo, Hello Postgres

#206

Post by beokop » Wed, Dec 19, 2018, 10:18 PM UTC

idbentley wrote: post #161

I really don't get this as an indictment of MongoDB, or their OpsManager product really. They used the version of OpsManager that doesn't manage the deployment - is specifically not a deployment manager. Mongo does offer a managed version of this software, which the author mentions - with a justification for why they couldn't use that offering. However, I think this was the main mistake that The Guardian made. As the…

I do like the article but it sounds like they’re in over their heads, this whole (very risky) project could have been avoided if they just brought in someone that knew what they were doing.

> Clocks are important – don’t lock down your VPC so much that NTP stops working.

> Automatically generating database indexes on application startup is probably a bad idea.

> Database management is important and hard – and we’d rather not be doing it ourselves.

This is true of any database infrastructure with redundancy/scalability requirements.

What they did was take a technical problem and solve it by buying an off the shelf solution. Which is fine, of course, but I’m a bit surprised by the reaction here on HN.

Re: Bye Bye Mongo, Hello Postgres

#207

Post by mulmen » Wed, Dec 19, 2018, 10:19 PM UTC

kibwen wrote: post #200

mulmen wrote: post #183

Earlier quoted context omitted.

Saying "I don't know SQL so I will just use JSON" really misses the point though. SQL is easy. Data is hard. NoSQL products offer to get rid of SQL which includes an implication that SQL itself was the challenge in the first place. The problem then is that you have lost one of the best tools for working with data.

I dunno that SQL is exactly easy, though. It's one thing to say "select statements are essentially identical to Python list comprehensions", but in practice I still have to look up the Venn diagram chart every time I need to join anything, and performance optimization is still a dark art. I'd say SQL is easy in the same way that Git is easy: you can get away with using just 5% of it, but you'll still need to consult…

The hard parts of "SQL" are the hard parts of data. Joins aren't easier in Mongo. The performance optimizations you reference are tuning of a relational database, not SQL itself.

If you want to work with databases a domain specific language like SQL really provides a lot of value in solving these hard data problems.

Re: Bye Bye Mongo, Hello Postgres

#208

Post by mrsuprawsm » Wed, Dec 19, 2018, 10:19 PM UTC

scrollaway wrote: post #90

mrsuprawsm wrote: post #80

>“But postgres isn’t a document store!” I hear you cry. Well, no, it isn’t, but it does have a JSONB column type, with support for indexes on fields within the JSON blob. > approximately 2.3m content items. I had a previous project where we did a similar thing (except with HSTORE instead of JSONB) and it exploded rather dramatically (very simple queries took multiple minutes or timed out entirely) after around 30m ro…

2.3m content items is tiny. So is 30m. You're at least 1-2 magnitudes away from something that will start to bother postgres. Anything before that is likely to be an index or IOPS issue.

Even with HSTORE/JSONB columns? I would imagine (gut feeling, no evidence) that the performance of these is worse than more traditional column types.

Re: Bye Bye Mongo, Hello Postgres

#209

Post by fastftw » Wed, Dec 19, 2018, 10:19 PM UTC

petilon wrote: post #95

Disappointing article. There are no clear reasons mentioned for migrating away from MongoDB. "All these problems" and some issues with Ops Manager are mentioned, where "all these problems" is a couple of outages. As if other database technologies prevent outages. As soon as they experience a couple of outages with their new stack they will migrate to something else, presumably.

Thank you! I was reading through the comments wondering how no one has seen through the bullshit veil purported by the title, followed by a detailed outline of their migration which I think is the "shiny object" to distract people from an unfounded argument.

Re: Bye Bye Mongo, Hello Postgres

#210

Post by quietbritishjim » Wed, Dec 19, 2018, 10:24 PM UTC

csytan wrote: post #68

nemild wrote: post #30

The Guardian example was heavily used by MongoDB as a case study to pitch their database to others in 2011: https://www.mongodb.com/customers/guardian https://www.mongodb.com/presentations/mongodb-guardian https://www.slideshare.net/tackers/why-we-chose-mongodb-for-... And reupping my previous, three-part series on MongoDB: On MongoDB NoSQL databases were the future. MongoDB was the database for "modern" web engineer…

I think you're asking the wrong question. The question should be: How did MongoDB become so successful? IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API. MongoDB was the natural choice because they excelled at being accessible to devs who were already familiar with Javascript and JSON. Not only that, their marketing/outreach efforts were also…

> IMO, the reason is that newer developers faced the choice of learning SQL or learning to use something with a Javascript API.

The thing I dislike about this type of comment – although I now notice yours doesn't explicitly say this – is the implication that devs don't like SQL because they're lazy or stupid. Well, sometimes that is probably true! But there are some tasks where you need to build the query dynamically at run time, and for those tasks MongoDB's usual query API, or especially its aggregation pipeline API, are genuinely better than stitching together fragments of SQL in the form of text strings. Injection attacks and inserting commas (but not trailing commas) come to mind as obvious difficulties. For anyone not familiar, just look at how close to being a native Python API pymongo is:

    pipeline = [
        {"$unwind": "$tags"},
        {"$group": {"_id": "$tags", "count": {"$sum": 1}}},
        {"$sort": SON([("count", -1), ("_id", -1)])}
    ]
    result_cursor = db.things.aggregate(pipeline)

Of course you could write an SQL query that does this particular job and is probably clearer. But if you need to compose a bunch of operations arbitrarily at runtime then using dicts and lists like this is clearly better.

Of course pipelines like this will typically be slow as hell because arbitrary queries, by their nature, cannot take advantage of indices. But sometimes that's OK. We do this in one of our products and it works great.

With JSONB and replication enhancements, Postgres is close to wiping out all of MongoDB's advantages. I would love to see a more native-like API like Mongo's aggregation pipeline, even if it's just a wrapper for composing SQL strings. I think that would finish off the job.