Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye MongoDB, Hello PostgreSQL (2015)

developer.olery.com

Post reply on HN
Print view View original

161–170 of 172 posts

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#161

Post by amenod »

kinos wrote: post #153
amenod wrote: post #120

Earlier quoted context omitted.

That's what I meant by "no persistence needed". In both cases, I wasn't actually saving anything to disk. I needed an in-memory storage for a never-ending stream of incoming data that would be read (and purged) by a separate process, no biggie if the data got lost occasionally. I also simply inserted and read the data, no complex queries or anything. Key-value store is all I needed and for this particular case Postgr…

Why did you use a database instead of just creating a global map and running in memory? Sharding?

No - different processes were reading and writing.

And once you implement a global map which is accessible from multiple processes, you might as well use an existing solution, like Redis. Or, as I've learned, pgSQL - which gave me the same performance with less moving parts (because I was already using it) when I disabled WAL.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#162

Post by kumarvvr »

StreamBright wrote: post #147
kumarvvr wrote: post #59

Earlier quoted context omitted.

And I wish they never get old. Every few years, new paradigms come to the forefront and debates like these certainly help many developers choose one way or the other.

In what paradigm are you referring to exactly with Postgres and Mongodb? Relational turning 51 years old in 2021 while key-value stores category (which MongoDB falls into) is at least 40. So in a sense this article is about which technology should we use, a 40 or a 50 years old one?

The paradigm of which type of database is best suited for large scale data, distributed in different locations.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#163

Post by citrin_ru »

psychometry wrote: post #122
anticristi wrote: post #5

I used both NoSQL and SQL databases. Proper replication topologies (e.g. multi-master configurations) and JSON columns killed the NoSQL movement. I'm glad the NoSQL movement existed and forced the SQL camp into out-of-the-box thinking, but I wouldn't go for any NoSQL solution today on a greenfield project.

No SQL database has the anywhere near the search capabilities of Elasticsearch/Lucene. I also wouldn't choose Postgres for storing logs, either, but I would use a document store (i.e. ELK stack).

ClickHouse suits well for string logs, especially if a log message have some fixed structure which can be mapped to separate columns. And most logs have some number of predefined fields which are easy to map to columns (e. g. timestamp, IP, request time, e. t. c.). You can store free-form JSON which doesn't have per-defined schema in a string column, but ELK would be probably better in this case.

And then you can query logs using all power of SQL.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#164

Post by Jean-Philipe »

timewarrior wrote: post #135
Jean-Philipe wrote: post #81

Earlier quoted context omitted.

> With mongo, you just add a column and you are done. No, usually you are not done. Your data doesn't have the column. So either you write your code in a way that it can handle the column being optional or you write code to read each document and add the column manually. I've started out with mongodb on Fit Analytics originally, until we noticed at some point that our data was an inconsistent mess and we'd implement…

Thanks for raising this good point. I didn't get a chance to elaborate on this in my original post because it was getting too long and it was end of a long day. The cost of adding a column is zero, in terms of schema migration. So that helps with developer velocity. We used an ORM (mongoose), so that we can have added constraints including default values. But Mongo did help us avoid the friction of adding columns.

We also used mongoose, which is awesome. In retrospect, I actually wish we had kept mongodb as a read-only database and used postgres as the source of truth. Our data was mostly read-only, with some rather complex relations behind the scenes.

For quick prototypes, I still prefer mongodb.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#165

Post by wilco_vd »

chris_wot wrote: post #156
menzoic wrote: post #155

Earlier quoted context omitted.

Without knowing the context it's impossible to say if it was user error or application error. I don't see it as a fair write up.

It's a fair write up in that they were unable to diagnose the actual problem with MongoDB. Which leads to one of the problems with the product - it is hard to know what is causing the issue when it has problems.

Hey, I was one of the developers on the team at that time. It's been fun re-reading that write up. I learned a lot since then, and I think the whole industry has learned a lot regarding NoSQL databases, their strengths and weaknesses and how to deal with those in your applications.

You are correct that the biggest issue was that we couldn't diagnose problems when they occurred, and therefore we couldn't learn. Some of those problems were for sure "user error", others were absolutely problems with MongoDB at the time. It was a "young" database back in 2015.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#166

Post by wilco_vd »

StreamBright wrote: post #20

These articles never get old. Hey we picked solution A and realized that it has some cons not only pros and now we are moving to solution B that only has pros. Few years later there is a new article, hey we are moving to solution A|C because...

I was one of the developers on the team that wrote that article. The main point of the article was (and still is) not that PostgreSQL is better than MongoDB, but that PostgreSQL suited us better at that point. In hindsight I think it suited us better because MongoDB was relatively new tech, PostgreSQL was more mature. Especially if it came to tooling and diagnosis on -what- is going wrong when things go wrong, as well as searchable help on the internet.

The extra maturity of PostgreSQL allowed us to learn faster and evolve our software faster with less surprises. As a small team in a start-up this was essential to us.

Considering other articles on that "picked a solution and didn't understand it well" - I do thing they are very valuable.

They provide an insight on how well frameworks and technology support a learning curve. It's not about having only Pro's. it's about how easy it is to get yourself unstuck if you engineered yourself in a corner where you don't want to be.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#167

Post by wcarss »

HelloNurse wrote: post #93
threeseed wrote: post #66

Earlier quoted context omitted.

Most data in large enterprises e.g. telcos, banks, insurance etc are not stored in a single data warehouse with well-defined relationships. They are in multiple disparate silos of which their EDW is just one. It's why Data Lakes become so commonplace because it was an easy way to just get all of the data out of the silos into one place so that the business could attempt to join between them. And it's why MongoDB (and…

If an enterprise wants to "get all the data out of the silos", unstructured storage is only "friendly" to the temptation to be sloppy in such an export. Not only missing or corrupted individual columns, but completely incoherent and arbitrary complex document structures. If "no-one knows the schemas and no-one knows how to join between them" the data lake project has already failed to provide value.

The secret is that it doesn't matter, because many, many tech businesses aren't business but investment ponzi schemes / "moonshots" on other people's money, who are mostly arm's length idiots.

Say you have a pile of garbage in silos: user metrics, analytics, usage stats, logs, sales, revenue across three different systems, etc.

You copy the garbage out of the silos and into a lake. It's a mess. No one could tell you anything real with that data.

So you write queries that ignore the errors, and don't show what you say they show, but look nice and you say they show what you know people want to hear. You know you have sales, right? You know things are going well, right? Or they will be soon, for sure. So you're making the numbers tell the story -- any problems in the data are just bumps in the measurement.

You show investors your fabrications, they show their investors your fabrications, and so on. You get another funding round for ~1.5x one year of salary for your employees + office costs. Life goes on, you have meetings. You make presentations.

Rinse. Repeat. You sit on boards, talk about innovation, sponsor a charity. Maybe make some investments! Be a local tech celebrity. 5 years go by, your business is huge and bleeding cash, and somehow... no one's willing to invest more.

So you sell what you can for what you can get. Some investors broke even, some lost the farm, some maybe even made 3-4x. No hard feelings overall! "Our Amazing Journey" blog post. The employees move on. You move on.

Rinse, and repeat.

Garbage in, garbage out.

It doesn't matter, because you were never really running a business.

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#168

Post by richardw »

aidos wrote: post #128
richardw wrote: post #106

Earlier quoted context omitted.

No sql injection, can set permissions for each proc, isolates app from DB implementation so you don’t have to redeploy app. Eg you can migrate older data to an archive table and query it on certain circumstances. App doesn’t need to know.

That’s stored ptocedures you’re talking about - I think. We’re talking about prepared statements.

Thanks! That was silly

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#169

Post by Twisell »

aidos wrote: post #128
richardw wrote: post #106

Earlier quoted context omitted.

No sql injection, can set permissions for each proc, isolates app from DB implementation so you don’t have to redeploy app. Eg you can migrate older data to an archive table and query it on certain circumstances. App doesn’t need to know.

That’s stored ptocedures you’re talking about - I think. We’re talking about prepared statements.

Please proceed to explain why in your opinion prepared statement don't help preventing SQL injection.

I might be wrong and I'm always happy to learn new things but give me some arguments instead of downvoting (as i guess it's you).

Re: Goodbye MongoDB, Hello PostgreSQL (2015)

#170

Post by HelloNurse »

wcarss wrote: post #167
HelloNurse wrote: post #93

Earlier quoted context omitted.

If an enterprise wants to "get all the data out of the silos", unstructured storage is only "friendly" to the temptation to be sloppy in such an export. Not only missing or corrupted individual columns, but completely incoherent and arbitrary complex document structures. If "no-one knows the schemas and no-one knows how to join between them" the data lake project has already failed to provide value.

The secret is that it doesn't matter, because many, many tech businesses aren't business but investment ponzi schemes / "moonshots" on other people's money, who are mostly arm's length idiots. Say you have a pile of garbage in silos: user metrics, analytics, usage stats, logs, sales, revenue across three different systems, etc. You copy the garbage out of the silos and into a lake. It's a mess. No one could tell you…

I'm sorry that you are involved with this kind of toxic "business" and with funding round beggars. I've only worked with real users of real data for real process integration, real marketing, real monitoring and auditing, under real management, in real companies.
Post reply on HN