Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make MongoDB not suck for analytics

scaleapi.com

Post reply on HN
Print view View original

51–60 of 98 posts

Re: How to make MongoDB not suck for analytics

#51

Post by twblalock »

mr_toad wrote: post #27
larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

> (ETL) from mongoDB Why not query the data directly in MongoDB?

You should at the very least be doing analytics queries on a replica, or you could be affecting the database performance (and the customer experience) in production.

But even if you did that, you'll find that you'll need joins and aggregations that are painful to do in Mongo yet trivial to do in a system that is designed for them.

Re: How to make MongoDB not suck for analytics

#52

Post by tixocloud »

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

I feel your pain on the lack of love for analysts and data scientists. IT has just gone and implemented “shiny new database infrastructure” and have been saying we are way ahead of the curve.

Problem is that it doesn’t work for analysis and we are still using our old platform, which works just fine.

Re: How to make MongoDB not suck for analytics

#55

Post by sztanko »

Just try this out: https://github.com/EXASOL/docker-db and you will be impressed. This is an embryo of a real analytical database.

Pros:

- an 8 CPU installation with 64gb memory will probably be hundred times faster then postgres.

-it supports full sql

- It is super stable, even as docker container

Cons:

- it does not support nested data

- once you reach volumes of around 2Tb, you will probably have to switch to a paid version (I mean, you still can continue running on a 200gb ram box, but it will be suboptimal)

P.s. I am not affiliated with Exasol.

Re: How to make MongoDB not suck for analytics

#56

Post by gaius »

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow

Seems to me that it’s on your devs to explain to the business why their poor technical choices now necessitate a substantial additional investment to get a usable solution. When they could have just used Postgres, and they knew it.

Re: How to make MongoDB not suck for analytics

#57

Post by georgewfraser »

stickfigure wrote: post #36

I tried using MongoDB for the customer-facing analytics of a large e-commerce marketplace. It didn't work very well. The problem is that at some point you end up wanting joins. MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you…

You say you need indexes because “users want to slice and dice by various axes”, but have you checked that the query planner actually uses these indices? When an aggregation includes a meaningful percentage of the table, a sequential scan will be faster than a series of indexed lookups. This is part of the reason why column stores don’t use indexes—they just aren’t that useful for analytical queries.

Re: How to make MongoDB not suck for analytics

#58

Post by arghwhat »

sztanko wrote: post #55

Just try this out: https://github.com/EXASOL/docker-db and you will be impressed. This is an embryo of a real analytical database. Pros: - an 8 CPU installation with 64gb memory will probably be hundred times faster then postgres. -it supports full sql - It is super stable, even as docker container Cons: - it does not support nested data - once you reach volumes of around 2Tb, you will probably have to switch to a pa…

> an 8 CPU installation with 64gb memory will probably be hundred times faster then postgres.

"Probably" not.

The way this usually goes down is that there may be a few synthetic benchmarks show a large performance benefit over existing established databases (x2, not x100), with any non-synthetic benchmark showing very poor performance (1/10th, 1/100th, sometimes even worse), and also often very unstable performance.

The product is then also usually beta quality, as it is hard to compete with the 36 years Postgres has been in development since its inception in 1982 (and that's not counting the 9 years of Ingres development, which Postgres—"Post-Ingres"—spawned from). Important features are usually also quite lacking.

If someone claims x10 or x100 performance improvement over established databases, they better have published a few papers about all the computer science research they must necessarily have done to get there.

Re: How to make MongoDB not suck for analytics

#59

Post by threeseed »

lomnakkus wrote: post #45
threeseed wrote: post #43

Earlier quoted context omitted.

You are completely contradicting yourself. On one hand you complain about using technologies before you have done a prototype and evaluated the product. Then you blindly tell startups to just use MySQL/PostgreSQL without having any idea of their use case or whether it matches their query patterns. If you are a startup the right way to go is to document your use case, understand what queries those use cases demand and…

Without wishing to put words in their mouth, I think parent poster's point might be that PostgreSQL will do at least a decent job at most things you'll want to throw at it. This is not the case for most of the NoSQL databases where you'll pay for lack of certain features either by a) having to write a lot of code, or b) bad-to-crippling performance for use cases it wasn't meant to solve. So, unless you're already ver…

> So, unless you're already very clear on what your exact use case is going why the spend time analysing before even getting your project off the ground?

Because if you don't know what you want you are almost guaranteed to pick the wrong technology.

Re: How to make MongoDB not suck for analytics

#60

Post by [deleted] »

stickfigure wrote: post #36

I tried using MongoDB for the customer-facing analytics of a large e-commerce marketplace. It didn't work very well. The problem is that at some point you end up wanting joins. MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you…

[deleted]
Post reply on HN