Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make MongoDB not suck for analytics

scaleapi.com

Post reply on HN
Print view View original

41–50 of 98 posts

Re: How to make MongoDB not suck for analytics

#41

Post by sbr464 »

twblalock wrote: post #4
jrochkind1 wrote: post #2

What is the benefit of having it in mongo in the first place, in this scenario?

The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…

I agree with this perspective, and have been researching it more lately.

Re: How to make MongoDB not suck for analytics

#42

Post by eksemplar »

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

I think it’s less of an issue. The way analytics often work enterprise, is that you pay someone a lot of money to build cubes on your data that does a big chunk of the actual data science, before it’s handed off to economists who can’t code.

Then they’ll build their BI models in some high level drag and draw system, and pay extra whenever they realize they didn’t get everything they needed in a cube.

The only place I’ve seen actual data scientists is at the university or at the 100% software companies that sell both the solution and the data cube. I’ve never met a real world analytic who could actually code. :p

You’d want to keep a separate dB for your analytics either way though, as they typically eat up quite a lot of load and you don’t want that to interfere with your production environment when you don’t have to.

Re: How to make MongoDB not suck for analytics

#43

Post by threeseed »

stickfigure wrote: post #36

I tried using MongoDB for the customer-facing analytics of a large e-commerce marketplace. It didn't work very well. The problem is that at some point you end up wanting joins. MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you…

You are completely contradicting yourself.

On one hand you complain about using technologies before you have done a prototype and evaluated the product. Then you blindly tell startups to just use MySQL/PostgreSQL without having any idea of their use case or whether it matches their query patterns.

If you are a startup the right way to go is to document your use case, understand what queries those use cases demand and then find the right database that satisfies it e.g. don't pick MongoDB if you are doing lots of joins and don't pick PostgreSQL if you are doing wide-table feature engineering type analytics.

Right tool for the right job.

Re: How to make MongoDB not suck for analytics

#44

Post by jinqueeny »

georgewfraser wrote: post #37
jinqueeny wrote: post #35

Earlier quoted context omitted.

If I understand it correctly, the "business intelligence stack" you are looking for is something that bridges the gap between the online transactional processing (OLTP) and online analytical processing (OLAP). If that's the case, then some new jargons might help you: - hybrid transactional and analytical processing (HTAP), coined by Gartner, - hybrid operational and analytical workloads (HOAP), by 451 Research - Tran…

The problem with hybrid solutions is that you usually want your data warehouse (which is queried by analysts) separate from your database (which is queried by your app). The data warehouse is getting random queries written by analysts, and its schema is constantly evolving as upstream data sources are added and changed. This is not a recipe for a high-availability system. Since you’re going to set up a separate data…

This is exactly how TiDB works like a charm. Internally, thanks to the Raft consensus algorithm (http://raft.github.io/), we could schedule and process the workloads separately: OLTP workloads to the leader replicas, OLAP to the follower replicas for the random or heavy analytics.

The past solution of the separate operational database and data warehouse poses great challenges for real-time analytics because it needs either data pipeline or the ETL process which could be the bottleneck of being "real-time", not to mention the waste of time, efforts and human resources maintaining multiple data warehouses. It was impossible for real-time analysis because, in the past, you would need a data pipeline, or message queue with the equivalent throughputs with your OLTP database, which I believe does not exist.

However, whether to adopt this hybrid solution depends on your specific usage scenario. For cases where users want to do real-time analysis in their data warehouse upon the same data table as in their OLTP database, TiDB is your choice.

Re: How to make MongoDB not suck for analytics

#45

Post by lomnakkus »

threeseed wrote: post #43
stickfigure wrote: post #36

I tried using MongoDB for the customer-facing analytics of a large e-commerce marketplace. It didn't work very well. The problem is that at some point you end up wanting joins. MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you…

You are completely contradicting yourself. On one hand you complain about using technologies before you have done a prototype and evaluated the product. Then you blindly tell startups to just use MySQL/PostgreSQL without having any idea of their use case or whether it matches their query patterns. If you are a startup the right way to go is to document your use case, understand what queries those use cases demand and…

Without wishing to put words in their mouth, I think parent poster's point might be that PostgreSQL will do at least a decent job at most things you'll want to throw at it.

This is not the case for most of the NoSQL databases where you'll pay for lack of certain features either by a) having to write a lot of code, or b) bad-to-crippling performance for use cases it wasn't meant to solve.

So, unless you're already very clear on what your exact use case is going why the spend time analysing before even getting your project off the ground?

Re: How to make MongoDB not suck for analytics

#47

Post by drej »

drej wrote: post #46

For those seeking tl;dr: The answer is not to use MongoDB.

A more complete answer is to dump your data into a columnar format into S3 and then use one of plethora analytics tools that can work with this format (AWS Athena and Drill are mentioned, other tools like Presto, Spark, Redshift Spectrum or BigQuery can help).

Re: How to make MongoDB not suck for analytics

#48

Post by drej »

drej wrote: post #46

For those seeking tl;dr: The answer is not to use MongoDB.

But I don't want to be just snarky. We faced the very same dilemma and solved it in a similar way - we use Apache Spark, which can connect to MongoDB directly. It loads fairly quickly and we can save it to Parquet on S3 directly, the whole thing is about 5 lines of code.

If you have a Spark platform in place, it's a decent solution for this.

Re: How to make MongoDB not suck for analytics

#49

Post by georgewfraser »

jinqueeny wrote: post #44
georgewfraser wrote: post #37

Earlier quoted context omitted.

The problem with hybrid solutions is that you usually want your data warehouse (which is queried by analysts) separate from your database (which is queried by your app). The data warehouse is getting random queries written by analysts, and its schema is constantly evolving as upstream data sources are added and changed. This is not a recipe for a high-availability system. Since you’re going to set up a separate data…

This is exactly how TiDB works like a charm. Internally, thanks to the Raft consensus algorithm ( http://raft.github.io/ ), we could schedule and process the workloads separately: OLTP workloads to the leader replicas, OLAP to the follower replicas for the random or heavy analytics. The past solution of the separate operational database and data warehouse poses great challenges for real-time analytics because it need…

It is absolutely a valuable feature to be able to update your data warehouse one-row-at-a-time, but this feature has to be weighed alongside all the other features. Also, the best commercial data warehouses are quite good at keeping up with small batches, so you can potentially run a "batch" data pipeline every minute or so, and get a "nearly-real-time" data warehouse in a conventional columnar design.

Re: How to make MongoDB not suck for analytics

#50

Post by twblalock »

drej wrote: post #46

For those seeking tl;dr: The answer is not to use MongoDB.

That doesn't get you out of having to face the problem. This is not a challenge unique to MongoDB or other NoSQL databases. Oracle or Postgres might be ideal for your transactional data store, and a columnar database might be ideal for your analytics.

I suppose you could choose one of those options and sacrifice either your customer experience or your analytics, but it's probably better to use the best database for each use case.

Post reply on HN