How to make MongoDB not suck for analytics • Hacker News Board

Re: How to make MongoDB not suck for analytics

#11

Post by marcinzm » Mon, Jul 02, 2018, 1:23 AM UTC

dizzystar wrote: post #7

twblalock wrote: post #4

Earlier quoted context omitted.

The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…

Except that devs have to do ETL every day so analysts can do their query work.

Which you'll likely want with any DB since analytics workloads are very different than the usual production DB workloads.

Re: How to make MongoDB not suck for analytics

#12

Post by larrydag » Mon, Jul 02, 2018, 1:40 AM UTC

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated.

In my mind there has to be a decent "business intelligence stack". I'm not sure I'm coining that because I didn't get good search results from that phrase. Believe me I've been trying to find solutions. I believe there is big opportunity in building out this sort of stack that bridges data management and data analysis. Sure you can call IBM, Microsoft, Dell, HP but be prepared for big costs and huge software bloat. I would like simplified solutions and options that can fit with most industry standard tools.

I'm also willing to work with anyone on this as well.

Re: How to make MongoDB not suck for analytics

#13

Post by ayw » Mon, Jul 02, 2018, 1:43 AM UTC

jrochkind1 wrote: post #2

What is the benefit of having it in mongo in the first place, in this scenario?

For better or for worse, MongoDB tends to be easier for developers move quickly, so it ends up getting adopted quite a bit. This is more about how to deal with it after it's already in your stack.

Re: How to make MongoDB not suck for analytics

#14

Post by ayw » Mon, Jul 02, 2018, 1:45 AM UTC

AznHisoka wrote: post #6

twblalock wrote: post #4

Earlier quoted context omitted.

The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…

"they can just copy the data into a different database that fits their needs." That's easier said than done when your database is over 10 TB big.

You just need to read the oplog, so it only needs to track your saves.

In general, you probably should have at least something in your stack which reads all changes from your DB, at the very least for backup reasons.

Re: How to make MongoDB not suck for analytics

#15

Post by ty64 » Mon, Jul 02, 2018, 1:46 AM UTC

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

This exists: https://www.mongodb.com/download-center#bi-connector

Re: How to make MongoDB not suck for analytics

#16

Post by omeid2 » Mon, Jul 02, 2018, 1:54 AM UTC

ayw wrote: post #13

jrochkind1 wrote: post #2

What is the benefit of having it in mongo in the first place, in this scenario?

For better or for worse, MongoDB tends to be easier for developers move quickly, so it ends up getting adopted quite a bit. This is more about how to deal with it after it's already in your stack.

RethinkDB blows MongoDB on easy to use factor out by a large margin, with the upside of being a project focused on actual quality rather than pure marketing.

Re: How to make MongoDB not suck for analytics

#17

Post by mitchell_h » Mon, Jul 02, 2018, 1:57 AM UTC

twblalock wrote: post #4

jrochkind1 wrote: post #2

What is the benefit of having it in mongo in the first place, in this scenario?

The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…

This is such a common issue there's an entire architecture pattern developed to solve it. https://en.wikipedia.org/wiki/Lambda_architecture . No classic ETL, and any number of folks/systems can plug into the messaging system and get all the data.

Re: How to make MongoDB not suck for analytics

#18

Post by scalablenotions » Mon, Jul 02, 2018, 2:31 AM UTC

ayw wrote: post #13

jrochkind1 wrote: post #2

What is the benefit of having it in mongo in the first place, in this scenario?

For better or for worse, MongoDB tends to be easier for developers move quickly, so it ends up getting adopted quite a bit. This is more about how to deal with it after it's already in your stack.

What about Managed solutions, like DynamoDB? What could be easier than that - with cloud scale analytics opportunities to boot.

Re: How to make MongoDB not suck for analytics

#19

Post by squirrelicus » Mon, Jul 02, 2018, 2:37 AM UTC

Okay so... To make MongoDB not suck for analytics, ETL it in a different format. For engineers trained in backed systems, this is pretty obvious. After reading this, I also don't know why I'd choose Pequot things over any other thing.

Baby's first ETL -- just scan the db with a cursor and analyze the data in a script -- tends to cover 90% of the use cases for BI db analytics with almost zero resource consumption anyway. Point being don't write a query to do analytics if your db can't answer your questions performantly, and don't build [latent, stale, slow] Enterprise ETL unless you really need it.

Re: How to make MongoDB not suck for analytics

#20

Post by threeseed » Mon, Jul 02, 2018, 2:39 AM UTC

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

You can connect Hadoop/Spark directly to MongoDB so in some cases you may not need to do an ETL at all.

You can also use something like NiFi which supports MongoDB and will allow you to shift data out to Avro/Parquet on HDFS/S3 for your data scientists to use.

As for BI stack. Not sure what you mean. There are hundreds of tools which blend data management and data analysis. You can do this with Hortonworks (Atlas + Spark) or Alteryx for example.