Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make MongoDB not suck for analytics

scaleapi.com

Post reply on HN
Print view View original

31–40 of 98 posts

Re: How to make MongoDB not suck for analytics

#31

Post by oneweekwonder »

ty64 wrote: post #15
larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

This exists: https://www.mongodb.com/download-center#bi-connector

> The MongoDB Connector for BI is available as part of the MongoDB Enterprise Advanced subscription,

So may one only use it with a subscription?

I got mongodb postgres foreign data wrapper[0] working in a previous life.

[0]: https://github.com/EnterpriseDB/mongo_fdw

Re: How to make MongoDB not suck for analytics

#32

Post by mac01021 »

squirrelicus wrote: post #19

Okay so... To make MongoDB not suck for analytics, ETL it in a different format. For engineers trained in backed systems, this is pretty obvious. After reading this, I also don't know why I'd choose Pequot things over any other thing. Baby's first ETL -- just scan the db with a cursor and analyze the data in a script -- tends to cover 90% of the use cases for BI db analytics with almost zero resource consumption anyw…

As someone who grew up around the home if the Pequot tribe, I'm amused by the choice made here by your input device's autocorrect feature.

Re: How to make MongoDB not suck for analytics

#33

Post by georgewfraser »

There are several companies, including mine (Fivetran) that will replicate MongoDB into a columnar data warehouse for analytics. For most people, a commercial replication tool + a commercial columnar data warehouse is the best trade off of cost/ease of use. Commercial DWHs deal with all the details of patching columnar formats under-the-hood, and commercial replication tools like us will deal with all the complexity of things like the mongo oplog. For not that much $ you can have a working system in like a day.

Re: How to make MongoDB not suck for analytics

#34

Post by [deleted] »

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

[deleted]

Re: How to make MongoDB not suck for analytics

#35

Post by jinqueeny »

larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

If I understand it correctly, the "business intelligence stack" you are looking for is something that bridges the gap between the online transactional processing (OLTP) and online analytical processing (OLAP). If that's the case, then some new jargons might help you:

- hybrid transactional and analytical processing (HTAP), coined by Gartner, - hybrid operational and analytical workloads (HOAP), by 451 Research - Translytical, by Forrester

If that's the solution you want to explore, TiDB (https://github.com/pingcap/tidb), the open source distributed scalable HTAP database, might be able to help you. ETL is no longer necessary with TiDB’s hybrid OLTP/OLAP architecture.

Here is a use case about how it helps the largest B2C fresh produce online marketplace in China to acquire real-time intelligence:

https://www.datanami.com/2018/02/22/hybrid-database-capturin...

Here is a tutorial about how you can try TiDB/TiSpark on your own laptop using Docker Compose: https://www.pingcap.com/blog/how_to_spin_up_an_htap_database...

Disclaimer: I work for TiDB.

Re: How to make MongoDB not suck for analytics

#36

Post by stickfigure »

I tried using MongoDB for the customer-facing analytics of a large e-commerce marketplace. It didn't work very well. The problem is that at some point you end up wanting joins.

MongoDB was actually the third try. My first two attempts were BigQuery and Keen, neither of which worked out because they support only one index - time. Users want to slice and dice by various axes! And there's an obvious additional index you need - "merchant" - which column stores usually say propose setting up isolated partitions for. If you do that, you can't ask questions across the whole system!

We ended up with Postgres. It was actually faster than MongoDB for simple aggregations, and joins made it much better/faster for complicated queries. Of course it only works quickly if your dataset fits in RAM, but terabyte-size instances are pretty affordable and give you a lot of headroom.

That was a couple years ago. I don't know what they're using now, probably the same. It was a frantic few weeks figuring out what was going to work - each of those systems made it to production and quickly discovered to be inadequate in vivo. If you're in a startup, even if you're using exotic NoSQL systems like Google Cloud Datastore or DynamoDB - just use Postgres or MySQL for analytics. It will work long enough for you to figure out something else when you need it.

Re: How to make MongoDB not suck for analytics

#37

Post by georgewfraser »

jinqueeny wrote: post #35
larrydag wrote: post #12

This is a huge concern for me at my current organization. Dev has decided to put all data into mongoDB. Yet all decisions are based on that data and the tools we have do not allow for seamless flow (ETL) from mongoDB. That data is important for deriving decisions that affect revenue and costs. Where are solutions for the data analysts and scientists? Frankly I'm pretty sick of hearing it can just be automated. In my…

If I understand it correctly, the "business intelligence stack" you are looking for is something that bridges the gap between the online transactional processing (OLTP) and online analytical processing (OLAP). If that's the case, then some new jargons might help you: - hybrid transactional and analytical processing (HTAP), coined by Gartner, - hybrid operational and analytical workloads (HOAP), by 451 Research - Tran…

The problem with hybrid solutions is that you usually want your data warehouse (which is queried by analysts) separate from your database (which is queried by your app). The data warehouse is getting random queries written by analysts, and its schema is constantly evolving as upstream data sources are added and changed. This is not a recipe for a high-availability system. Since you’re going to set up a separate data warehouse anyway, having a single database that can do both types of workload isn’t as useful as you might expect.

Re: How to make MongoDB not suck for analytics

#38

Post by oneweekwonder »

nevi-me wrote: post #30
riboflavin wrote: post #3

Dremio helps with a lot of this, particularly the speed aspect – uses Parquet as well as Apache Arrow. (I work at Dremio.) Speeding things up: https://docs.dremio.com/acceleration/reflections.html

Dremio quickly becomes useless with MongoDB given that for a while it's not been possible to join data from two MongoDB collections by their object IDs. Last time I checked, Dremio mangled the id into some string that can't even be matched to the same id on a separate collection. I had data in PG and Mongo, but couldn't join it together. I asked about this on the forum, was told it's a known issue; and it seemed to e…

> The downside is that one needs a business license to use the BI connector.

Have you looked the postgres mongo fdw[0] before?

[0]: https://github.com/EnterpriseDB/mongo_fdw

Re: How to make MongoDB not suck for analytics

#39

Post by sdoering »

dizzystar wrote: post #22
twblalock wrote: post #10

Earlier quoted context omitted.

That can be automated.

In theory, yes; in practice, not really.

I tend to disagree. Having multiple automated ETL processes running for different projects/clients/colleagues I see that the code does not change as often, as I had anticipated.

Automation here (in my case) is a net win on time.

Re: How to make MongoDB not suck for analytics

#40

Post by sbr464 »

sateesh wrote: post #23
mindingdata wrote: post #21

Earlier quoted context omitted.

Others are sharing out of the box solutions. But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "tail" the oplog, and get every transaction in near real time. We used this for updating Elasticsearch indexes etc in what is basically realtime, without having to poll or modify existing code at all.

Yes you can tail oplog, but in a sharded setup with multiple clusters this becomes a pain.

The new change streams api resolved a lot of this, since it allows your to utilize the aggregation framework to subscribe to highly filtered/specific queries. The upcoming 4.0 release is expanded further.
Post reply on HN