How to make MongoDB not suck for analytics
scaleapi.com
How to make MongoDB not suck for analytics
1–10 of 98 posts
Re: How to make MongoDB not suck for analytics
#2What is the benefit of having it in mongo in the first place, in this scenario?
Re: How to make MongoDB not suck for analytics
#3Dremio helps with a lot of this, particularly the speed aspect – uses Parquet as well as Apache Arrow. (I work at Dremio.) Speeding things up: https://docs.dremio.com/acceleration/reflections.html
Re: How to make MongoDB not suck for analytics
#4What is the benefit of having it in mongo in the first place, in this scenario?
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both.
If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because they can just copy the data into a different database that fits their needs.
Re: How to make MongoDB not suck for analytics
#5What is the benefit of having it in mongo in the first place, in this scenario?
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…
Fair enough. Perhaps there are other uses of mongo going on in addition to exporting it to a different database/format? Otherwise I'd be curious what justifies the mongo choice.
Certainly sometimes you are in a position where you just gotta take what other units in the org give you and deal with it.
But _someone_ in the org is hopefully in the position to be able to articulate why they are using mongo in the first place...
Re: How to make MongoDB not suck for analytics
#6What is the benefit of having it in mongo in the first place, in this scenario?
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…
"they can just copy the data into a different database that fits their needs."
That's easier said than done when your database is over 10 TB big.
Re: How to make MongoDB not suck for analytics
#7What is the benefit of having it in mongo in the first place, in this scenario?
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…
Except that devs have to do ETL every day so analysts can do their query work.
Re: How to make MongoDB not suck for analytics
#8Earlier quoted context omitted.
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…
"they can just copy the data into a different database that fits their needs." That's easier said than done when your database is over 10 TB big.
You only need to copy the stuff that changed since the last time you copied stuff.
If you are really generating 10TB of data more than a few times a day, you can look into putting it in something like Kafka for real-time consumption by the analytics team instead of batch copying.
Re: How to make MongoDB not suck for analytics
#9Dremio helps with a lot of this, particularly the speed aspect – uses Parquet as well as Apache Arrow. (I work at Dremio.) Speeding things up: https://docs.dremio.com/acceleration/reflections.html
I’m only familiar with speeding up Parquet - it looks like you’re mainly sorting or partitioning the data into different views so that you can choose the best view format at runtime based on your desired query or aggregation? We’ve seen this increase speeds by many orders of magnitude, so I wouldn’t be surprised that this creates meaningful speed-ups when done automatically (Which is cool!).
Random aside, how do you handle the consistency problems that can occur when you have multiple views when doing deletes?
Re: How to make MongoDB not suck for analytics
#10Earlier quoted context omitted.
The people who write the business logic and the people who do the analytics have different concerns. It's sometimes better to make different database choices for those two systems and just copy the data into the analytics system, rather than make a substandard choice of database to try to accommodate both. If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because t…
Except that devs have to do ETL every day so analysts can do their query work.
That can be automated.