31–34 of 34 posts
How do you decide what sets of users you pre aggregate? It seems like without some limits in place you could end up with huge number of sets, especially if you are calculating these based on event properties.
How do you decide what sets of users you pre aggregate? It seems like without some limits in place you could end up with huge number of sets, especially if you are calculating these based on event properties.
That's a great observation. Somewhere along the spectrum of query flexibility you reach a point where pre-aggregation doesn't work anymore. We have a separate column-store based system in place for certain types of queries which we'll almost certainly blog about in the future!
I guess we'll find out in a future post.
Earlier quoted context omitted.
Did you use Camus for ETL, and if so, did you have to modify it to work with S3?
We don't use Camus; IIRC, it didn't exist at the time that we built most of the infrastructure. We just read data directly out of Kafka using client libraries.
For the real time layer I see it as not being mission critical for most data sets to be 100% correct, but for the ETL part of the process, the guarantees provided by Camus (ensured by the OutputCommitters part of MR I believe) are invaluable.