Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
Did you use Camus for ETL, and if so, did you have to modify it to work with S3?
Scaling Analytics at Amplitude
11–20 of 34 posts
Re: Scaling Analytics at Amplitude
#12What shortcomings of Redis set operations does the in-memory data store address, and how? Unrelated rant: regardless of its merits, "Lambda" Architecture is probably the most annoying overloaded term in use today, second only to "Isomorphic" Javascript. Just because something has a passing resemblance to the functional style doesn't grant license to re-appropriate a well understood term of art.
Redis is a great piece of software, and we leverage it for several uses cases outside of managing sets. For our use case, there were a couple of blockers that prevented Redis from being a viable solution:
1. It's tricky to scale out a Redis node when it gets too big. Because RDB files are just a single dump of all data, it's not easy to make a specific partitioning of the dataset. This was a very important requirement for us in order to ease scaling (redis-cluster wasn't ready yet -- we've been following that carefully).
2. When you store hundreds of GB of persistent data in Redis, the startup process can be very slow (restoring from RDB/AOF). Since it can't serve reads or writes during this time, you're unavailable (setting up a slave worsens the following problem).
3. The per-key overhead in Redis (http://stackoverflow.com/questions/10004565/redis-10x-more-m...). We have many billions of sets that are often only a few elements in size -- think of slicing data by city or device type -- which means that the resulting overhead can be larger than the dataset itself.
If you think about these problems upfront, they're not too difficult to solve for a specific use case (partition data on disk, allow reads from disk on startup), but Redis has to be generic and so can't leverage the optimizations we made.
Re: Scaling Analytics at Amplitude
#13Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
You could store the sets in postgresql arrays(to remove row overhead) (1GB maximum field size) and build some efficient union,intersect functions so you wouldn't have to unnest?
We tried a variety of PostgreSQL-based approaches, including this one. Unfortunately, the way you do set insertions using arrays is to do a O(n) membership check, which means your set will take O(n^2) to construct -- very inefficient.
Re: Scaling Analytics at Amplitude
#14Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
How did you guys split the databases per customer? Is it all one big stream of data for you or does it get split at a pretty early level? Is data of multiple customers in every database or do you maintain a cluster per customer?
Re: Scaling Analytics at Amplitude
#15Out of curiosity why weren't products like Druid http://druid.io/ or influxdb https://influxdb.com/ or possibly opentsdb taken into consideration ?
To be totally honest, there are so many technologies out there that claim to solve analytics that it's tough to seriously consider all of them. That said, we have looked at Druid, which is also a good example of using lambda architecture in practice ( http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-sys…
Druid does pre-aggregation (roll-up) of data at ingestion time and is also used at scale (30+ trillion events, ingesting over 1M+ events/s) by numerous large technology companies: http://druid.io/druid-powered.html
Re: Scaling Analytics at Amplitude
#16Earlier quoted context omitted.
To be totally honest, there are so many technologies out there that claim to solve analytics that it's tough to seriously consider all of them. That said, we have looked at Druid, which is also a good example of using lambda architecture in practice ( http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-sys…
Druid does pre-aggregation (roll-up) of data at ingestion time and is also used at scale (30+ trillion events, ingesting over 1M+ events/s) by numerous large technology companies: http://druid.io/druid-powered.html
Note that the commenter mentioned mid-2014. That page first appeared on (or about) July 29th, 2014[1], and at that time only contained 4 names:
Metamarkets
Netflix
LiquidM
N3twork
So while today Druid may be in use by "numerous large technology companies", at the time the commenter was researching it wasn't showcasing as many large companies.
[1] https://web.archive.org/web/20140729014707/http://druid.io/d...
Re: Scaling Analytics at Amplitude
#17Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
How did you guys split the databases per customer? Is it all one big stream of data for you or does it get split at a pretty early level? Is data of multiple customers in every database or do you maintain a cluster per customer?
Most of our databases are multi-tenant, so a single cluster will handle all customer data. The exception is Redshift, which has a separate cluster for each customer since we allow them to have direct access to it (https://amplitude.com/blog/2015/06/05/optimizing-redshift-pe...).
Re: Scaling Analytics at Amplitude
#18Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
Did you use Camus for ETL, and if so, did you have to modify it to work with S3?
We don't use Camus; IIRC, it didn't exist at the time that we built most of the infrastructure. We just read data directly out of Kafka using client libraries.
Re: Scaling Analytics at Amplitude
#19"Finally, at query time, we bring together the real-time views from the set database and the batch views from S3 to compute the result"
so how in the heck does this work? at query time you decide what file to get our of s3 (hwo do u decide this?), parse it, filter it, and merge with the results from the custom made Redis like real time database?
Re: Scaling Analytics at Amplitude
#20"Finally, at query time, we bring together the real-time views from the set database and the batch views from S3 to compute the result" so how in the heck does this work? at query time you decide what file to get our of s3 (hwo do u decide this?), parse it, filter it, and merge with the results from the custom made Redis like real time database?
The files in S3 are pre-aggregated results keyed by how we fetch them (e.g. there will be a file containing all of the users active on a particular day). What you've described is a pretty accurate description of what happens :)