Scaling Analytics at Amplitude
amplitude.com
Scaling Analytics at Amplitude
1–10 of 34 posts
Re: Scaling Analytics at Amplitude
#2The lambda architecture and the split between the heavy slow processing and the interactive processing reminds me of how a few of our customers are blending Hadoop and Couchbase for similar use cases: http://www.couchbase.com/fr/ad_platforms
Re: Scaling Analytics at Amplitude
#3Out of curiosity why weren't products like Druid http://druid.io/ or influxdb https://influxdb.com/ or possibly opentsdb taken into consideration ?
Re: Scaling Analytics at Amplitude
#4Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
Re: Scaling Analytics at Amplitude
#5> in-memory database holds only a limited set of data
MemSQL is not just in-memory, but also has column-store (note: I don't know VoltDB). You can think of MemSQL not as "does everything in-memory", but "uses memory at the best".
Re: Scaling Analytics at Amplitude
#6What shortcomings of Redis set operations does the in-memory data store address, and how?
Unrelated rant: regardless of its merits, "Lambda" Architecture is probably the most annoying overloaded term in use today, second only to "Isomorphic" Javascript. Just because something has a passing resemblance to the functional style doesn't grant license to re-appropriate a well understood term of art.
Re: Scaling Analytics at Amplitude
#7What shortcomings of Redis set operations does the in-memory data store address, and how? Unrelated rant: regardless of its merits, "Lambda" Architecture is probably the most annoying overloaded term in use today, second only to "Isomorphic" Javascript. Just because something has a passing resemblance to the functional style doesn't grant license to re-appropriate a well understood term of art.
I also wonder what sacrifices were made to the design used in Redis so that it is able to handle better performance.
Re: Scaling Analytics at Amplitude
#8Out of curiosity why weren't products like Druid http://druid.io/ or influxdb https://influxdb.com/ or possibly opentsdb taken into consideration ?
To be totally honest, there are so many technologies out there that claim to solve analytics that it's tough to seriously consider all of them.
That said, we have looked at Druid, which is also a good example of using lambda architecture in practice (http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-systems is very similar to it. We still believe the pre-aggregation approach is critical for performance in our use case, though. Lastly, when we started building the architecture (mid-2014), Druid was very new, and I'm generally wary of designing everything around a new and potentially unstable piece of software.
Re: Scaling Analytics at Amplitude
#9Author of the post here. Happy to talk about how we've designed/built our architecture at Amplitude!
You could store the sets in postgresql arrays(to remove row overhead) (1GB maximum field size) and build some efficient union,intersect functions so you wouldn't have to unnest?
Re: Scaling Analytics at Amplitude
#10Out of curiosity why weren't products like Druid http://druid.io/ or influxdb https://influxdb.com/ or possibly opentsdb taken into consideration ?
To be totally honest, there are so many technologies out there that claim to solve analytics that it's tough to seriously consider all of them. That said, we have looked at Druid, which is also a good example of using lambda architecture in practice ( http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-sys…
Just to note, with druid you are able to have preaggregated tables based on dimensions. Overall good article and thanks for sharing.