Do you store raw data ingested from Kafka directly in S3 or have an intermediate database for hot data?
Scaling Analytics at Amplitude
21–30 of 34 posts
Re: Scaling Analytics at Amplitude
#22"Finally, at query time, we bring together the real-time views from the set database and the batch views from S3 to compute the result" so how in the heck does this work? at query time you decide what file to get our of s3 (hwo do u decide this?), parse it, filter it, and merge with the results from the custom made Redis like real time database?
The files in S3 are pre-aggregated results keyed by how we fetch them (e.g. there will be a file containing all of the users active on a particular day). What you've described is a pretty accurate description of what happens :)
We'll be sharing more about our query architecture in the future as well as other parts of the stack that we haven't included here. The query layer is an impressive piece of architecture that handles fast access to multiple distributed data stores.
Re: Scaling Analytics at Amplitude
#23Do you store raw data ingested from Kafka directly in S3 or have an intermediate database for hot data?
The raw data is stored directly into S3 every hour, but there are multiple systems that process the data directly from Kafka in order to produce the real-time pre-aggregated views.
Re: Scaling Analytics at Amplitude
#24Earlier quoted context omitted.
To be totally honest, there are so many technologies out there that claim to solve analytics that it's tough to seriously consider all of them. That said, we have looked at Druid, which is also a good example of using lambda architecture in practice ( http://druid.io/docs/0.8.0/design/design.html -- note the historical vs realtime distinction). They use many of the same design principles as us, and one of our sub-sys…
Druid does pre-aggregation (roll-up) of data at ingestion time and is also used at scale (30+ trillion events, ingesting over 1M+ events/s) by numerous large technology companies: http://druid.io/druid-powered.html
Hey Fan, I know you feel very strongly about Druid but at the time it wasn't the way to go, I can see how they might have opted to steer clear.
Re: Scaling Analytics at Amplitude
#25What shortcomings of Redis set operations does the in-memory data store address, and how? Unrelated rant: regardless of its merits, "Lambda" Architecture is probably the most annoying overloaded term in use today, second only to "Isomorphic" Javascript. Just because something has a passing resemblance to the functional style doesn't grant license to re-appropriate a well understood term of art.
Redis is a great piece of software, and we leverage it for several uses cases outside of managing sets. For our use case, there were a couple of blockers that prevented Redis from being a viable solution: 1. It's tricky to scale out a Redis node when it gets too big. Because RDB files are just a single dump of all data, it's not easy to make a specific partitioning of the dataset. This was a very important requiremen…
Hello Jeffrey,
First I wanted to say that your post is very nicely written and full of juicy details! :)
Regarding the sets database, I had to solve quite a similar problem at the company where I work and instead of sets I actually chose to use the Redis HypeLogLog structure instead of sets because for near real time results you just need an approximate count of the sets / or their intersection and you don't need to know the specific set members. I just wanted to let you know that it works great for us for with doing intersections (PFMERGE) on sets containing hundreds of millions of members. If anybody is interested I can do a writeup about it.
Did you ever consider using that?
Re: Scaling Analytics at Amplitude
#26Earlier quoted context omitted.
Redis is a great piece of software, and we leverage it for several uses cases outside of managing sets. For our use case, there were a couple of blockers that prevented Redis from being a viable solution: 1. It's tricky to scale out a Redis node when it gets too big. Because RDB files are just a single dump of all data, it's not easy to make a specific partitioning of the dataset. This was a very important requiremen…
Hello Jeffrey, First I wanted to say that your post is very nicely written and full of juicy details! :) Regarding the sets database, I had to solve quite a similar problem at the company where I work and instead of sets I actually chose to use the Redis HypeLogLog structure instead of sets because for near real time results you just need an approximate count of the sets / or their intersection and you don't need to…
Hyperloglog is approximate. You also can't do set complement. Also can't get the ids. But other than that it's great!
Re: Scaling Analytics at Amplitude
#27Earlier quoted context omitted.
Redis is a great piece of software, and we leverage it for several uses cases outside of managing sets. For our use case, there were a couple of blockers that prevented Redis from being a viable solution: 1. It's tricky to scale out a Redis node when it gets too big. Because RDB files are just a single dump of all data, it's not easy to make a specific partitioning of the dataset. This was a very important requiremen…
Hello Jeffrey, First I wanted to say that your post is very nicely written and full of juicy details! :) Regarding the sets database, I had to solve quite a similar problem at the company where I work and instead of sets I actually chose to use the Redis HypeLogLog structure instead of sets because for near real time results you just need an approximate count of the sets / or their intersection and you don't need to…
Thanks! We have considered using HLL, and it's a pretty cool algorithm.
For us, however, it's important to get the set members at the end of the day. Amplitude is unique from other analytics products in that we put a lot of emphasis on the actual users that correspond to a data point on a graph -- one of our key features, Microscope, is the ability to view those users, see more context around the events they are performing, and potentially create a dynamic cohort out of them. As such, approximations that don't allow us to get the set members don't quite satisfy our use case.
Re: Scaling Analytics at Amplitude
#28Earlier quoted context omitted.
Hello Jeffrey, First I wanted to say that your post is very nicely written and full of juicy details! :) Regarding the sets database, I had to solve quite a similar problem at the company where I work and instead of sets I actually chose to use the Redis HypeLogLog structure instead of sets because for near real time results you just need an approximate count of the sets / or their intersection and you don't need to…
Hyperloglog is approximate. You also can't do set complement. Also can't get the ids. But other than that it's great!
I am not gonna suggest that one solution fits all. Maybe you have different requirements but in our case (adtech) offering near real time reporting with a standard error rate of 0.81% is very good.
Re: Scaling Analytics at Amplitude
#29Earlier quoted context omitted.
Hello Jeffrey, First I wanted to say that your post is very nicely written and full of juicy details! :) Regarding the sets database, I had to solve quite a similar problem at the company where I work and instead of sets I actually chose to use the Redis HypeLogLog structure instead of sets because for near real time results you just need an approximate count of the sets / or their intersection and you don't need to…
Thanks! We have considered using HLL, and it's a pretty cool algorithm. For us, however, it's important to get the set members at the end of the day. Amplitude is unique from other analytics products in that we put a lot of emphasis on the actual users that correspond to a data point on a graph -- one of our key features, Microscope, is the ability to view those users, see more context around the events they are perf…
Sorry, I wanted to say that I am not actually not familiar with your product and was not aware of the feature in which you can create audiences for running ad campaigns. From the article I thought the sets were used mostly for real time analytics and that is why I started to talk about HLL.
If you do need the actual set members in real time then of course you can't use HLL :)
Re: Scaling Analytics at Amplitude
#30How do you decide what sets of users you pre aggregate?
It seems like without some limits in place you could end up with huge number of sets, especially if you are calculating these based on event properties.