The Twitch Statistics Pipeline
ossareh.posthaven.com
The Twitch Statistics Pipeline
1–10 of 13 posts
Re: The Twitch Statistics Pipeline
#2"Our clients send base64 encoded JSON objects to our edge server using HTTP GETs."
Encoded json in get parameters? That's not what GET requests are for. use POST requests for that, or you'll quickly be limited by the max size of a get request, somewhere around 8KB.
Re: The Twitch Statistics Pipeline
#3"Our clients send base64 encoded JSON objects to our edge server using HTTP GETs." Encoded json in get parameters? That's not what GET requests are for. use POST requests for that, or you'll quickly be limited by the max size of a get request, somewhere around 8KB.
Yes, totally agree on that part! In fact, we tend to have had bad experiences with URLs over 4KB. Fortunately most of our stats requests come in at about 1KB.
We're in the process of moving over to POSTs. The second part of this series will go into more detail as to the "whys". The primary reason we use GETs is backwards compatibility; we wanted our new team to be a success and in ensuring that we thought carefully about the battles worth fighting - the Mixpanel stats client is a good one, so we opted for being "Mixpanel Protocol"-compatible; they use base64 encoded json blobs shipped using an HTTP GET, and thusly so do we.
Re: The Twitch Statistics Pipeline
#4"Our clients send base64 encoded JSON objects to our edge server using HTTP GETs." Encoded json in get parameters? That's not what GET requests are for. use POST requests for that, or you'll quickly be limited by the max size of a get request, somewhere around 8KB.
If you are sending over 8KB of JSON, something is fucked
Re: The Twitch Statistics Pipeline
#5The post says that latency is an issue. In this case I would look at using Kinesis (AWS's hosted Kafka equivalent) which can barf data directly into Redshift (as I understand it; haven't used this functionality.)
Re: The Twitch Statistics Pipeline
#6"Our clients send base64 encoded JSON objects to our edge server using HTTP GETs." Encoded json in get parameters? That's not what GET requests are for. use POST requests for that, or you'll quickly be limited by the max size of a get request, somewhere around 8KB.
If you are sending over 8KB of JSON, something is fucked
I could see a bulk metric insert [e.g. a modified version of StatsD] being >8KB.
Re: The Twitch Statistics Pipeline
#7The post says that latency is an issue. In this case I would look at using Kinesis (AWS's hosted Kafka equivalent) which can barf data directly into Redshift (as I understand it; haven't used this functionality.)
Our latency concerns are "can our product managers make decisions quickly", to that end we're OK with a 24 hour latency for stats. The fact that we're at 5 hour latency gives us a lot of breathing room.
Kinesis looks really interesting, and is definitely something that we're going to look at once we work out our ETL process.
The order of priority for us has been:
1 - Get a pipeline up and running 2 - Make it robust 3 - Make it fast.
Pipeline v3, our current one, satisfies (2). We expect to be working on (3) in the near future. ETL is the latter part of (2). (3) results in powering dashboards, we expect those to contain a lot of joined data and having a robust ETL process is pretty key to that.
Re: The Twitch Statistics Pipeline
#8The post says that latency is an issue. In this case I would look at using Kinesis (AWS's hosted Kafka equivalent) which can barf data directly into Redshift (as I understand it; haven't used this functionality.)
Kinesis was released in mid-November 2013 so this team's effort predates that service.
I think Kinesis could replace the first three boxes in their diagram, and do it in real-time. (I haven't used kinesis so I could be wrong.)
It's amazing how fast big data infrastructure is evolving. For anybody looking to build something it seems your chosen solution will be obsolete by the time you release.
Re: The Twitch Statistics Pipeline
#9Nice to see lots of similarities with how we do things at Snowplow (https://github.com/snowplow/snowplow)...
Re: The Twitch Statistics Pipeline
#10The post says that latency is an issue. In this case I would look at using Kinesis (AWS's hosted Kafka equivalent) which can barf data directly into Redshift (as I understand it; haven't used this functionality.)
Our latency concerns are "can our product managers make decisions quickly", to that end we're OK with a 24 hour latency for stats. The fact that we're at 5 hour latency gives us a lot of breathing room. Kinesis looks really interesting, and is definitely something that we're going to look at once we work out our ETL process. The order of priority for us has been: 1 - Get a pipeline up and running 2 - Make it robust 3…
Yes - Kinesis is awesome. This is already a little out of date but shows how we're now porting our open source event pipeline to on top of Kinesis: http://snowplowanalytics.com/blog/2014/02/04/snowplow-0.9.0-...