Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Serverless Analytics Built from Scratch

statsbotco.github.io

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Show HN: Serverless Analytics Built from Scratch

#12

Post by cosmie »

InGodsName wrote: post #8
jimktrains2 wrote: post #2

Interesting. I once built a ga clone using appengine, cloud dataflow, and big query. I guess that would count as serverless? Benchmarked it against the official dumps to big quey too and it was pretty spot on for every metric we could lookup!

Bigquery takes minimum 2-3 seconds for every query. Google Analytics is much faster, responds in a few hundreads milliseconds. What did you use dataflow for? How did you get data from end points and insert them into bigquery? Using streaming inserts?

> Google Analytics is much faster, responds in a few hundreads milliseconds.

Are you referring to their reporting API, or their collection endpoint? The collection endpoint is certainly fast to respond, but the actual reporting API can be quite slow depending on what you're trying to get from it.

> What did you use dataflow for? How did you get data from end points and insert them into bigquery? Using streaming inserts?

I'm not the parent, but I've created setups like what was mentioned. It sounds like they hosted the collection endpoint on AppEngine, then used DataFlow for streaming the data into BigQuery. Potentially using a Pub/Sub topic to queue up for DataFlow, since that has native integrations with DataFlow and even has a template available to support it[1].

[1] https://cloud.google.com/dataflow/docs/guides/templates/prov...

Re: Show HN: Serverless Analytics Built from Scratch

#13

Post by teej »

InGodsName wrote: post #9
teej wrote: post #6

Earlier quoted context omitted.

I’ve seen folks put their pixel endpoint behind Fastly and process the access log delivered in S3. A Fastly VCL can handle the same transform that this Lambda is doing.

Is fastly free? Why would they use fastly and not s3?

S3 access logs alone are not sufficient to replicate this pipeline. This pixel is stateful (for the anonymous user ID) and S3 access logs don’t include arbitrary headers, in this case the cookie with the user id. Fastly would let you eliminate API gateway, Lambda, and both Kinesis steps.

API Gateway by itself is $3.50/million requests, which is 2-4x more expensive than Fastly at $0.75 - $1.60/million

Re: Show HN: Serverless Analytics Built from Scratch

#14

Post by mrkurt »

teej wrote: post #6
cheriot wrote: post #3

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

I’ve seen folks put their pixel endpoint behind Fastly and process the access log delivered in S3. A Fastly VCL can handle the same transform that this Lambda is doing.

We have people doing exactly this with fly.io, you could also do it with lambda@edge if you're a masochist. Or with Cloud Flare workers if you dislike small startups.

Re: Show HN: Serverless Analytics Built from Scratch

#20

Post by teej »

tyingq wrote: post #16

Genuine question. What does this do that just inserting vanilla GA code in the page doesn't? Trying to understand the "why".

Some people don’t want to put a GA tag on their site because of concerns around how Google uses the data. Also you can’t arbitrarily query GA data so this gives you that capability.
Post reply on HN