Show HN: Serverless Analytics Built from Scratch

#1

Post by keydunov » Thu, Dec 20, 2018, 3:09 PM UTC

Show HN: Serverless Analytics Built from Scratch
statsbotco.github.io

Re: Show HN: Serverless Analytics Built from Scratch

#2

Post by jimktrains2 » Thu, Dec 20, 2018, 5:01 PM UTC

Interesting. I once built a ga clone using appengine, cloud dataflow, and big query. I guess that would count as serverless? Benchmarked it against the official dumps to big quey too and it was pretty spot on for every metric we could lookup!

Re: Show HN: Serverless Analytics Built from Scratch

#3

Post by cheriot » Thu, Dec 20, 2018, 5:07 PM UTC

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

Re: Show HN: Serverless Analytics Built from Scratch

#4

Post by pavel_tiunov » Thu, Dec 20, 2018, 5:08 PM UTC

jimktrains2 wrote: post #2

Interesting. I once built a ga clone using appengine, cloud dataflow, and big query. I guess that would count as serverless? Benchmarked it against the official dumps to big quey too and it was pretty spot on for every metric we could lookup!

Yes. I guess your setup is serverless as well. Big Query is one of Serverless MPP databases that shares similar concepts with AWS Athena.

Re: Show HN: Serverless Analytics Built from Scratch

#5

Post by yahelc » Thu, Dec 20, 2018, 5:08 PM UTC

cheriot wrote: post #3

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

I've done this a few times and have found it to be an extremely effective way to do simple pixel tracking (for custom emails and the like).

Re: Show HN: Serverless Analytics Built from Scratch

#6

Post by teej » Thu, Dec 20, 2018, 5:22 PM UTC

cheriot wrote: post #3

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

I’ve seen folks put their pixel endpoint behind Fastly and process the access log delivered in S3. A Fastly VCL can handle the same transform that this Lambda is doing.

Re: Show HN: Serverless Analytics Built from Scratch

#7

Post by InGodsName » Thu, Dec 20, 2018, 5:30 PM UTC

cheriot wrote: post #3

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

Here is some more data: http://highscalability.com/blog/2018/4/2/how-ipdata-serves-2...

I don't understand what's cubejs doing in this app?

Once data is inside athena, it's matter of querying it right.

Re: Show HN: Serverless Analytics Built from Scratch

#8

Post by InGodsName » Thu, Dec 20, 2018, 5:32 PM UTC

jimktrains2 wrote: post #2

Interesting. I once built a ga clone using appengine, cloud dataflow, and big query. I guess that would count as serverless? Benchmarked it against the official dumps to big quey too and it was pretty spot on for every metric we could lookup!

Bigquery takes minimum 2-3 seconds for every query.

Google Analytics is much faster, responds in a few hundreads milliseconds.

What did you use dataflow for? How did you get data from end points and insert them into bigquery? Using streaming inserts?

Re: Show HN: Serverless Analytics Built from Scratch

#9

Post by InGodsName » Thu, Dec 20, 2018, 5:33 PM UTC

teej wrote: post #6

cheriot wrote: post #3

I'd be curious to see a cost estimate for some traffic level. I wonder if there's a way to put the pixel in s3 and process the access logs more cheaply.

I’ve seen folks put their pixel endpoint behind Fastly and process the access log delivered in S3. A Fastly VCL can handle the same transform that this Lambda is doing.

Is fastly free? Why would they use fastly and not s3?

Re: Show HN: Serverless Analytics Built from Scratch

#10

Post by lucb1e » Thu, Dec 20, 2018, 5:33 PM UTC

"serverless" is really the misnomer of the year.