Show HN: Connect Pub/Sub to BigQuery with an open source Dataflow pipeline

#1

Post by homosaphien » Wed, Aug 09, 2017, 2:46 PM UTC

Show HN: Connect Pub/Sub to BigQuery with an open source Dataflow pipeline
github.com

Re: Show HN: Connect Pub/Sub to BigQuery with an open source Dataflow pipeline

#2

Post by craptocurrency » Wed, Aug 09, 2017, 3:51 PM UTC

Awesome job bro!

Re: Show HN: Connect Pub/Sub to BigQuery with an open source Dataflow pipeline

#3

Post by BrandonY » Wed, Aug 09, 2017, 5:12 PM UTC

Very cool! I like how configurable your solution is.

Google has an example app (https://github.com/GoogleCloudPlatform/kubernetes-bigquery-p...) that demonstrates getting Pub/Sub data into BigQuery, but it just does it directly as a little Kubernetes job rather than using Dataflow. I like your more serverless solution.

Re: Show HN: Connect Pub/Sub to BigQuery with an open source Dataflow pipeline

#4

Post by homosaphien » Wed, Aug 09, 2017, 6:46 PM UTC

BrandonY wrote: post #3

Very cool! I like how configurable your solution is. Google has an example app ( https://github.com/GoogleCloudPlatform/kubernetes-bigquery-p... ) that demonstrates getting Pub/Sub data into BigQuery, but it just does it directly as a little Kubernetes job rather than using Dataflow. I like your more serverless solution.

Thank you for the feedback.