Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a High Performance Data Integration Framework in Go

cloudquery.io

Post reply on HN
Print view View original

11–20 of 32 posts

Re: Building a High Performance Data Integration Framework in Go

#11

Post by yevpats »

elchief wrote: post #10

Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?

Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here (https://www.cloudquery.io/docs/cq-vs-others/overview).

There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this question will come up more and we will add those products as well to our comparison list.

The protocol is just standard/plain gRPC (https://www.cloudquery.io/docs/developers/architecture). Protobuf is defined here - https://github.com/cloudquery/plugin-sdk/tree/main/internal/...

Basically just the normalized, verified data + metadata, types

Re: Building a High Performance Data Integration Framework in Go

#12

Post by elchief »

yevpats wrote: post #11
elchief wrote: post #10

Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?

Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here ( https://www.cloudquery.io/docs/cq-vs-others/overview ). There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this ques…

understood :) thanks for the reply

Re: Building a High Performance Data Integration Framework in Go

#13

Post by StefanWestfal »

yevpats wrote: post #2

Awesome to see this on HN! Founder/Author here. I'll be happy to answer questions.

Thank you for putting your time into the project!

Besides this project I like sqlc a lot but in both scenarios I was wondering how one gets started in code generation tooling? So that I could contribute, extend or even create my own (I.e. I would like to extend sqlc for better integration with FastAPI).

Re: Building a High Performance Data Integration Framework in Go

#14

Post by yevpats »

StefanWestfal wrote: post #13
yevpats wrote: post #2

Awesome to see this on HN! Founder/Author here. I'll be happy to answer questions.

Thank you for putting your time into the project! Besides this project I like sqlc a lot but in both scenarios I was wondering how one gets started in code generation tooling? So that I could contribute, extend or even create my own (I.e. I would like to extend sqlc for better integration with FastAPI).

Thanks! I think the best place for code generation in Go would be: 1) Go templating: https://pkg.go.dev/text/template 2) Go reflection: https://go.dev/blog/laws-of-reflection (there are many articles and documentation on this).

Then I would try to do something small to experience both libraries hands-on as otherwise it's hard to get the hang of it. Once you did a small project I would go and try to contribute a small fix/feat to sqlc to understand sqlc code structure and from there potentially a bigger feature/extension for sqlc.

Re: Building a High Performance Data Integration Framework in Go

#15

Post by leetbulb »

Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

Check out Ent https://entgo.io/docs/code-gen

Pretty easy to generate GraphQL (most fully featured extension), OpenAPI, Protobuf, etc. from your database schema. Ent also makes it easy to implement your own generators (e.g. OAS, glue logic, etc).

Re: Building a High Performance Data Integration Framework in Go

#16

Post by stuzenz »

zinclozenge wrote: post #6
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

Gongular looks pretty cool, but it seems like development stopped. Is there any other library that does similar things? I love learning about new frameworks/libraries in golang for exactly that reason of reducing boilerplate and spending more time on logic.

Interesting response here in the issues.

The project is still active. It seems that feature requests and issues are not being asked of the project.

https://github.com/mustafaakin/gongular/issues/21

Re: Building a High Performance Data Integration Framework in Go

#17

Post by ilyt »

Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

It's real shame it's kinda bad language for it. Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments"

And I say it without being sarcastic but Go makes me miss C preprocessor and nothing should make anyone miss C preprocessor.

Re: Building a High Performance Data Integration Framework in Go

#18

Post by tomrod »

yevpats wrote: post #11
elchief wrote: post #10

Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?

Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here ( https://www.cloudquery.io/docs/cq-vs-others/overview ). There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this ques…

Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss.

Re: Building a High Performance Data Integration Framework in Go

#19

Post by rad_gruchalski »

ilyt wrote: post #17
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

It's real shame it's kinda bad language for it. Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments " And I say it without being sarcastic but Go makes me miss C preprocessor and nothing should make anyone miss C preprocessor.

> put code instructing the compiler to do stuf in fucking comments

Funny considering how Rust does stuff.

Both approaches have pros and cons. One either digs through a shitload of macros, or commits explicit auto generated code.

Re: Building a High Performance Data Integration Framework in Go

#20

Post by teeray »

ilyt wrote: post #17
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

It's real shame it's kinda bad language for it. Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments " And I say it without being sarcastic but Go makes me miss C preprocessor and nothing should make anyone miss C preprocessor.

It’s like metaprogramming in Ruby again, except with the idea that it’s somehow more straightforward because you can inspect the generated code. Except nobody will ever inspect auto-generated code because a) it’s usually awful code b) ignoring that code was kind of the whole point.
Post reply on HN