Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a High Performance Data Integration Framework in Go

cloudquery.io

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Building a High Performance Data Integration Framework in Go

#21

Post by gmhafiz »

Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

Sqlc quickly breaks down when you want to do dynamic select. This is where an orm, or query builder shines

Re: Building a High Performance Data Integration Framework in Go

#22

Post by Xeoncross »

leetbulb wrote: post #15
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

Check out Ent https://entgo.io/docs/code-gen Pretty easy to generate GraphQL (most fully featured extension), OpenAPI, Protobuf, etc. from your database schema. Ent also makes it easy to implement your own generators (e.g. OAS, glue logic, etc).

The problem with Ent is you are neither 1) writing your SQL structures or 2) writing your Go structures. Instead, you're writing Ent-specific structures.

It's still a good option, just sad we keep inventing more abstraction layers that are unique in each language. That is a benefit of GraphQL typedefs, JSON, and SQL - it's the same in every language.

Re: Building a High Performance Data Integration Framework in Go

#23

Post by User23 »

Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

It's nice that they exist, but these workarounds are only necessary because Go isn't a very expressive language. I don't really consider having to use a third party tool to generate boilerplate as part of a multi-stage build process to really be a great thing, but I guess it's better than not having those tools if you choose to use Go for some reason.

Re: Building a High Performance Data Integration Framework in Go

#24

Post by stereosteve »

gmhafiz wrote: post #21
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

Sqlc quickly breaks down when you want to do dynamic select. This is where an orm, or query builder shines

Agreed. I am experimenting with using go-jet instead and it’s going real good so far.

https://github.com/go-jet/jet

Re: Building a High Performance Data Integration Framework in Go

#25

Post by sghosh2 »

elchief wrote: post #10

Seems like strange choices for "CloudQuery vs Others". Why not compare against FiveTran, Airbyte, Meltano or other EL tools? Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?

Much more of a focus on security and infrastructure than other EL tools. Like these security policies for example - https://www.cloudquery.io/docs/core-concepts/policies

Re: Building a High Performance Data Integration Framework in Go

#26

Post by yevpats »

tomrod wrote: post #18
yevpats wrote: post #11

Earlier quoted context omitted.

Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here ( https://www.cloudquery.io/docs/cq-vs-others/overview ). There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this ques…

Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss .

Interesting. We will take a look at this even though we usually build our own connectors due the huge concurrency/speed advantages when using CloudQuery SDK.

Re: Building a High Performance Data Integration Framework in Go

#28

Post by philosopher1234 »

laerus wrote: post #27

So golang macros when? Honestly the language is the new JS and just another one to remind us how miserable software can be with half baked enterprizy solutions to just get the job done faster.

This is a lot of confidence for an opinion that doesn't seem very thought through. There are a lot of downsides to macros, and a lot of (apparently under-appreciated) upsides to code generation.

Re: Building a High Performance Data Integration Framework in Go

#29

Post by philosopher1234 »

ilyt wrote: post #17
Xeoncross wrote: post #4

Code generation is becoming really important in Go. Why even use an ORM when https://sqlc.dev/ will generate everything from vanilla SQL? Why make the frontend team write a Typescript client when https://goa.design on the backend will produce an OpenAPI schema they can just point a https://openapi-generator.tech at? Why write out GraphQL boilerplate when https://github.com/99designs/gqlgen will take your GQL typedef…

It's real shame it's kinda bad language for it. Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments " And I say it without being sarcastic but Go makes me miss C preprocessor and nothing should make anyone miss C preprocessor.

>Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments"

Why is it a shame? Rely on macros and this is what you get:

1. Slow compile times

2. Unsearchable (grep, sourcegraph...) code

3. Magic codebases. Longer learning curve

To me, working in a large organization, those are big downsides.

Re: Building a High Performance Data Integration Framework in Go

#30

Post by ilyt »

rad_gruchalski wrote: post #19
ilyt wrote: post #17

Earlier quoted context omitted.

It's real shame it's kinda bad language for it. Looking at what people did with Rust macros it's shame that Go code generation story is either "just run some random binaries to compile stuff" or "put code instructing the compiler to do stuf in fucking comments " And I say it without being sarcastic but Go makes me miss C preprocessor and nothing should make anyone miss C preprocessor.

> put code instructing the compiler to do stuf in fucking comments Funny considering how Rust does stuff. Both approaches have pros and cons. One either digs through a shitload of macros, or commits explicit auto generated code.

Yeah the annotations in Rust suffer from bit of that but macros don't.

I yearn for ability to make error handling macros...

Post reply on HN