Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Went All In on sqlc/pgx for Postgres and Go

brandur.org

Post reply on HN
Print view View original

81–90 of 160 posts

Re: How We Went All In on sqlc/pgx for Postgres and Go

#81

Post by mohanmcgeek »

Fire-Dragon-DoL wrote: post #49
bitwize wrote: post #29

Earlier quoted context omitted.

> Why would someone author an ORM, painstakingly creating Go functions that just map to existing SQL features? The answer to this question lies in the assumption you make in this statement: > the one which every engineer already knows: SQL. Not every engineer knows, or wants to learn, SQL. I've met very competent engineers, SMEs over their particular system, who were flummoxed by SQL. And many more just want to work…

If you are using an ORM and need to write a query that will run on a SQL server you *need SQL knowledge and ORM knowledge*. If you are missing one the two, you probably have just wrote something with big performance penalties. This has been seen over and over in the Rails community.

Exactly my experience too.

The amount of SQL hatred from rails learning resources is unjustifiable.

If you're dealing with a relational database with SQL as its primary interface, you'll end up learning SQL eventually because all abstractions leak!

Re: How We Went All In on sqlc/pgx for Postgres and Go

#82

Post by benhoyt »

arwineap wrote: post #80
benhoyt wrote: post #50

Earlier quoted context omitted.

Are you sure? Well, it uses "reflection" in a general sense of introspecting your SQL code, but not in the Go sense of using using type information at runtime via the "reflect" package. sqlc compiles your SQL at build time to statically typed, non-reflect-using functions, as shown here: https://docs.sqlc.dev/en/stable/howto/select.html

It needs reflect in order to translate struct field names into column names at the very least

sqlc doesn't quite work that way (though "sqlx" and other packages do). sqlc generates code at build time that avoids reflect by using database/sql's Rows.Scan() with pointers to fields (see the link above):

  var i Author
  if err := rows.Scan(&i.ID, &i.Bio, &i.BirthYear); ...
This generated code is exactly what you'd write by hand if you were using database/sql directly.

As earthboundkid points out, database/sql itself may use reflection under the hood to convert the individual fields (though the common cases are done without reflect, using ordinary type switches: https://github.com/golang/go/blob/d62866ef793872779c9011161e...).

Re: How We Went All In on sqlc/pgx for Postgres and Go

#83

Post by benhoyt »

earthboundkid wrote: post #79
benhoyt wrote: post #50

Earlier quoted context omitted.

Are you sure? Well, it uses "reflection" in a general sense of introspecting your SQL code, but not in the Go sense of using using type information at runtime via the "reflect" package. sqlc compiles your SQL at build time to statically typed, non-reflect-using functions, as shown here: https://docs.sqlc.dev/en/stable/howto/select.html

Technically, reflection is used by Go’s database Scanner interface, but it’s not what most people think of when they complain about reflection.

Good point. I had assumed Rows.Scan() would have just used type switches for efficiency -- it looks like it does for common cases (https://github.com/golang/go/blob/d62866ef793872779c9011161e...) but then falls back to reflect. I wonder why it doesn't just do all of that with type switches? Maybe there are just too many cases and it ends up slower than reflect for the rest of the cases.

Scanner.Scan() is actually just called via a type assertion, though I guess implementations of Scan() might use reflection.

Re: How We Went All In on sqlc/pgx for Postgres and Go

#84

Post by hactually »

sam0x17 wrote: post #44
wvenable wrote: post #33

Earlier quoted context omitted.

There's plenty of non-trival code written in C as well. That's not a good argument for the benefit of a programming language. You can work around any limitation with enough work -- this article is a perfect example. It's an ugly solution to a simple problem but it works.

C has few enough restrictions though that you can for example make a struct and then make an array of that struct. In Go this is like rocket science.

Which one are you struggling with?

https://play.golang.org/p/P8L0lSMhNgF

https://play.golang.org/p/E8rM7JdrfkD

Re: How We Went All In on sqlc/pgx for Postgres and Go

#85

Post by gravypod »

I attempted to make something similar to this except the opposite direction at a previous job. It was called Pronto: https://github.com/CaperAi/pronto/

It allowed us to store and query Protos into MongoDB. It wasn't perfect (lots of issues) but the idea was rather than specifying custom models for all of our DB logic in our Java code we could write a proto and automatically and code could import that proto and read/write it into the database. This made building tooling to debug issues very easy and make it very simple to hide a DB behind a gRPC API.

The tool automated the boring stuff. I wish I could have extended this to have you define a service in a .proto and "compile" that into an ORM DAO-like thing automatically so you never need to worry about manually wiring that stuff ever again.

Re: How We Went All In on sqlc/pgx for Postgres and Go

#86

Post by hactually »

sam0x17 wrote: post #46
hactually wrote: post #15

Earlier quoted context omitted.

They're really not a `must`. What a silly comment - Docker and Kubernetes and substantial parts of Google wouldn't be classed as trivial. For the thousands of devs shipping non-trivial code, keep going!

> Docker and Kubernetes Both of these projects have had to go way out of their way to make things work without generics, but they are large enough projects and have enough resources that they can do this. Both are actually really great examples of why Go is a bad choice until generics are added and have first-class support. Even C would be preferable over something where there are no generics.

You're incorrect but that's OK.

I've been writing Golang for 5 years and have never had need for generics.

It's nothing to with resources, it's just understanding how to build software.

Your retreat to C is a bit sad - I'd be happy to help you if you've got any Go you're struggling with?

Re: How We Went All In on sqlc/pgx for Postgres and Go

#87

Post by scrubs »

shriek wrote: post #35
bitwize wrote: post #29

Earlier quoted context omitted.

> Why would someone author an ORM, painstakingly creating Go functions that just map to existing SQL features? The answer to this question lies in the assumption you make in this statement: > the one which every engineer already knows: SQL. Not every engineer knows, or wants to learn, SQL. I've met very competent engineers, SMEs over their particular system, who were flummoxed by SQL. And many more just want to work…

> Not every engineer knows, or wants to learn, SQL. Which is bizarre cause you pretty much need some form of RDBs in most of the apps. And because of ANSI SQL, the syntax/concepts are relatively same on different databases too. No point in not making this investment.

Agree. This is the weird kid down the street who gets by on manyioise and saltines. To engage persistent storage is to engage sql for the first 75% of all work. Hey you gotta have some competence in the domain of work. I like Jordache (orm) but not Calvin Klein (sql) isn't wisdom; it's merely personal predilection.

Re: How We Went All In on sqlc/pgx for Postgres and Go

#88

Post by jrockway »

hactually wrote: post #86
sam0x17 wrote: post #46

Earlier quoted context omitted.

> Docker and Kubernetes Both of these projects have had to go way out of their way to make things work without generics, but they are large enough projects and have enough resources that they can do this. Both are actually really great examples of why Go is a bad choice until generics are added and have first-class support. Even C would be preferable over something where there are no generics.

You're incorrect but that's OK. I've been writing Golang for 5 years and have never had need for generics. It's nothing to with resources, it's just understanding how to build software. Your retreat to C is a bit sad - I'd be happy to help you if you've got any Go you're struggling with?

Kubernetes does in fact need generics. There are types that all work the same but are different types all over the place. For example, you might make a reflector for a v1.Node, and that will call methods on your cache that are like Add(v1.Object), when the signatures should really be v1.Node. Instead, you have to cast the API to your internal implementation. It's not a big deal, but it's a solid example of something that generics would clean up.

Re: How We Went All In on sqlc/pgx for Postgres and Go

#89

Post by sa46 »

conroy wrote: post #72
sa46 wrote: post #71

Earlier quoted context omitted.

Yep, that’s the main downside. pggen works out of the box with Docker under the hood if you give it some schema files. Notably, the recommended way to run sqlc also requires Docker. I check in the generated code so I only run pggen at dev time, not build time. I do intend to move to build time codegen with Bazel but I built out tooling to launch new instances of Bazel managed Postgres in 200 ms so not that painful. M…

> Notably, the recommended way to run sqlc also requires Docker. This isn't accurate. sqlc is a self-contained binary and does not have any dependencies. Docker is one of the many ways to install and run it, but it is not required. (author of sqlc)

Oh hey! Apologies, not trying to throw shade. Last time I used sqlc was a while ago and I ran it in docker.

Re: How We Went All In on sqlc/pgx for Postgres and Go

#90

Post by rapnie »

conroy wrote: post #32

Author of sqlc here. Just wanted to say thanks to everyone in this thread. It's been a really fun project to work on the last two years. Excited to get to work on adding support for more databases and programming languages.

Thanks a lot for this great project. I looked in the issues for Sqlite support and saw the merge of PR to "Add three new experimental engines, including SQLite" [0] and there's major architecture changes involved. That merge was 1.5 yr ago though, and I am curious what the plans are to take that further.

[0] https://github.com/kyleconroy/sqlc/pull/331

Post reply on HN