Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ent: An Entity Framework for Go

github.com

Post reply on HN
Print view View original

11–20 of 68 posts

Re: Ent: An Entity Framework for Go

#11

Post by jitl »

Fire-Dragon-DoL wrote: post #4
jitl wrote: post #3

I think Ent for Go has a ton of potential. Although I haven’t used this library, I have spent a lot of time studying the design space of ORMs because I’m currently iterating on an internal library that does ORM-like things at Notion. I really like the Ent approach because it allows working with a graph-based data model directly in the embedding language, instead of forcing developers to learn a new query language lik…

I keep thinking that the query portion should be entirely split from the ORM portion. Query builder have specific usage, but are not the norm. None of them can model an advanced postgres query, but at the same time, they provide little (or nothing) utilities to map the data coming from a complex query, back to objects, which is funny, to me that would be the definition of "object relational mapper". I know why active…

In a typical ORM I agree about the query builder. Most ORMs encourage the typical use of a relational database where SQL fits the domain well.

But Ent is modeling data as a graph, and the DB behind the scenes is more of an implementation detail. The production version of Ent used at Meta stores data in TAO (https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... my impression is that product engineers do not write any SQL when working with Ent at Meta.

Re: Ent: An Entity Framework for Go

#12

Post by Fire-Dragon-DoL »

jitl wrote: post #11
Fire-Dragon-DoL wrote: post #4

Earlier quoted context omitted.

I keep thinking that the query portion should be entirely split from the ORM portion. Query builder have specific usage, but are not the norm. None of them can model an advanced postgres query, but at the same time, they provide little (or nothing) utilities to map the data coming from a complex query, back to objects, which is funny, to me that would be the definition of "object relational mapper". I know why active…

In a typical ORM I agree about the query builder. Most ORMs encourage the typical use of a relational database where SQL fits the domain well. But Ent is modeling data as a graph, and the DB behind the scenes is more of an implementation detail. The production version of Ent used at Meta stores data in TAO ( https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... my impression is that product engineers do not…

I might be misreading here, but are you suggesting that the DB behind the scenes is not modeling things in the traditional sense but rather as a graph. Thinking of postgres, this could be by dumping a big json object in a generic "graph" table. Is that the case?

That would be acceptable from my point of view. The problem is using SQL as a graph produces very inefficient queries once hitting a certain scale, but if the DB is already a graph, the problem should be limited.

Re: Ent: An Entity Framework for Go

#13

Post by jitl »

rowanG077 wrote: post #9
valenterry wrote: post #7

Earlier quoted context omitted.

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and h…

Totally agree with you. Especially about point 1. Codegen is one of the bluntest tools there is and should be almost always avoided.

Why is that? I’ve done a few toy attempts to build out ORM-like model classes in Typescript that wrap an input schema type using typelevel features, and my profiles of the Typescript compiler show a lot of slowdowns coming from combinatoric explosions in the typechecker. In this language specifically, the type system has quite advanced expressiveness but using those features in practice runs into limitations of the checker’s implementation frequently, and more so once you have many input types. We have ~80 table types and often a single table type is a union of variants - our biggest, the type for a Notion block is a union with 64 variants. I wrote a type that takes such row type as input, and outputs an OO-style Model interface for it with accessor methods, relational shortcuts, etc, and a matching Proxy that does the same at runtime. It works, and for smaller types seems like a fine solution. But in practice all the magic gums up Typescript’s inference and makes the language service crash into memory limits. Or the compiler just gives up and says “this union type is too complex to represent”.

If I move the same type magic into codegen, the compiler is much happier; plus stack traces lead directly to a simple method implementation instead of jumping into difficult-to-follow proxy machinery. The downside is that the codegen logic that consumes the Typescript compiler API to understand the schema types is more complex than the mapped types before, but it does make it feasible to have my cake (derive an ORM/model API from a schema) and eat it too (not make the compiler so unstable and touchy).

Re: Ent: An Entity Framework for Go

#14

Post by jitl »

Fire-Dragon-DoL wrote: post #12
jitl wrote: post #11

Earlier quoted context omitted.

In a typical ORM I agree about the query builder. Most ORMs encourage the typical use of a relational database where SQL fits the domain well. But Ent is modeling data as a graph, and the DB behind the scenes is more of an implementation detail. The production version of Ent used at Meta stores data in TAO ( https://engineering.fb.com/2013/06/25/core-data/tao-the-powe... my impression is that product engineers do not…

I might be misreading here, but are you suggesting that the DB behind the scenes is not modeling things in the traditional sense but rather as a graph. Thinking of postgres, this could be by dumping a big json object in a generic "graph" table. Is that the case? That would be acceptable from my point of view. The problem is using SQL as a graph produces very inefficient queries once hitting a certain scale, but if th…

The way TAO (and Notion to some degree) deal with this is by tightly coupling the persisted (SQL?) data store to an in-memory cache that make KV style reads like “get foo by foo_id” extremely fast. They also limit the kinds of queries developers can write to the ones more likely to be very fast on the infrastructure.

Pushing graph queues down to a single SQL query so that the SQL DB spends massive CPU time on joins can be an issue. Instead these systems can avoid talking to the SQL DB at all for example if they’re on the happy path just chasing graph edges.

Re: Ent: An Entity Framework for Go

#15

Post by ivanvanderbyl »

metadat wrote: post #2

I thought this looked familiar.. Discussed 2 years ago: https://news.ycombinator.com/item?id=26008521 . (39 points, 9 comments)

I first heard about Ent on HN about 2 years ago. At the time I was evaluating every ORM in Go to find something that would fit the bill and Ent happened to be exactly what we needed. We've now been using it for 2 years and it's been one of the best bets we made on a technology choice.

Things that have worked really well:

* Auto DB migrations using Ent+Atlas. Ent implements a lot of great low level defaults that I just don't want to need to think about (foreign key constraints, indexes, naming of join tables etc)

* Generating our GraphQL API from Ent Schema

* Generating Protobuf definitions for internal tools to talk to (now using Buf for the actual tooling, but having the protos generated saved a huge lot of time).

* Being able to quickly craft really complex multi-edge joins without really thinking through the SQL allows for quick implementation of new features.

* Query optimizations such as using WINDOW clauses for pagination in GraphQL queries (I wouldn't have even thought this was possible).

* The generated code is quite a lot of lines, but it's really nicely structured and idiomatic, making it easy to extend.

There's been heaps of other neat finds along the way, but that's a summary.

Shout out to Ariel and Rotem for being excellent stewards of the Ent community and helping us solve some complex problems along the way.

Re: Ent: An Entity Framework for Go

#16

Post by Fire-Dragon-DoL »

jitl wrote: post #14
Fire-Dragon-DoL wrote: post #12

Earlier quoted context omitted.

I might be misreading here, but are you suggesting that the DB behind the scenes is not modeling things in the traditional sense but rather as a graph. Thinking of postgres, this could be by dumping a big json object in a generic "graph" table. Is that the case? That would be acceptable from my point of view. The problem is using SQL as a graph produces very inefficient queries once hitting a certain scale, but if th…

The way TAO (and Notion to some degree) deal with this is by tightly coupling the persisted (SQL?) data store to an in-memory cache that make KV style reads like “get foo by foo_id” extremely fast. They also limit the kinds of queries developers can write to the ones more likely to be very fast on the infrastructure. Pushing graph queues down to a single SQL query so that the SQL DB spends massive CPU time on joins c…

That makes sense. In a certain way, it's doing something similar to react-query

Re: Ent: An Entity Framework for Go

#17

Post by jitl »

valenterry wrote: post #7
jitl wrote: post #3

I think Ent for Go has a ton of potential. Although I haven’t used this library, I have spent a lot of time studying the design space of ORMs because I’m currently iterating on an internal library that does ORM-like things at Notion. I really like the Ent approach because it allows working with a graph-based data model directly in the embedding language, instead of forcing developers to learn a new query language lik…

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and h…

None of the industrial languages I’ve looked at (Kotlin, Typescript, Go, Java, Swift, Rust) can do the things I want to do using only typelevel features, especially at practical scale. Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining.

Often in Java/Kotlin projects people end up writing compiler plugins / annotation processors which is like moving the codegen to build time and making it harder to inspect/understand, while simultaneously spending more developer time on it since those systems may need to run for every compiler invocation. The same goes for Rust; rust’s various macro systems target the problem I have, but Rust users report long compile times due to macro magic. More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons.

Of course there’s always the runtime only approach in Ruby/ActiveRecord but that runs into correctness problems at practical scales.

Codegen is usually brittle and is often ugly, but is guaranteed to have more metaprogramming power than in-language typelevel features. Ultimately any such system can be complex, I just hope to build one where the leverage I payed for with complexity is well worth it.

Re: Ent: An Entity Framework for Go

#18

Post by syvolt »

I've been using Ent for some time on a project and its been quite nice to just be able to write the schema in Go, testing has been a breeze with the enttest package, hooks work well, and everything feels intuitive to me unlike most other ORMs or ORM-adjacent tools.

My preferred package before Ent was Squirrel [1] but I definitely plan to use Ent for future projects.

[1] https://github.com/Masterminds/squirrel

Re: Ent: An Entity Framework for Go

#20

Post by valenterry »

jitl wrote: post #17
valenterry wrote: post #7

Earlier quoted context omitted.

> 1. Codegen is powerful, and often easier to understand than a mountain of typelevel magic. I disagree here. And I also don't like your framing. Yes, type-level logic is harder to learn than understanding code you already read everyday. It's also easier to count and calculate with your fingers and use concrete examples rather than learning abstract math. But we still do it, because in the end you learn it once and h…

None of the industrial languages I’ve looked at (Kotlin, Typescript, Go, Java, Swift, Rust) can do the things I want to do using only typelevel features, especially at practical scale. Eg, transform a schema type that describes my database tables into an easy-to-use query builder and data abstraction API with rich method chaining. Often in Java/Kotlin projects people end up writing compiler plugins / annotation proce…

> More esoteric languages like Scala and Haskell might have the expressive power I want, but don’t seem practical to implement for other reasons.

Well, if you can't use languages that make it possible/feasible, then code generation might indeed be the best option. It's just that your claim sounded quite general, so I had to jump in. :)

Post reply on HN