Live data from Hacker News

Show HN: Octo – Generate a serverless API from an SQL query

octoproject.github.io

41–50 of 62 posts

Re: Show HN: Octo – Generate a serverless API from an SQL query

#41
post #24

Perhaps I'm old, but who needs an API for an SQL query? I'm not sure I understand the use case, or the advantage of something like this over a regular API call to a backend which would also allow you to do e.g. authentication. Enlighten me?

In my case, I have a simple obvious use case.

I work for a large corporation. They want to implement the Bezos mandate [1]. No direct database access between teams, API abstraction for everything.

OK, now let's think in onion layers (or hexagonal/clean architecture if you like). Think of layers of services with different purposes - data services (containing no application/business logic), application services (for business logic, orchestration, process), and UI/UX services (to power differing end user experiences).

Data services don't have to do much - be the data/repository layer, expose productive interfaces for CRUD. Need to read across data sources? Think of federated data services that can combine data on the fly, perhaps like GraphQL.

These kinds of tools are perfect for the first layer of services that abstract the database world from the application world. Just simple services, even ones that effectively let you mimic what SQL queries can do (filter, sort, page, etc.). Individual record oriented interfaces, and bulk oriented interfaces. The query side of CQRS (command query separation).

Many will say, "I don't need all this complexity and layers" - and sure, for smaller or simpler applications, probably not!

But, if you have to operate on any kind of larger scale, with multiple data sources, systems, etc., you end up needing the layers. And these types of tools automate some of the lower layers.

Perhaps when we talk about this, we shouldn't be focusing on "oh it's too complicated", and instead building frameworks or reference architectures that automate away the complexity - so it looks easy again, but now it is more flexible, perhaps easier to scale.

I believe that we are on the cusp still, of an almost fully defined, service based architecture (microservices and server less were just one part of the continuing development of that story). Federation is another part of that story oft ignored. Thinking of the onion as service layers is another part. Erasing the network boundary as a concern through much higher speed internetworking is another part.

Eventually we may come to see, that it is all a big "system", some parts just aren't connected to each other directly.

Sorry, got a bit rant-y at the end there :) Just passionate about sharing this world view with others - as I continue to see this architecture developing!

[EDIT] I wanted to add, it's not just that the use case for this is in a data service layer for automation - from a logical perspective I mean. In big companies such as the one I work for, we never get the resources we need, ironically. We are overwhelmed with demands, and must operate under the Bezos mandate rules. Tools such as Octo are not a panacea, but, they are a good compromise if you have to move fast, they are time-and-cost-savers. And they can get you surprisingly far.

[1] https://www.calnewport.com/blog/2018/09/18/the-human-api-man...

Re: Show HN: Octo – Generate a serverless API from an SQL query

#42
post #24

Perhaps I'm old, but who needs an API for an SQL query? I'm not sure I understand the use case, or the advantage of something like this over a regular API call to a backend which would also allow you to do e.g. authentication. Enlighten me?

One of the best use cases for this is say you have a backend/internal system and you want other things to start interacting with it. Instead of having to write the api to interface with it, you can just use something like this and with little effort you have an api and can talk with the database.

I think the point he was making is: why the API if you just want to talk to the db? You can connect to a SQL db over the network and protect the data with views and stored procedures.

Re: Show HN: Octo – Generate a serverless API from an SQL query

#43

Earlier quoted context omitted.

One of the best use cases for this is say you have a backend/internal system and you want other things to start interacting with it. Instead of having to write the api to interface with it, you can just use something like this and with little effort you have an api and can talk with the database.

I think the point he was making is: why the API if you just want to talk to the db? You can connect to a SQL db over the network and protect the data with views and stored procedures.

Because you can't (easily) connect to an SQL DB from any arbitrary client, like a web browser or mobile app.

Re: Show HN: Octo – Generate a serverless API from an SQL query

#44
post #40

Earlier quoted context omitted.

> In my case, I simply could not figure out how to use the Users table without having the password returned in all queries The recommended way of doing this is to store anything you don't want public in a separate table with a one-to-one relationship, and then controlling access to that table through computed columns and such ( https://www.graphile.org/postgraphile/postgresql-schema-desi... ) What I can't recommend e…

You’re completely right about the starter and I feel if I had discovered it first instead of trying to roll my own migration system with Knex and stuff, I would have stuck with Postgraphile. But instead of rebasing my project with the starter I decided to go back to what I was comfortable with and do a Rails gql api. I’ll definitely be trying it out in the future though! I felt so incredibly productive with rolling s…

Yes it's been incredibly quick for me to iterate and build my current project. You are still right about migrations though, I'm not sure how that would work in a bigger team with feature PRs etc.

Re: Show HN: Octo – Generate a serverless API from an SQL query

#45

Looks great! If you like this, check out OctoSQL[0]... Also in Go... Though OctoSQL lets you query multiple databases / files / event streams like kafka using SQL from your command line, not as a server, so a fairly different use case, but you should check it out nevertheless! The naming clash is funny. [0]: https://github.com/cube2222/octosql

I realy like your tool. In fact I am slowly integrating it into a solution which will expose a REST API and workspaces identified by a UUID. In our organisation it is so common to receive an Excel or csv which you have to join with the database. Octosql is great for that.

I am wondering what future role badger will play in the future? It would also make a great additional KV backend btw.

Re: Show HN: Octo – Generate a serverless API from an SQL query

#46

Earlier quoted context omitted.

I think the point he was making is: why the API if you just want to talk to the db? You can connect to a SQL db over the network and protect the data with views and stored procedures.

Because you can't (easily) connect to an SQL DB from any arbitrary client, like a web browser or mobile app.

That's actually exactly what we're trying to build at Splitgraph [0]. :) We're building a "data delivery network" (DDN), which is like a CDN, except instead of forwarding HTTP requests to upstream web servers, we forward SQL queries to upstream databases.

The premise of the idea is that we can cut out the middle-man for a lot of data distribution use cases. We give you a way to deliver your data in native SQL, using the Postgres wire protocol. We've decoupled authentication from the database, so we can do it in a gateway / LB layer using PgBouncer + Lua/Python scripting. Any SQL client can connect to the public Splitgraph endpoint (as far as a client is concerned, Splitgraph is just a really big Postgres database). You can write queries referencing and joining across any of the 40k datasets on the platform.

In fact, just this week we've been working on v0.0.0 of our web client. This lets you do things like share and embed SQL queries on Splitgraph, e.g. [1] (this query actually joins across two live data portals at data.cityofchicago.org and data.cambridgema.gov).

There's also an example here of using an Observable notebook with the Splitgraph REST API [2]. It also works with the Splitgraph DDN configured as a Postgres database, but that's only supported in private notebooks for now (since normally it's a bad idea to expose your DB to the public!)

In general, we like the idea of adding more logic to the database. Tools like OP's are useful in this regard. In fact, at Splitgraph we use Postgraphile internally (along with graphql-codegen for autogenerated types) and we have nothing but good things to say about it.

[0] https://www.splitgraph.com/

[1] https://www.splitgraph.com/workspace/ddn?layout=hsplit&query...

[2] https://observablehq.com/@mbostock/hello-splitgraph

Re: Show HN: Octo – Generate a serverless API from an SQL query

#47

I've got a similar project that reads your db schema and generates a Go REST API and a TypeScript/React web interface. (The code-generation is language agnostic so at some point I'd like to add at least a Java REST API as well.) It supports PostgreSQL, MySQL, and SQLite. Unlike PostgREST/Hasura and some other dynamic tools you can "eject" at this point if you'd like and continue on development without the generator i…

This is not dis-similar to what Strapi.io does, although I don't think they realize that's a big selling point from their marketing materials.

With strapi you configure your DB and get code generated in JS that supports a standard CRUD REST API. If you want to add business logic, you can override any particular endpoint you want. Their docs even come with the default implementation for easy copy/paste.

I would love to see research in this space continue, I think it's the future of bringing non-technical people into the product development process (if you can understand building a workflow with Excel/Google Sheets/Airtable, you can understand building an API). I'm excited to check out your project.

Re: Show HN: Octo – Generate a serverless API from an SQL query

#48

Looks great! If you like this, check out OctoSQL[0]... Also in Go... Though OctoSQL lets you query multiple databases / files / event streams like kafka using SQL from your command line, not as a server, so a fairly different use case, but you should check it out nevertheless! The naming clash is funny. [0]: https://github.com/cube2222/octosql

I realy like your tool. In fact I am slowly integrating it into a solution which will expose a REST API and workspaces identified by a UUID. In our organisation it is so common to receive an Excel or csv which you have to join with the database. Octosql is great for that. I am wondering what future role badger will play in the future? It would also make a great additional KV backend btw.

That's really great to hear!

We're considering moving to a more in-memory model, as we're not sure if the badger storage idea was a good one and worth it.

TBH we're still not quite sure in what direction we'll be continuing. Though we're surely gonna be developing it further.

But currently we're considering a rewrite with multiple assumptions changed (column oriented).

Re: Show HN: Octo – Generate a serverless API from an SQL query

#50

I've got a similar project that reads your db schema and generates a Go REST API and a TypeScript/React web interface. (The code-generation is language agnostic so at some point I'd like to add at least a Java REST API as well.) It supports PostgreSQL, MySQL, and SQLite. Unlike PostgREST/Hasura and some other dynamic tools you can "eject" at this point if you'd like and continue on development without the generator i…

In case anyone else is wondering what's doing the templating in this project, it's Scriban.

https://github.com/lunet-io/scriban

Post reply on HN