Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dbdev – A database package manager for PostgreSQL trusted language extensions

database.dev

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#31

Post by jerrysievert »

LunaSea wrote: post #29
jerrysievert wrote: post #21

Earlier quoted context omitted.

wow, thanks for the compliment! it hasn't always been easy, believe me.

Thank you for the amazing work indeed! I know that at one point the idea of using a non-v8 JavaScript engine was floating around, due in part to the V8 build complexity and the API breaking changes. Is this still something that is up in the air? I know that Bun decided to use JavaScriptCore as a recent example.

I have been slowly but surely working on pljs in my spare time. it is nowhere near feature parity with plv8, but I hope to get it closer with more time. currently missing:

    * query execution/cursors
    * window functions
    * sub transactions
it is based on quickjs, is very fast/easy to compile, and while in long-running functions it is slower than v8, it has a very fast startup time that negates that for shorter functions.

but, it's not ready for prime time yet.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#32

Post by mike_hearn »

nijave wrote: post #17
mike_hearn wrote: post #14

At KotlinConf today I gave a talk on designing apps with two-tier architecture, where you implement your entire app without the web stack appearing anywhere at all. Instead you publish desktop and mobile apps that connect directly to an RDBMS like PostgreSQL via its native protocol, and use server extensions for any logic that is inconvenient to do with SQL. This approach might seem horrifyingly outside-the-box but h…

Are there any performance issues with connection handling? Not sure how/where a connection pooler fits in.

This is also discussed in the talk, which I hope will be on YouTube soon enough (it was live streamed).

DB connections are heavier than web server connections, in particular in terms of memory (when using postgres, not sure about mysql). Multiplexers like pgbouncer can address this, but then you lose a bit of the deployment simplicity. Fortunately RAM is pretty cheap nowadays, you can configure the DB to disconnect idle connections, you can use multiplexers and you can make the client transparently reconnect when a connection is needed. So it turns into a RAM/user latency tradeoff.

There are some benefits though. Native DB protocols will tend to stream results in binary, avoiding a JSON encoding that can expand the bandwidth requirements. Also you can get rid of the server side load balancing, because good db drivers know how to load balance amongst R/O replicas already. A lot of deployment complexity in web apps is driven by the fact that browsers handle errors by just throwing up an error page and stopping, so you need to avoid even tiny amounts of downtime. When you control the client you can do failover and recovery more gracefully, which in turn relaxes the availability constraints on the servers a bit.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#33

Post by mike_hearn »

jerrysievert wrote: post #21
claytongulick wrote: post #19

I'd love to see plv8 added to this. Jerry S. has done an amazing job of wrangling the insane v8 build process into something that mostly works across platforms, but installation is still not particularly simple. Unfortunately, due to the problems with that build process, Ubuntu and other flavors dropped the prebuilt apt packages. Making plv8 a simple install would be a huge boost to the community.

wow, thanks for the compliment! it hasn't always been easy, believe me.

That's interesting - could you describe what makes the build and install so hard? I can't think of any obvious reason why it should be harder than PL/Java. V8 is pretty widely used!

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#34

Post by jerrysievert »

mike_hearn wrote: post #33
jerrysievert wrote: post #21

Earlier quoted context omitted.

wow, thanks for the compliment! it hasn't always been easy, believe me.

That's interesting - could you describe what makes the build and install so hard? I can't think of any obvious reason why it should be harder than PL/Java. V8 is pretty widely used!

v8 has its own build chain, which involves ~5gb of downloads. that build chain doesn't work on many platforms, and requires very specific versions of prerequisites including specific glibc versions.

the build chain consists of specific compilers, specific versions of chromium parts and pieces, its own sys root, everything.

since the API for v8 changes on a regular basis (not as much as it has in the past, at least), you cannot simply change out v8 versions, as they likely will not work.

v8 is also compiled differently for each embedding application, and thus a shared object won't work correctly across multiple applications, and besides is strongly advised against by the v8 team.

nodejs has its own build process that they've been maintaining since the v8 3.1.4 days, but it is specific to node.

so, it just adds up to a difficult maintenance.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#35

Post by mike_hearn »

saghm wrote: post #27
mike_hearn wrote: post #14

At KotlinConf today I gave a talk on designing apps with two-tier architecture, where you implement your entire app without the web stack appearing anywhere at all. Instead you publish desktop and mobile apps that connect directly to an RDBMS like PostgreSQL via its native protocol, and use server extensions for any logic that is inconvenient to do with SQL. This approach might seem horrifyingly outside-the-box but h…

> This approach might seem horrifyingly outside-the-box but has a lot of advantages, and some of the reasons we didn't do things this way historically have been solved in recent years. From a development standpoint, I definitely think this seems easier! My concerns about this would be almost entirely about security. How does this approach avoid leaking credentials that people could use to access the database outside…

See my other post for a discussion of non-user security topics.

Every app user has a database user, mapped 1:1, so there are no credentials to leak. The DB credentials are the user credentials. In the demo app repository I implement open signup with email confirmation using stored procedures/a server extension. It works by creating a guest user with a well known password, but it's locked down so that the only things it can do are run a couple of stored procedures. Those then send an email to confirm ownership of the address before creating a real DB user.

Open work: don't use passwords, mint SSL client certificates during signup with the private key stored in the OS keystore. Now there's no cookies or passwords to steal! Also, OAuth integration.

The guest user / open signup is certainly a weakness for the demo, which is public on the internet (a dedicated cluster though). It's locked down as much as possible but there are probably lots of ways to screw with the server even without being granted access to anything. And there are no rate limits on signup. It's really a very basic exploration.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#36

Post by mike_hearn »

jerrysievert wrote: post #34
mike_hearn wrote: post #33

Earlier quoted context omitted.

That's interesting - could you describe what makes the build and install so hard? I can't think of any obvious reason why it should be harder than PL/Java. V8 is pretty widely used!

v8 has its own build chain, which involves ~5gb of downloads. that build chain doesn't work on many platforms, and requires very specific versions of prerequisites including specific glibc versions. the build chain consists of specific compilers, specific versions of chromium parts and pieces, its own sys root, everything. since the API for v8 changes on a regular basis (not as much as it has in the past, at least),…

Wow. That's kind of surprising. HotSpot is just autotools: ./configure && make and you're done. Plus the embedding APIs are stable.

Do you have any insight into what led them to have such a complex build? Is it just because it's a part of the Chromium tree? It doesn't seem inherent to the problem space.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#37

Post by thewataccount »

mmmeff wrote: post #10

What's stopping me from adding an extension to this repository that adds something valuable while simultaneously exfiltrating entire databases back to me? It's a cool idea, but I won't be the first to try it. You're placing an awful lot of trust in these modules.

Nothing? It's a classic supply chain attack.

The same goes for pypi, npm, crates.io, docker.com, github, chrome extensions, etc.

Generally it comes down to trusted authors/package names. Many people trust `requests` from pypi. That being said even this doesn't prevent the package to be sold, or hacked, or from the owner later adding things, or a 3rd party sneaking something in via a PR. Some like docker (and technically most git based ones because of md5) can have specific tags changed at any time, so you could later add a malicious package to version 3.0.4 or whatever.

Many package managers (mostly linux ones) include key-signing, some package managers also have built-in scanners. Preferably you proxy/cache versions you need, and have a way to audit version upgrades - but in reality it's a lot of work to do properly.

Recent example of `pytorch` nightly on pypi accidentally having a dependancy named `torchtriton` that I think was name squatting? Anyway installing the pytorch nightly between December 25th to 30th would result in your home directory being uploaded, etc. https://www.bleepingcomputer.com/news/security/pytorch-discl...

EDIT: That said if anyone from supabase reads this - I'd love to see GPG signing, version hashing/integrity verification, 2FA if it's not their already, and I know this one is a lot of extra work - but any type of automated even cursory glance for malicious code.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#38

Post by jerrysievert »

mike_hearn wrote: post #36
jerrysievert wrote: post #34

Earlier quoted context omitted.

v8 has its own build chain, which involves ~5gb of downloads. that build chain doesn't work on many platforms, and requires very specific versions of prerequisites including specific glibc versions. the build chain consists of specific compilers, specific versions of chromium parts and pieces, its own sys root, everything. since the API for v8 changes on a regular basis (not as much as it has in the past, at least),…

Wow. That's kind of surprising. HotSpot is just autotools: ./configure && make and you're done. Plus the embedding APIs are stable. Do you have any insight into what led them to have such a complex build? Is it just because it's a part of the Chromium tree? It doesn't seem inherent to the problem space.

they use their own build systems for everything, even skia. though skia has a bit less strict build requirements.

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#39

Post by conceptme »

mike_hearn wrote: post #35
saghm wrote: post #27

Earlier quoted context omitted.

> This approach might seem horrifyingly outside-the-box but has a lot of advantages, and some of the reasons we didn't do things this way historically have been solved in recent years. From a development standpoint, I definitely think this seems easier! My concerns about this would be almost entirely about security. How does this approach avoid leaking credentials that people could use to access the database outside…

See my other post for a discussion of non-user security topics. Every app user has a database user, mapped 1:1, so there are no credentials to leak. The DB credentials are the user credentials. In the demo app repository I implement open signup with email confirmation using stored procedures/a server extension. It works by creating a guest user with a well known password, but it's locked down so that the only things…

Connections are pretty expensive in PG how can you do pooling if every user connects directly?

Re: Dbdev – A database package manager for PostgreSQL trusted language extensions

#40

Post by michelpp »

thewataccount wrote: post #37
mmmeff wrote: post #10

What's stopping me from adding an extension to this repository that adds something valuable while simultaneously exfiltrating entire databases back to me? It's a cool idea, but I won't be the first to try it. You're placing an awful lot of trust in these modules.

Nothing? It's a classic supply chain attack. The same goes for pypi, npm, crates.io, docker.com, github, chrome extensions, etc. Generally it comes down to trusted authors/package names. Many people trust `requests` from pypi. That being said even this doesn't prevent the package to be sold, or hacked, or from the owner later adding things, or a 3rd party sneaking something in via a PR. Some like docker (and technica…

> That said if anyone from supabase reads this - I'd love to see GPG signing, version hashing/integrity verification, 2FA if it's not their already

Yep, we've got all these on the road map, thanks!

> and I know this one is a lot of extra work - but any type of automated even cursory glance for malicious code.

Unfortunately a "cursory glance" is trivially sidestepped, and anything complete enough to be useful is probably impossible. Maybe you want to write an extension that automates logically replicating your data off the server, or truncates a table after processing it, to you that's useful, to someone else that's malicious, there's just no way to know.

If you don't trust the authors then the only way to trust a TLE is to inspect the code yourself in the context of what "safe" means to you. The pytorch example you mentioned was very likely caught by a human, if an automated system were possible for python (but likely impossible) it would have caught it sooner.

Post reply on HN