Live data from Hacker News

Simplify: move code into database functions

sivers.org

71–77 of 77 posts

Re: Simplify: move code into database functions

#71
post #68
post #13

Stored procedures, functions, triggers, etc, are very useful tools. They are used regularly in Microsoft SQL Server and Oracle systems. But, one has to be very careful to use them when warranted. The out-of-the-box tooling for debugging, maintaining, versioning, and testing code that lives in your database is not nearly as robust as what you have for most other development environments. The key here is to use the too…

Nice write-up. > This code has to be maintained, versioned, etc. Specifically to a Microsoft stack this specific pain point is outright negated, in fact, Microsoft's answer (Visual Studio SQLPROJ/Data Dude) exposes migrations for the horrible metahistory hack that they are. SQL projects are stored in source control as though they were C++/C#/what-have-you. Just like the rest of our code, the 347kloc SQL portion is ha…

> 6. Triggers+family are invisible logic. They can cause confusion during debugging.

I'm not sure I can agree. This is the appropriate venue to validate/copy/mutate data. An heavily constrained table should be documented as a matter of API instructions, because (as you mentioned) they can cause confusion, but let's not throw the baby out with the bathwater, eh?

Re: Simplify: move code into database functions

#72
post #68

Earlier quoted context omitted.

Nice write-up. > This code has to be maintained, versioned, etc. Specifically to a Microsoft stack this specific pain point is outright negated, in fact, Microsoft's answer (Visual Studio SQLPROJ/Data Dude) exposes migrations for the horrible metahistory hack that they are. SQL projects are stored in source control as though they were C++/C#/what-have-you. Just like the rest of our code, the 347kloc SQL portion is ha…

> 6. Triggers+family are invisible logic. They can cause confusion during debugging. I'm not sure I can agree. This is the appropriate venue to validate/copy/mutate data. An heavily constrained table should be documented as a matter of API instructions, because (as you mentioned) they can cause confusion, but let's not throw the baby out with the bathwater, eh?

Yeah it's totally a debatable one, hence the "can." I should have listed it as a caveat or something.

Re: Simplify: move code into database functions

#73
post #13

Stored procedures, functions, triggers, etc, are very useful tools. They are used regularly in Microsoft SQL Server and Oracle systems. But, one has to be very careful to use them when warranted. The out-of-the-box tooling for debugging, maintaining, versioning, and testing code that lives in your database is not nearly as robust as what you have for most other development environments. The key here is to use the too…

You commonly encounter "legacy" code that uses this design and often people who have cut their teeth on currently fashionable techniques put it down. If you want to construct high performance applications, however, moving code close to data solves a lot of problems, particular those involving the reconciliation of durability and scalability. (i.e. transactions work a lot better if you can get all the coordinated part…

This doesn't just go for SQL, but it will go for NoSQL systems that support advanced functionality.

My current project went nuts with the Redis + Lua scripts. Trace statements and manual inspection. Feels like I'm developing on the Apple ][. It started innocently enough. Now it's a hydra. Nutty.

I found a HOWTO for debugging embedded Lua. Basically run Redis from within an IDE. Definitely not turnkey. Might be worth the setup effort, next time I visit that code.

Re: Simplify: move code into database functions

#74
post #69

Earlier quoted context omitted.

The out-of-the-box tooling for debugging, maintaining, versioning, and testing code that lives in your database is not nearly as robust as what you have for most other development environments. This is the key point for me. I've seen SQL databases with 60k+ LOC in stored procedures, and functions (thankfully they avoided triggers). This code wasn't versioned, tested, or commented. While it's technically true that you…

> I've never seen it done. See my comment[1]. We're doing it and it works beautifully. [1]: https://news.ycombinator.com/item?id=9485502

Awesome, I'd love to try working in an environment like this to see how it works and experience the tradeoffs.

Re: Simplify: move code into database functions

#75

A serious problem with this approach is scaling. When your code is in Ruby/JS/whatever on separate servers, and your traffic increases, you can easily scale up by adding more stateless servers. Going from 1 to 2 DB servers is much harder, and can easily mean a major rewrite, during which your site may not really work.

> A serious problem with this approach is scaling. Stack overflow runs off one SqlServer. Do you think your app is going beyond that? If yes, maybe this approach will not work for you. If no, you are in the 99,99999% of all other sites/apps that are not as big as SO, FB or google and you can use this simple way to make your site or app.

I think the original comment was talking about the need to scale out application servers to take on additional load. Each application sever still communicates with (usually) the one database.

Normally people talk about that as a way to increase the throughput of requests that your site can respond to, but it has another benefit: more time spent performing data validation, string munging, date checking, etc adds up to less time spent in the database on the same thing.

Stack Overflow may run off of one relational database, but it solves the scaling problem through liberal application-level caching and reverse-proxy caching, all using about a dozen servers that connect to the one database.

Re: Simplify: move code into database functions

#76

It depends what you do... If you need to manipulate a lot of data that ends up travelling on the wire, you can save a lot of time by running it on the server. On the other hand, the language on the DB servers are often a bit harder to reason with.

> the language on the DB servers are often a bit harder to reason with

But can't we say that about every other language? I've been writing a lot of PL/pgSQL lately and I've enjoyed every bit of it. For PostgreSQL there's also PL/Python, PL/V8, etc. Those are packages for writing the server code in python and JavaScript respectively and out the window goes the obscure language excuse. But that's if you're on Postgres.

Re: Simplify: move code into database functions

#77
post #24

Earlier quoted context omitted.

The other side of the coin is that you probably won't need to scale. The performance differences can be staggering: I've replaced Java-based procedures which took more than an hour to execute with Stored Procedures which completed within milliseconds. That's an extreme example - the original code was simply incompetent - but I regularly see significant (100x) performance increase in core functions through the switch.…

I see databases in general get a lot of flack from developers for this reason, when on the other side of the fence I see a lot of really smart people misunderstanding the tool (as your example alludes to.) I have often come to a stored procedure and not returned 100x, but more like 50,000x by simply changing a few lines of code. tl;dr anyone can write slow code in any language.

> I have often come to a stored procedure and not returned > 100x, but more like 50,000x by simply changing a few > lines of code.

Oh, those beautiful nested cursors :)

Post reply on HN