Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

11–20 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#11
post #4

... Which begs the question: what good is an ORM if it does not prevent by design such issues? Here, we are essentially saying users of ORM must also have in their mind the SQL version. Or call an expert after the mess is done :/...

I'm not very sure that Jooq and SQLAlchemy and other 'not quite ORMs' really help that much either.

I think, as the article suggests, that 'opaque' is as much the problem as the classic Object-Relational impedance mismatch, but I would posit that the opaqueness isn't just that the programmer can't _see_ what happens, but that they also don't _care_.

So I'd put my faith in tools instead. If programmers don't want to think about this, let the tools try instead.

The static analyser was interesting. Although static analysers have shallow comprehension, they were able to identify anti-patterns that were common enough to make it a useful linter. And of course their classic-mistakes approach can be applied to other languages too, even if each language and ORM needs its own corpus.

I sincerely hope that JetBrains and other IDEs bring this kind of analysis into their IDEs. I write a lot of non-Ruby code, and the JetBrain IDEs do keep suggesting nice 'code simplification' and 'generate boilerplate' checkers as I type.

But while they check my regex for well-formed-ness and colour my SQL but don't have any kind of meta analysis and comprehension. Missing opportunity.

You could take this further for normal non-DB code too - they could warn me when I have inefficient O(n) search in a loop and so on.

Re: How not to structure database-backed web apps: performance bugs in the wild

#12
post #3

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions. 10 billion instructions to send a few kilobytes of data. There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

on the other hand there are savings in development time, which might or might not be more expensive than cpu cycles in question. Especially for smaller scale projects. I'm not saying that 4 seconds to load a page is something to be happy about, but sometimes it's cheaper to "waste" cpu cycles than to spend more time developing faster solution.

Re: How not to structure database-backed web apps: performance bugs in the wild

#13
post #12
post #3

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions. 10 billion instructions to send a few kilobytes of data. There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

on the other hand there are savings in development time, which might or might not be more expensive than cpu cycles in question. Especially for smaller scale projects. I'm not saying that 4 seconds to load a page is something to be happy about, but sometimes it's cheaper to "waste" cpu cycles than to spend more time developing faster solution.

Although its really difficult to fathom how loading a webpage takes billions of cycles anyhow. It never used to. Remember the Internet in 2000? Pages are getting slower despite broadband speeds going through the roof and processors getting faster and there being so much more RAM etc.

Re: How not to structure database-backed web apps: performance bugs in the wild

#14
post #4

... Which begs the question: what good is an ORM if it does not prevent by design such issues? Here, we are essentially saying users of ORM must also have in their mind the SQL version. Or call an expert after the mess is done :/...

ORMs are good for 90% of your use cases, the rest I'm happy to use SQL for.

I could attempt to deconstruct the ORM problem, but Martin Fowler had a nice article [1] on replying to "ORM hate" a few years back which is worth the read for folks that question the usefulness of ORMs (and I believe questioning things is a healthy thing to do!)

[1] https://martinfowler.com/bliki/OrmHate.html

Re: How not to structure database-backed web apps: performance bugs in the wild

#16
post #3

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions. 10 billion instructions to send a few kilobytes of data. There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

From my experience, most of my waiting time is IO, not CPU load. Even simple stuff, such as SELECT COUNT(*) to display total count in grid on big table can take seconds. It does not have to be complicated.

Re: How not to structure database-backed web apps: performance bugs in the wild

#17
post #4

... Which begs the question: what good is an ORM if it does not prevent by design such issues? Here, we are essentially saying users of ORM must also have in their mind the SQL version. Or call an expert after the mess is done :/...

I find this question to be very weird. I have never seen ORMs as tools for completely abstracting away the database. I have always seen them as convenience APIs for using the database. They exist to make your code shorter, less repetitive, more readable, easier to reason about, and more maintainable; not to make you forget about the database altogether.

I also don't see ORMs as exclusive. It's fine to use ORMs for 95% of your use cases, but drop down to raw SQL for the remaining 5% where it's not a good match. That's still a win for the goals I mentioned above.

Re: How not to structure database-backed web apps: performance bugs in the wild

#18
post #6
post #3

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions. 10 billion instructions to send a few kilobytes of data. There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

Too much CPU and too much available memory will get you there. When in time of abundance, we humans are not good at optimizing.

It's more of an economics issue. It does not make sense to optimize, because bandwidth and CPU power are so abundant. Optimizing will take time away from adding new features.

Re: How not to structure database-backed web apps: performance bugs in the wild

#19
post #10

I never understood why I don't enjoy working with ORM but reading this article make it clearer for me. They blend the distinction between working in memory vs accessing the db. From one side it is very convenient, however I really feel that such performance sensitive operation should be carefully considered and that most SQL should be written by hand.

Or you could just spend 15 minutes learning how to avoid N+1 queries by using the includes() function, and some of the other super obvious performance optimization patterns that every junior Rails dev should know...

Re: How not to structure database-backed web apps: performance bugs in the wild

#20
post #4

... Which begs the question: what good is an ORM if it does not prevent by design such issues? Here, we are essentially saying users of ORM must also have in their mind the SQL version. Or call an expert after the mess is done :/...

The value proposition of ORM's is they simplify data access code, making the application simpler to write and more maintainable.

I have never seen anybody claim ORMs would eliminate performance issues.

Post reply on HN