How not to structure database-backed web apps: performance bugs in the wild • Hacker News Board

Re: How not to structure database-backed web apps: performance bugs in the wild

#211

Post by steventhedev » Thu, Jun 28, 2018, 2:32 PM UTC

memset wrote: post #207

blattimwind wrote: post #196

Earlier quoted context omitted.

0 - Caching antipattern 101: key = calculate_cache_key() if not cache.has(key): data = expensive_calculation() cache.store(key, data) else: data = cache.get(key)

This is interesting because this is generally how I implement caching! What would pseudocode look like for a non-antipattern?

there's a race condition between checking if something is in the cache and actually putting it in the cache. A more correct solution would have all threads wait while one does the actual work.

This gets more complicated if you have a distributed cache and/or distributed application servers. The typical solution there is to allow at most one computation per process/device.

Re: How not to structure database-backed web apps: performance bugs in the wild

#212

Post by mnutt » Thu, Jun 28, 2018, 2:34 PM UTC

nicktrocado92 wrote: post #209

blattimwind wrote: post #196

Earlier quoted context omitted.

0 - Caching antipattern 101: key = calculate_cache_key() if not cache.has(key): data = expensive_calculation() cache.store(key, data) else: data = cache.get(key)

Why it's an antipattern ?

It’s susceptible to Thundering Herd whereby more requests come in for the same cache key before the initial computation is finished, and so you end up with lots of cache misses. The fix is usually to lock the cache key and have subsequent requests wait on the original computation but it’s a bit more complex to code.

Re: How not to structure database-backed web apps: performance bugs in the wild

#213

Post by manigandham » Thu, Jun 28, 2018, 2:34 PM UTC

coldtea wrote: post #193

manigandham wrote: post #167

Earlier quoted context omitted.

Encouragement and attitude doesn't force you to do anything. You still have to choose to use it, either well or poorly. This is no different than any other tool that makes things easy but with obvious limits. Proper decision making is still up to you. There really isn't much controversial here if you get past the whole "ORM" hype/hate cycle.

> Encouragement and attitude doesn't force you to do anything. You still have to choose to use it, either well or poorly. I don't believe in choices, people are flimsy. I believe in creating an environment that encourages good behavior.

> I don't believe in choices

...ok, people still make choices though, you're not controlling their minds. Perhaps educate your workforce so they make the right decisions by themselves, it's more effective and takes less effort than trying to coerce them through generalizations.

Re: How not to structure database-backed web apps: performance bugs in the wild

#214

Post by vidarh » Thu, Jun 28, 2018, 2:35 PM UTC

theapemachine wrote: post #130

vidarh wrote: post #100

Earlier quoted context omitted.

It is the developer, but ORMs are so controversial in part because they often obscure that you're doing something crazily ineffective in ways that makes developers that don't understand the abstraction fail to see that they're doing something obviously wrong. It's more stark that you're doing something crazy if you do a SELECT, instantiate objects from each returned row, then apply a filtering rule to that object, th…

Pretty sure that is what tailing the log during development is all about. I don't see what is so obscure about it, in the Rails based examples in the post, one can just see the logs flying by with the exact queries being executed.

It is, but a developer that's using ORMs because they're scared of SQL tends to be the same type of developer that will decide not to look at those.

I'm not dismissing ORMs - I use Sequel (the Ruby ORM) for almost all my database access. But I've also seen enough people use ORMs as an excuse to pretend they don't need to understand SQL or understand the database to understand why some people look at ORMs with suspicion.

A lot of code that is obviously bad when the database queries are plain for everyone to see are not so obviously bad when it's less clear if that method call translates to a database query or just extracts data locally.

Note that this is a general issue with this type of abstraction: It is a common complaint against transparent RPC wrappers as well that if they're too good at hiding that an object is remote it's easy for someone to carelessly cause massive amounts of unnecessary roundtrips.

Re: How not to structure database-backed web apps: performance bugs in the wild

#215

Post by tootie » Thu, Jun 28, 2018, 2:39 PM UTC

mnutt wrote: post #212

nicktrocado92 wrote: post #209

Earlier quoted context omitted.

Why it's an antipattern ?

It’s susceptible to Thundering Herd whereby more requests come in for the same cache key before the initial computation is finished, and so you end up with lots of cache misses. The fix is usually to lock the cache key and have subsequent requests wait on the original computation but it’s a bit more complex to code.

I've heard it called Cache Stampede. Any decent framework for memoizing method calls would cover this case though.

Re: How not to structure database-backed web apps: performance bugs in the wild

#216

Post by Cthulhu_ » Thu, Jun 28, 2018, 2:40 PM UTC

projectramo wrote: post #185

The real joy of this article was the following: Some academics (Yang, Subramanian, Lu, Yan, Cheung) were able to produce massive improvements in about a dozen large, mature, battle tested open source projects using just a few lines of code. This should give hope to all those tepidly trying to get into open source. Just go and take a look at the dozens of open source projects in Django or whatever and you could improv…

What's intimidating to me is that to make a change like that, you first need to get the code, get it and the tests running, and actually understand what the code does - and whether your fix has the equivalent result or is subtly broken.

They didn't just change a single line in an application, they wrote a huge benchmark suite and dug through miles of code to find issues like this. I've no clue how much time they spent on it, it must've been months.

Re: How not to structure database-backed web apps: performance bugs in the wild

#217

Post by vidarh » Thu, Jun 28, 2018, 2:45 PM UTC

olavk wrote: post #149

Arnt wrote: post #145

Earlier quoted context omitted.

I read the paper. A third, maybe half of the stuff the ORMs didn't detect looks simple to detect. (Greybeards may remember how perl's -W switch switch suddenly detected a frightful amount of performance problems using mostly simple tests. Almost 20 years ago now.)

Maybe you are on to something. All the major ORM's are open source (AFAIK) so if you made a write-up of the common issues and how you suggest they could be detected by the ORM, I think it would be very well received.

I'd think you could do warnings on this simply by processing a log of the queries, looking for query patterns that match common poor code patterns.

E.g.

    SELECT id FROM table [some conditions]

followed by a number of

   SELECT * FROM table WHERE id = ...

is one example of a anti-pattern that suggests that someone is doing an overly simplistic query followed by a loop. Similar with signs of triggering loading of related objects instead of a JOIN.

Doing it on the emitted SQL would also make it quite easy to make it reasonably ORM agnostic - it doesn't need to be perfect, after all, so doing relatively crude pattern matching ought to be able to find at least the more basic problems.

Re: How not to structure database-backed web apps: performance bugs in the wild

#218

Post by ahallock » Thu, Jun 28, 2018, 2:47 PM UTC

The problem with ORMs is that they make DB calls seem cheap, and when your application is young and you don't have a lot of rows, everything works wonderfully. Then all the N+1 problems start adding up and the SELECT COUNT(*) performance issues. Since the ORM code is baked into the models, your code is littered with these calls -- in the service layer, views, background workers.

That's why I really like Promises/Futures in Javascript. You know exactly when you're executing a DB operation and you have to think about the implications more. It's not just a simple object accessor.

Re: How not to structure database-backed web apps: performance bugs in the wild

#219

Post by krainboltgreene » Thu, Jun 28, 2018, 2:50 PM UTC

There are some really glaring issues in this article. For example, the author suggests that each -> update is worse than update_all, and while the performance may be better it's comparing apples to oranges! update_all, in ActiveRecord, bypasses business logic, bypasses updating the updated_at, bypasses validation and more. It's not the same thing at all. There is no difference between where.first and find_by, that is just a bikeshed. Not every map(column) can be turned into pluck(column) if there are business logic wrappers around that column.

Re: How not to structure database-backed web apps: performance bugs in the wild

#220

Post by blattimwind » Thu, Jun 28, 2018, 2:51 PM UTC

memset wrote: post #207

blattimwind wrote: post #196

Earlier quoted context omitted.

0 - Caching antipattern 101: key = calculate_cache_key() if not cache.has(key): data = expensive_calculation() cache.store(key, data) else: data = cache.get(key)

This is interesting because this is generally how I implement caching! What would pseudocode look like for a non-antipattern?

The easiest workaround is to acquire a lock (e.g. redis redlock) before starting to refresh the cache. That way only one process will perform expensive_calculation(). A good caching library will generally do some variation on this for you.

For larger, more complex scenarios the techniques mentioned by GGP above work well, i.e. not doing expensive_calculation() in your app process at all.

The issue has a bunch of names, I know at least three: cache stampede, thundering herd and dogpile effect.