Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

261–270 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#261
post #202
post #200

Earlier quoted context omitted.

Sure, writing SQL won't solve design problems, but some ORMs need multiple queries to deliver data with relationships, joins, etc, that could be done in a single SQL query.

What ORM's does not support joins?

Waterline (for Sails.js) is horrible. Last I looked there's no way to do a kind of `deepPopulate` (requiring an inner join)

Re: How not to structure database-backed web apps: performance bugs in the wild

#262
post #234
post #215

Earlier quoted context omitted.

I've heard it called Cache Stampede. Any decent framework for memoizing method calls would cover this case though.

Do you have an example of one? I'm curious what features they provide over ad hoc memoization mechanics.

https://docs.spring.io/spring/docs/5.0.0.BUILD-SNAPSHOT/spri...

Re: How not to structure database-backed web apps: performance bugs in the wild

#263
post #215

Earlier quoted context omitted.

I've heard it called Cache Stampede. Any decent framework for memoizing method calls would cover this case though.

It depends on the backing, if you're using just memcached there's no way to completely lock the key. None of the popular rails caching frameworks I've seen handle this either.

[deleted]

Re: How not to structure database-backed web apps: performance bugs in the wild

#264
post #202
post #200

Earlier quoted context omitted.

Sure, writing SQL won't solve design problems, but some ORMs need multiple queries to deliver data with relationships, joins, etc, that could be done in a single SQL query.

What ORM's does not support joins?

I imagine all of the non-toy ones would support joins for getting raw tabular data or for restricting what single kind of item (e.g. "Customer") you're pulling out.

Where it might get trickier is when you have something like a Customer-Region-Vendor join, and you actually want it to go ahead and create objects for the all of the Customers and Regions and Vendors whose data was pulled out of that one select.

Re: How not to structure database-backed web apps: performance bugs in the wild

#265

This may be a little OT (OT because the points raised in the study are totally valid and mine is just a comment) but for small companies and solo developers ORM or whatever that gets the job done quickly is the way to go. Most sites and web apps never even break a 100k/day hit mark for which I believe inefficiency may not be the biggest issue. But wasting a month tryig to write native Sql queries can hurt your projec…

I actually make this calculation for most projects that I work on -- what level of traffic do I expect this to see? If it's a back-office function or is limited to a few hundred clients at a time, I don't even worry about performance.

But that said... writing native SQL queries wastes a month? Um, no. ORMs are convenient, but writing select, insert, update, and delete queries by hand isn't that hard. It's mildly verbose and you end up repeating yourself lots so it's annoying, particularly if you (like me) have the kind of programmer's mind that constantly wants to factor that out. But it's doable without adding a high factor to development time.

Re: How not to structure database-backed web apps: performance bugs in the wild

#266

Earlier quoted context omitted.

True but cache invalidation is famously hard to get perfect. A nicer way to put it might be that finite TTLs should be viewed as an opportunity to optimize, not ideal or standard.

True, caching even simple data dependant on 2-3 variables may require writing dozens of test cases - in practice, it often turns out that the data doesn't rely on 2-3 but 5-10+ variables. However, just because it's hard it doesn't mean we shouldn't do it the right way.

Of course it does - negotiating trade-offs and accepting the set that best achieves your objectives is a large part of software design (and also undermines the notion of doing things a "right way").

Wouldn't deploying a less-than-precise TTLs be an appropriate trade-off for non-transactional caches, caches subject to network partitions, caches that can't enroll in invalidation messages, caches that can't poll for change sets, caches that can't implement eviction policy, etc?

Certainly the cache should be transparent about the trade-offs it has made (such as not promising authoritative data if it's accepting eventual consistency via TTL).

Re: How not to structure database-backed web apps: performance bugs in the wild

#267
post #60

Earlier quoted context omitted.

Even if you get 1 hit / day and that takes a long time to return, the ORM has failed your one customer. Inefficiency exists even at small scales with ORMs. If your developers don't know how to write SQL, let them learn. Or fire them if they won't.

> Inefficiency exists even at small scales with ORMs. In other words, you have no idea what you're talking about.

Personal attacks will get you banned here, regardless of how wrong someone else is. You've unfortunately been uncivil in at least one other comment in this thread too.

Could you please (re-)read https://news.ycombinator.com/newsguidelines.html and not use this site that way? The idea here is to post civilly and substantively, or not at all.

Re: How not to structure database-backed web apps: performance bugs in the wild

#268
post #82

When I was inexperienced I feared ORMs because of the negative performance impacts I've read they could have. I constantly worried about what would happen if the amount of data increased and I hit ORM induced problem that I could not resolve without major rewrite of data access layer. However, whenever I've actually hit those problems in production, I found the similar thing the authors of the article did - ORM induc…

The fact that you feared ORMs probably pushed you to use raw database, and such experience really helps you understand how ORMs work.

Same is true with any abstraction – if you want to learn to use it right, first learn to make do without it.

Re: How not to structure database-backed web apps: performance bugs in the wild

#269
post #264
post #202

Earlier quoted context omitted.

What ORM's does not support joins?

I imagine all of the non-toy ones would support joins for getting raw tabular data or for restricting what single kind of item (e.g. "Customer") you're pulling out. Where it might get trickier is when you have something like a Customer-Region-Vendor join, and you actually want it to go ahead and create objects for the all of the Customers and Regions and Vendors whose data was pulled out of that one select.

It is actually not that bad if instead of a single query, you do one query per table.

It even offsets some of the burden to the app, which is generally easier to scale than the database.

The idea is that you query the "root" table, loop over the results and build an array of IDs, then do additional queries to the other tables with a "where whatever_id IN (...)".

Re: How not to structure database-backed web apps: performance bugs in the wild

#270
post #264
post #202

Earlier quoted context omitted.

What ORM's does not support joins?

I imagine all of the non-toy ones would support joins for getting raw tabular data or for restricting what single kind of item (e.g. "Customer") you're pulling out. Where it might get trickier is when you have something like a Customer-Region-Vendor join, and you actually want it to go ahead and create objects for the all of the Customers and Regions and Vendors whose data was pulled out of that one select.

In NHibernate and Entity Framework you just say .Include("Region").Include("Vendor") to eagerly load the associated entities. I imagine other ORM's have similar facilities.
Post reply on HN