Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

Post reply on HN
Print view View original

311–319 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#311

Post by dangets »

Jach wrote: post #258
ndrr wrote: post #234

Earlier quoted context omitted.

Do you have an example of one? I'm curious what features they provide over ad hoc memoization mechanics.

If you're in the Java ecosystem, the CacheBuilder in Guava is pretty good: https://google.github.io/guava/releases/19.0/api/docs/com/go... By default it handles the case of concurrent retrieval on the same key (the second one will just wait for the first one to finish and use that value rather than starting a duplicate computation). It also lets you configure more interesting things like eviction strategies, removal…

A coworker introduced me to Caffeine which I think shares authors with Guava's cache. Kind of a 2.0 /lessons learned iteration.

https://github.com/ben-manes/caffeine

Re: How not to structure database-backed web apps: performance bugs in the wild

#312

Post by itsderek23 »

syastrov wrote: post #297

I’ve experienced a lot of n+1 queries problems as causes for bad performance. Often times this was a result of wrapping the ORM in abstraction layers (for business logic and fears of being “locked in” to the ORM). We rewrote that part of the application using a different ORM (in Python). Making use of a tool that could help find these problems automatically helped greatly and we didn’t have a single performance probl…

Scout also detects these for Django, ordering by the most performing N+1s: http://blog.scoutapp.com/articles/2018/04/30/finding-and-fix...

Re: How not to structure database-backed web apps: performance bugs in the wild

#313

Post by rapind »

brightball wrote: post #308
rapind wrote: post #306

Earlier quoted context omitted.

CacheEx sounds interesting. Basically a debounce. Pretty sure you could solve this outside of the application layer with Varnish but that depends on how the view is composed. I prefer using a grace / stale period but that only works if it's acceptable to return stale data during computation instead of queuing it up.

Well CacheEx is just using functionality that comes naturally on the BEAM here. This is one of the reasons that a lot of CDN's like Cloudfront are written in Erlang. CacheEx gets the ability to check for the presence of the cache key in Erlang Term Storage (ETS) which is basically an in-memory cache. If the key is present, it just returns the value. If it's not, it sends checks to see if a process exists with the cac…

That's pretty awesome. Varnish is my go to for Rails apps (and almost always necessary).

Re: How not to structure database-backed web apps: performance bugs in the wild

#314

Post by dang »

lc94 wrote: post #103
TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

What a stupid comment. Your point has absolutely nothing to do with ORMs and only shows your utter inexperience with them. You could replace everything you said with using SQL directly and just doing a select * from cars

This comment breaks the site guideline against calling names in arguments. Could you please read https://news.ycombinator.com/newsguidelines.html and stick to the rules, regardless of how wrong or annoying another comment might be?

Re: How not to structure database-backed web apps: performance bugs in the wild

#315

Post by lelelelemon »

rmc wrote: post #299

One of the example Rails applications they use is the code which powers the OpenStreetMap website. They populated their install by randomly filling in fields on the website. Which doesn't include any map editing! For OSM they suggest changing how the diary feature operates, which is a tiny, almost irrelevant part of the OSM website software stack. The OSM database has millions of geographic objects, and they talk abo…

Thanks for looking into this - I am one of the authors of the study. We have updated the submitted issues link to https://github.com/hyperloop-rails/study-replication/tree/ma.... Let us know if you have any further questions.

Re: How not to structure database-backed web apps: performance bugs in the wild

#316

Post by sut-123 »

itsderek23 wrote: post #312
syastrov wrote: post #297

I’ve experienced a lot of n+1 queries problems as causes for bad performance. Often times this was a result of wrapping the ORM in abstraction layers (for business logic and fears of being “locked in” to the ORM). We rewrote that part of the application using a different ORM (in Python). Making use of a tool that could help find these problems automatically helped greatly and we didn’t have a single performance probl…

Scout also detects these for Django, ordering by the most performing N+1s: http://blog.scoutapp.com/articles/2018/04/30/finding-and-fix...

And bullet is used for Rails to detect N + 1 queries. https://github.com/flyerhzm/bullet

Re: How not to structure database-backed web apps: performance bugs in the wild

#317

Post by sut-123 »

soberhoff wrote: post #286
mklepaczewski wrote: post #251

Earlier quoted context omitted.

This is not necessarily true. You would be surprised how many absolutely trivial performance issues can be found in almost every project. Sometimes it's really just about moving computation of a constant value out of a 'for' loop.

I once sped up a program by 90% by turning `new String("foo")` into just `"foo"`. The project was still rotten though, so it ultimately didn't matter.

Your example sounds interesting, could you explain more

Re: How not to structure database-backed web apps: performance bugs in the wild

#318

Post by soberhoff »

sut-123 wrote: post #317
soberhoff wrote: post #286

Earlier quoted context omitted.

I once sped up a program by 90% by turning `new String("foo")` into just `"foo"`. The project was still rotten though, so it ultimately didn't matter.

Your example sounds interesting, could you explain more

Are you familiar with Java? If you use just `"foo"`, then the string will be interned and reused. If you use `new String("foo")`, each call creates a new copy on the heap. This call was inside a very hot loop, thus eating almost all the application's runtime.

Re: How not to structure database-backed web apps: performance bugs in the wild

#319

Post by sut-123 »

soberhoff wrote: post #318
sut-123 wrote: post #317

Earlier quoted context omitted.

Your example sounds interesting, could you explain more

Are you familiar with Java? If you use just `"foo"`, then the string will be interned and reused. If you use `new String("foo")`, each call creates a new copy on the heap. This call was inside a very hot loop, thus eating almost all the application's runtime.

Thanks a lot for your explanation.
Post reply on HN