Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

Post reply on HN
Print view View original

51–60 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#51

Post by [deleted] »

czechdeveloper wrote: post #16
mozumder wrote: post #3

Its always amazing to see web pages take 2-4 seconds of back-end processing. On a modern CPU, that's about 10 billion instructions. 10 billion instructions to send a few kilobytes of data. There's such a waste in back-end server design. If you're measuring response times are in seconds, and not in microseconds, you're doing something seriously wrong.

From my experience, most of my waiting time is IO, not CPU load. Even simple stuff, such as SELECT COUNT(*) to display total count in grid on big table can take seconds. It does not have to be complicated.

[deleted]

Re: How not to structure database-backed web apps: performance bugs in the wild

#52

Post by TekMol »

fuzzy2 wrote: post #37
TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

An ORM doesn’t have to be slow. Most can source their data from custom queries or stored procedures. Not using an ORM requires writing lots of code to do what the ORM would otherwise do. Is that really appropriate today? Should we not take advantage of the available CPU/RAM? I say ORM is the correct choice in many cases. Not always.

    I say ORM is the correct choice in many cases
I'm still undecided about this. Would be fun to compare some actual approaches. Which is your favorite ORM?

Re: How not to structure database-backed web apps: performance bugs in the wild

#53

Post by irishsultan »

TekMol wrote: post #45
manicdee wrote: post #43

Earlier quoted context omitted.

That’s how I have seen it done in Perl, PHP, Django, etc as well. It’s not an ORM thing it is a naive programmer thing. The ORM makes it easier to do things like Class.filter(insurance_end_date But hey what do I know, I am not berating the youf of today so I don’t belong in this skit.

Class.filter(insurance_end_date Which ORM is that? It would be cool to compare some approaches to common problems.

Diesel has something similar (except that it can't overload < (you can overload < in Rust, but the type doesn't match what's needed), so you need to specify it as insurance_end_date.lt(today_date))

Re: How not to structure database-backed web apps: performance bugs in the wild

#54

Post by grey-area »

TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

I use an ORM. I don't perform queries like that because the ORM makes it easy to build complex queries, joins, limiting the data returned etc, then execute them in one query, it's a convenience, not a straightjacket.

The problem is not ORMs, the problem is just people not thinking about the resources their query over some data takes, and trying to do things in memory that are better done in the database (as in your example), because databases are hard, and sql is not a particularly friendly language, so they take the easy way out. They'd do things like iterating over all instances just as much if not using a framework/ORM, because they understand their programming language and don't care to understand SQL, and also because they get away with it when there are 1000 cars in the fleet, and didn't anticipate having 1 million.

Re: How not to structure database-backed web apps: performance bugs in the wild

#56

Post by TekMol »

dagw wrote: post #47
TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

That has nothing to do with ORMs. I've seen plenty of people write code like: SELECT * FROM DATA for each r in result if r.x > 12 do_something(r.y) Exactly the same thing without any ORM. If anything ORMs should improve performance for novices since it makes it a lot easier for people to writer better queries that run on the database.

The ORM makes it more likely to do stuff in the application instead of the DB because many queries are hard or impossible to represent in an ORM fashion.

Additionally, the ORM makes it much harder to see what is going on under the hood. For example you don't know if a value of an object is stored in the main table or lazy loaded from a related table. So you have no clue if echo "$user.name lives in $user.city" results in 0 sql queries or one or two.

Re: How not to structure database-backed web apps: performance bugs in the wild

#57

Post by czechdeveloper »

rypskar wrote: post #35
czechdeveloper wrote: post #16

Earlier quoted context omitted.

From my experience, most of my waiting time is IO, not CPU load. Even simple stuff, such as SELECT COUNT(*) to display total count in grid on big table can take seconds. It does not have to be complicated.

>Even simple stuff, such as SELECT COUNT(*) edit seems like my old performance book was outdated or wrong :(

SELECT COUNT(*) should be the one which is resolved by index/metadata

Re: How not to structure database-backed web apps: performance bugs in the wild

#58

Post by grey-area »

marcus_holmes wrote: post #31
olavk wrote: post #20

Earlier quoted context omitted.

The value proposition of ORM's is they simplify data access code, making the application simpler to write and more maintainable. I have never seen anybody claim ORMs would eliminate performance issues.

Right up until the point where you have to hack around the ORM to get performance where it needs to be. Then you suddenly have more complex and harder to maintain "magic" code than if you'd just written some simple-but-boring boilerplate and SQL from the start. I'm wading through the tedious and boring process of writing a data access layer for an application at the moment. It's repetitive, there's lots of error-chec…

Hacking around the ORM should be a question of:

    query.SQL("my complex query here")
If it's not, your ORM is broken and you should write a new one (they're really not that complex).

Re: How not to structure database-backed web apps: performance bugs in the wild

#59

Post by mnm1 »

ORMs are really only useful for throwaway projects and beginners. I have yet to see one without serious downsides in both performance and speed of development, something that they are touted to improve but actually make worse. The ORM I'm stuck with now (Doctrine2) adds a 10x overhead to queries. For the most part, we don't even bother optimizing queries in such situations because why waste time on something that could at most improve performance by 1/10th? The real fix is to get rid of the ORM. That requires a rewrite of two apps that have been in development for four years each. I can't think of a worse architectural decision for business (CRUD) software than using an ORM. Irreversible and forever fucked.

Re: How not to structure database-backed web apps: performance bugs in the wild

#60

Post by mnm1 »

superasn wrote: post #28

This may be a little OT (OT because the points raised in the study are totally valid and mine is just a comment) but for small companies and solo developers ORM or whatever that gets the job done quickly is the way to go. Most sites and web apps never even break a 100k/day hit mark for which I believe inefficiency may not be the biggest issue. But wasting a month tryig to write native Sql queries can hurt your projec…

Even if you get 1 hit / day and that takes a long time to return, the ORM has failed your one customer. Inefficiency exists even at small scales with ORMs. If your developers don't know how to write SQL, let them learn. Or fire them if they won't.
Post reply on HN