Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

Post reply on HN
Print view View original

101–110 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#101

Post by iKSv2 »

latch wrote: post #23

I've worked on moderately busy backend platforms (~10K-20k rps handled on a ~4 e5-2650 and aiming for 5ms 95p response times). It greatly depends on what you're doing, but for the majority of systems which are read heavy (and that most certainly includes "dynamic" sites like Amazon or Wikipedia), I hold to two major beliefs: 1 - Have very long TTLs on your internal cache servers with a way to proactively purge (messa…

Do this (point 1) properly and ~80% of your "speed" problems are solved, for a simple web-app-db tiered app.

Re: How not to structure database-backed web apps: performance bugs in the wild

#102

Post by asfdsfggtfd »

TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

All the ORMs I have used can easily do the equivalent of this

     select count(*) from cars where condition;
If programmers are doing what you say that they are then the programmers are the problem not the library.

Furthermore most ORMs (certainly any that I would consider using!) allow escaped SQL to be used - and if the query gets much more complicated than a couple of where clauses I consider using this feature.

A decent ORM used well allows programmers to program faster on the simple stuff but still write fast code for the complex stuff.

Re: How not to structure database-backed web apps: performance bugs in the wild

#103

Post by lc94 »

TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

What a stupid comment. Your point has absolutely nothing to do with ORMs and only shows your utter inexperience with them.

You could replace everything you said with using SQL directly and just doing a select * from cars

Re: How not to structure database-backed web apps: performance bugs in the wild

#104

Post by EGreg »

l0b0 wrote: post #95
manigandham wrote: post #85

Earlier quoted context omitted.

Not this again. ORMs are tools, basically dynamic code generators that run SQL and map the results to in-memory objects, and vice-versa. Some are simplistic and others are incredibly advanced, and the code itself is usually faster than your own sql->objects logic that you would write otherwise. The issues with performance are almost always with the way the tool is used, like choosing a bad algorithms or the wrong dat…

Not this again. ORMs are some of the most impenetrable code you'll ever see. Understanding the performance characteristics of a CRUD application using SQL is vastly simpler than trying to puzzle out the arcane ways in which ORMs decide to throw up garbage once you add one more thing to the mix.

Not that again. ORMs achieve one main thing: being able to map your app’s objects to a relational database and back.

But there are tons of other benefits:

1) Avoid all injection attacks by default by binding variables rather than interpolating their vakues

2) Write SQL code for you to automatically, so you always have balanced parentheses and no typos or errors mixing statements

3) Autogenerate classes and methods from the SQL model automatically so there is a place where you can add custom methods on objects

4) Model fields and relationships in a way the app can understand, so you can make meaningful error checks and manipulations in the app instead of relying on the database engine to give you a nice error message or manually copying the data type logic.

5) Play nice with version control, making sure to encapsulate the code in ONE PLACE instead of a million places when the schema or model changes.

6) Pissing people off on HN so we can better discuss and explain principles of architecting good softeare while talking anout the benefits of ORM

7) Let you write an adapter to move to eg a graph database which is far faster.

8) In fact, just making you avoid doing joins in the database by default is already a feature as you can make your app far more scalable w sharding and possibly think about making it byzantine fault tolerant and distributed!

See for example this:

https://qbix.com/platform/guide/database

https://qbix.com/platform/guide/models

Re: How not to structure database-backed web apps: performance bugs in the wild

#105

Post by asfdsfggtfd »

threeseed wrote: post #61
TekMol wrote: post #56

Earlier quoted context omitted.

The ORM makes it more likely to do stuff in the application instead of the DB because many queries are hard or impossible to represent in an ORM fashion. Additionally, the ORM makes it much harder to see what is going on under the hood. For example you don't know if a value of an object is stored in the main table or lazy loaded from a related table. So you have no clue if echo "$user.name lives in $user.city" result…

> ORM makes it much harder to see what is going on under the hood I've never in the last 20+ years seen an ORM that doesn't allow you to log the SQL queries with a single configuration option.

With an add on Django even displays them in the browser!

Re: How not to structure database-backed web apps: performance bugs in the wild

#106

Post by manigandham »

l0b0 wrote: post #95
manigandham wrote: post #85

Earlier quoted context omitted.

Not this again. ORMs are tools, basically dynamic code generators that run SQL and map the results to in-memory objects, and vice-versa. Some are simplistic and others are incredibly advanced, and the code itself is usually faster than your own sql->objects logic that you would write otherwise. The issues with performance are almost always with the way the tool is used, like choosing a bad algorithms or the wrong dat…

Not this again. ORMs are some of the most impenetrable code you'll ever see. Understanding the performance characteristics of a CRUD application using SQL is vastly simpler than trying to puzzle out the arcane ways in which ORMs decide to throw up garbage once you add one more thing to the mix.

It's just code, compiled into a library that you can use, and queries are either logged by the library or in your database. What exactly is so impenetrable? That's just strange. Also CRUD is the ideal scenario for ORMs which handle all the mapping, security, parameterization, and even conversions between data types and models seamlessly while letting you just work with your objects.

It seems you either didn't use ORMs correctly or used a very poor one.

Re: How not to structure database-backed web apps: performance bugs in the wild

#107

Post by mukundasai111 »

I figure you can have a touch of segment An, a touch of segment B.

My home city is Mumbai (in fact, the rural city of Thane, north of Mumbai) a standout amongst the most thick expansive urban areas on the planet.

Open transportation in Mumbai, as far as recurrence, limit, dependability, and cost is now superior to anything any American city (I'm including NYC that). It could be better; for example the city is served by standard check prepares however the metro railroad (whatever measure a tram is) that goes underground is still under development.

In spite of all that, and regardless of the way that the tallest working in the city is a unimportant 88 stories[1] the foundation is stressing under the weight each day. All the steady employments (government, tech, managing an account) are moved in only maybe a couple parts of the city (South Mumbai and the Andheri-Bandra-Dadar triangle, perhaps Powai; my geology may be somewhat obsolete in light of the fact that I haven't lived there for a long time). What's more, land in those spots is cosmically expensive[2] (here's a non-extravagance 3-bed condo for USD 1.3m in South Mumbai). That implies everybody whose isn't living with guardians who purchased decades back in those spots has 3-hour round trek drives each day (I was one of those individuals).

Building taller wouldn't densify the city since rich individuals will probably purchase that new lodging and they have littler families by and large. Enhancing open transportation won't change laws of separation and movement - like I said it's as of now very great. It would be better for individuals in North Mumbai if there were more occupations closer to them so they wouldn't need to drive so far consistently.

Rural, auto driven, sprawl isn't perfect for city development, however nor is the grouping of occupations in a single place.

http://sobhalake-garden.in/price.html

Re: How not to structure database-backed web apps: performance bugs in the wild

#108

Post by vidarh »

mpweiher wrote: post #91
TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

That's one thing I love about Gemstone/Smalltalk. In Smalltalk, you would write such a query as: (cars select: [ :car | car insurance endDate (There probably is a shortcut for just counting the elements, though admittedly you have to have each car for the query). So what Gemstone does is that when cars is a DB collection, it interprets the block and turns it into a query. So the code for iterating in-memory and the c…

In Ruby, using Sequel:

    DB[:cars].where { insurance_end_date
(replacing "Time.now" with Sequel.function("NOW") if you want server side, or whatever date/time formatting function you want that'll return time/date in your desired format client side; client side it'll be evaluated once)

I think most ORMs can do a reasonable job at this in languages where you have sufficient flexibility in overloading behaviour.

Re: How not to structure database-backed web apps: performance bugs in the wild

#109

Post by bryanrasmussen »

vidarh wrote: post #100
manigandham wrote: post #85

Earlier quoted context omitted.

Not this again. ORMs are tools, basically dynamic code generators that run SQL and map the results to in-memory objects, and vice-versa. Some are simplistic and others are incredibly advanced, and the code itself is usually faster than your own sql->objects logic that you would write otherwise. The issues with performance are almost always with the way the tool is used, like choosing a bad algorithms or the wrong dat…

It is the developer, but ORMs are so controversial in part because they often obscure that you're doing something crazily ineffective in ways that makes developers that don't understand the abstraction fail to see that they're doing something obviously wrong. It's more stark that you're doing something crazy if you do a SELECT, instantiate objects from each returned row, then apply a filtering rule to that object, th…

True, but on the other hand if the developer gets told they are loading nearly 20,000 objects they should be smart enough to realize something is going to need to be optimized here, and although I don't know how to do it I better find out.

Re: How not to structure database-backed web apps: performance bugs in the wild

#110

Post by Arnt »

manigandham wrote: post #85
TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

Not this again. ORMs are tools, basically dynamic code generators that run SQL and map the results to in-memory objects, and vice-versa. Some are simplistic and others are incredibly advanced, and the code itself is usually faster than your own sql->objects logic that you would write otherwise. The issues with performance are almost always with the way the tool is used, like choosing a bad algorithms or the wrong dat…

Just a UI issue.

Compilers will warn you if you do stupid common mistakes. Not all mistakes, but many of the stupid common ones. If you make stupid common mistakes with an ORM, why doesn't the ORM warn you?

Post reply on HN