Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to structure database-backed web apps: performance bugs in the wild

blog.acolyer.org

Post reply on HN
Print view View original

151–160 of 319 posts

Re: How not to structure database-backed web apps: performance bugs in the wild

#151

Post by olavk »

EGreg wrote: post #141
olavk wrote: post #128

Earlier quoted context omitted.

So you have to use sharding because your databases have limited capacity. But you can just join in the app, because the app have unlimited memory?

The app doesn’t hold the entire database at a time. The app simply does the following: 1) Get the root record(s) from id(s) 2) See what related records it needs, combine them into a list of ids, partition list by shard 3) Ask each shard for the corresponding records 4) Repeat from 2 if necessary 5) Return this whole tree / graph to the user Graph databases can do this in O(1) instead of O(log N) lookups. Relational j…

A graph database basically caches related entities with the root entity, so lookups are fast when the query follows the paths of the graph. The price is that any other query is extremely costly. But if it works for your use case, more power to you. But not really relevant in a discussion about ORMs - Object Relational Mappers.

The relational model was designed to address the limitations of the network/graph database, especially to allow arbitrary (ad-hoc) querying and to decouple the physical storage from the logical model. But if you don't need all that, a graph database may be fine.

Re: How not to structure database-backed web apps: performance bugs in the wild

#152

Post by Rafert »

Interestingly their code checker is just a bunch of regular expressions: https://github.com/hyperloop-rails/static-checker - I would've expected custom Rubocop rules.

Rubocop already knows about `where.first? => find_by` for exmple: https://github.com/rubocop-hq/rubocop/blob/master/lib/ruboco...

Re: How not to structure database-backed web apps: performance bugs in the wild

#153

Post by chii »

AmericanChopper wrote: post #114
vidarh wrote: post #100

Earlier quoted context omitted.

It is the developer, but ORMs are so controversial in part because they often obscure that you're doing something crazily ineffective in ways that makes developers that don't understand the abstraction fail to see that they're doing something obviously wrong. It's more stark that you're doing something crazy if you do a SELECT, instantiate objects from each returned row, then apply a filtering rule to that object, th…

Perhaps the anti-pattern to rule them all is that as things become easier to do, it gives more people the opportunity to do them badly. We have so many incredible tools, and simple but powerful high level languages that allow us to write apps without bothering with complicated stuff like assembler. However, these tools don’t mean that we get to ignore how CPUs work, and ORMs don’t mean that we get to ignore how data…

> as things become easier to do, it gives more people the opportunity to do them badly.

ala, this is why we have electron, php, and javascript. None of those things behave badly, but it's very easy to write something useable in those, but resource intensive, or insecure, or poorly maintainable.

Re: How not to structure database-backed web apps: performance bugs in the wild

#154

Post by mullsork »

AsyncAwait wrote: post #122
strken wrote: post #77

Earlier quoted context omitted.

One of the biggest problems with ActiveRecord (the ORM in the article) is that it uses extremely obtuse names for extremely common methods. There are three different methods for finding how many things meet a given criteria[0], and none of them do the same thing. There are three different methods for loading data from an associated table[1], and none of them do the same thing either. None of the method names explain…

This is mostly due to its popularity and various people needing different use cases, the long-time maintainer is now working on a mach cleaner ORM for Rust, called Diesel[1]. 1 - http://diesel.rs

Compile time SQL query checks... amazing! I've been trying to find a similar library for a while. Thanks for linking

Re: How not to structure database-backed web apps: performance bugs in the wild

#155

Post by tfigment »

TekMol wrote: post #132
fuzzy2 wrote: post #94

Earlier quoted context omitted.

Something along the lines of db.Query ("SELECT * FROM Cars WHERE InsuranceEndDate /edit: Sorry, missed the "number of". Well, you get the idea. It'd use 'QuerySingle' instead.

That does not look very ORMish to me. It might return objects. But isn't one point of an ORM that you tell the ORM which data you want and not what SQL query to send to the DB?

Also Dapper fan. I like mapping into objects without code generation and complicated queries are easier to write compared to LINQ if familiar with actual SQL. Only thing I really don't like is bulk inserts as it will do one statement per object. You can still use both Dapper and LINQ at same time but it might confuse others.

Re: How not to structure database-backed web apps: performance bugs in the wild

#156

Post by saiya-jin »

gerbilly wrote: post #150
vidarh wrote: post #100

Earlier quoted context omitted.

It is the developer, but ORMs are so controversial in part because they often obscure that you're doing something crazily ineffective in ways that makes developers that don't understand the abstraction fail to see that they're doing something obviously wrong. It's more stark that you're doing something crazy if you do a SELECT, instantiate objects from each returned row, then apply a filtering rule to that object, th…

>[...] ORMs are so controversial in part because they often obscure that you're doing something crazily ineffective ... Then just turn on detailed SQL logging while you are coding and keep an eye on the queries generated by your ORM calls. If you do this methodically for all your DAOs, then you should spot performance problems early when they can still be easily fixed.

With all this continuous effort required, wouldn't it be simply easier to ditch the whole 'use ORM because it makes life easier' nonsense and use just some SQL-mapping tool like iBatis? Where you really, truly have 100% immediate control over queries, no dancing around with logging and wasting time figuring out why the lib decided to fetch this and not that.

I honestly don't get the fear some people have from SQL. If you can't do SQL good enough for 98% of use cases out there, you are not senior dev, not even experienced one. It's just one of those basic dev requirements that won't go away in next 50 years anyway.

Re: How not to structure database-backed web apps: performance bugs in the wild

#157

Post by viraptor »

dagw wrote: post #76
TekMol wrote: post #56

Earlier quoted context omitted.

The ORM makes it more likely to do stuff in the application instead of the DB because many queries are hard or impossible to represent in an ORM fashion. Additionally, the ORM makes it much harder to see what is going on under the hood. For example you don't know if a value of an object is stored in the main table or lazy loaded from a related table. So you have no clue if echo "$user.name lives in $user.city" result…

The ORM makes it more likely to do stuff in the application instead of the DB Again this doesn't match my experience. A mediocre .NET developer who can access the database with LINQ is far more likely to run their queries on the database than a mediorce .NET developer who has to try to write SQL queries by hand. And the truth is that modern ORMs are often pretty clever with their optimizations, and in many cases the…

Also ORMs have enough visibility into your code to tell you of for doing stupid things. There are libraries working with ActiveRecord (and likely others too) which will warn you that you could skip the specific query if you added .eager_load(). Or about 1+N you're doing needlessly.

This can be done on SQL itself, but it would be much harder.

Re: How not to structure database-backed web apps: performance bugs in the wild

#158

Post by Ntrails »

TekMol wrote: post #21

Give me an O! Give me an R! Give me an M! What does that spell? SLOW PERFORMANCE! Todays programmers dont understand data. They understand frameworks. To find the nr of all cars that are out of insurance they write: 10 Nr=0 20 Hey framework, give me all cars! Framework: Ok, here are 8001093 business objects representing all the cars in our DB. Each has all the attributes the car has. Color, mileage etc. 30 Thanks! 40…

Interesting. I have had a couple of people in interviews (who know SQL/Databases on cv) writing code a lot like that pulling in two whole tables, joining them, and then filtering the joined results.

It is possible i was overly harsh in my judgement, as assumed meant no real familiarity at all.

Re: How not to structure database-backed web apps: performance bugs in the wild

#159

Post by acdha »

strken wrote: post #77
grey-area wrote: post #54

Earlier quoted context omitted.

I use an ORM. I don't perform queries like that because the ORM makes it easy to build complex queries, joins, limiting the data returned etc, then execute them in one query, it's a convenience, not a straightjacket. The problem is not ORMs, the problem is just people not thinking about the resources their query over some data takes, and trying to do things in memory that are better done in the database (as in your e…

One of the biggest problems with ActiveRecord (the ORM in the article) is that it uses extremely obtuse names for extremely common methods. There are three different methods for finding how many things meet a given criteria[0], and none of them do the same thing. There are three different methods for loading data from an associated table[1], and none of them do the same thing either. None of the method names explain…

> This is a problem that goes beyond the usual ORM antipatterns and bad database designs and becomes its own special kind of hell, because it makes code far harder to reason about and thus far harder to review, in a way that I don't remember encountering in Hibernate, Gorm, or anything else.

I feel like this brings up an annoying cultural problem in tech wgere developers hit intermediate skill levels and feel the need to proclaim their expertise by writing something in the you’re-doing-it-wrong genre but don’t have the breadth of experience to realize that they’re over-generalizing and so they go from “ActiveRecord has this problem” to “ORMs are bad” without asking whether anyone else has it better. I’ve read tons of similar posts where e.g. Java proved that exceptions, OOP, or static typing were bad or Twitter proved you shouldn’t use Rails.

Re: How not to structure database-backed web apps: performance bugs in the wild

#160

Post by TeMPOraL »

crdoconnor wrote: post #133
bryanrasmussen wrote: post #109

Earlier quoted context omitted.

True, but on the other hand if the developer gets told they are loading nearly 20,000 objects they should be smart enough to realize something is going to need to be optimized here, and although I don't know how to do it I better find out.

That depends upon the context. I've seen developers argue over optimizing ETL processes that weren't in the slightest bit time sensitive, took 25 minutes and were going to be run once. More often than not I've seen developers fret over performance when it's not actually bothering users. This fetishization of performance seems to be a cultural thing in tech. Conversely, data integrity, normalization and transactionali…

> This fetishization of performance seems to be a cultural thing in tech.

Tell me where you've seen that, because I'd love to move there. From my experience, there is a common fetishization of non-performance. The "premature optimization" adage taken to its extreme - "we can buy more hardware", "developer time > machine time", "who cares about wasting electricity of millions of our users", etc.

Post reply on HN