Lessons learned defying Joel Spolsky with Django • Hacker News Board

Re: Lessons learned defying Joel Spolsky with Django

#111

Post by gngeal » Sat, Apr 27, 2013, 6:03 PM UTC

drdaeman wrote: post #27

Aqueous wrote: post #15

Earlier quoted context omitted.

Why not just write the SQL?

The only problem I see, is we can't rewrite SQL. I.e., in pseudocode: query = SQL("SELECT * FROM entities WHERE owner = ?", owner=me) ... if some_condition: query = query + SQL.WHERE("OR public = TRUE") ... if other_condition: query = query + SQL("LEFT JOIN things AS t" " ON t.entity_id = entities.id") \ + SQL.WHERE("things.value > 0") ... my_nice_list_of_results = run(query + SQL("LIMIT ?", count)) This should be te…

"The only problem I see, is we can't rewrite SQL."

I believe that the phrase you're looking for is "lack of compositionality".

Re: Lessons learned defying Joel Spolsky with Django

#112

Post by jokull » Sat, Apr 27, 2013, 6:04 PM UTC

simonw wrote: post #94

jokull wrote: post #75

Earlier quoted context omitted.

Django middlewares is a poorly designed system. Flask has all of the things you mentioned except for forms. For that you can use Flask-WTF, but these libraries are becoming outdated anyway because they’re HTML based and don’t work so well for JSON validation.

I've found Django forms to work extremely well for validating non-form-based data - the API is a neat way of representing a set of validation rules and running them against a set of data.

Hi Simon - I greatly appreciate your work and opinion! The reason I think HTML forms libraries are insufficient for the problem domain: JSON is richer, both in datatypes and structure. We’re working on a forms library of sorts that is better suited for API serialization/validation and having one central place to define models for multiple internal DB’s.

https://github.com/plain-vanilla-games/schematics

Re: Lessons learned defying Joel Spolsky with Django

#113

Post by macspoofing » Sat, Apr 27, 2013, 6:24 PM UTC

jroseattle wrote: post #91

macspoofing wrote: post #71

Earlier quoted context omitted.

>I've never bought into Spolsky's vision that re-writing code is poor strategy. I think it's something that's true in general but false in some specific cases. Rewriting involves spending enormous time and resources to at best standstill, and at worst move backwards ( chances are your re-written product will be poorer in features, and initially buggier than your old, stable, battle-tested version). For smaller compan…

Yes, very true. Although I'd like to add some context. > Rewriting involves spending enormous time and resources to at best standstill, and at worst move backwards ( chances are your re-written product will be poorer in features, and initially buggier than your old, stable, battle-tested version). There is plenty of evidence that this has happened in many places and with many companies. A very real scenario that's pl…

>Spolsky's hard-line essentially says re-building your application from scratch is bad strategy.

I don't think Spolsky's position is as "hard-line" as you think it is. I always took it as a very very strong "rule-of-thumb". Every time a rewrite is proposed, it should sound warning bells in all stakeholder. And we're talking about a clean-room rewrite, and not a sub-module rewrite. Spolsky has no problem with rewriting and re-architecturing sub-pieces of an application.

Twitter is a great example of Spolky's philosophy. They had a big architectural and technology problem with Ruby and RoR, but they didn't do a clean-room rewrite with Java. Instead they tackled one sub-component at a time, and now they have something that rocks, but still allowed them to keep the time/code investment they made originally. Just as well, it allowed them to incrementally 'upgrade' their service.

Re: Lessons learned defying Joel Spolsky with Django

#114

Post by jl6 » Sat, Apr 27, 2013, 6:24 PM UTC

Aqueous wrote: post #15

zzzeek wrote: post #4

Don't use just one ORM and then declare "ORM's are stupid". The "object = None" / "object_id = None" issue illustrated here is certainly not a mistake every ORM makes.

Why not just write the SQL?

Just my opinion of course, but it seems that designers who think primarily in terms of the application will prefer to use an ORM to abstract away details of the storage layer; whereas a designer who thinks primarily in terms of data will prefer to write SQL, with the application being just one of many possible applications of that data.

Re: Lessons learned defying Joel Spolsky with Django

#115

Post by hcarvalhoalves » Sat, Apr 27, 2013, 6:31 PM UTC

tomsthumb wrote: post #95

hcarvalhoalves wrote: post #62

Earlier quoted context omitted.

The advantages aren't really in using Flask, but being able to use a better ORM (or no ORM at all, just SQL abstraction) and template engine. After working with Django ORM for a large project, I started to rethink the usefulness of models. It turns out just treating data as sets lends to a functional style and is simpler down the road.

> It turns out just treating data as sets lends to a functional style and is simpler down the road. Could you elaborate on this or point in the direction of something that does? I'm pretty sure I get the gist but having some more meat to chew to make sure the perspective is fleshed out fully would be awesome.

The bottom line is, tying data to objects seems like an elegant idea, but in practice it sucks. ORMs introduce various problems, like statefulness (when the whole point of dealing with a database is atomicity) and mismatches (your data doesn't necessarly map to one row or one object, it doesn't have to). I'll try to give two examples.

Consider how people often write update code on Django:

    instance = Model.objects.get(id=some_id)
    instance.foo = bar
    instance.save()

This sucks a lot. It's doing two queries, and you have concurrency issues. This would be optimal, since it's atomic:

    Model.objects.filter(id=some_id).update(foo=bar)

But in this case, it's not instantiating any object nor triggering any signals, so what's the point of the ORM after all? We might as well just use a sane DB API.

Here's another place where Django's ORM fails:

    Model.objects.annotate(Count('foos')).filter(foos__lt=10)

This won't work. Django will complain 'foos' is not a field on the model - even though 'foos' is a column in the result set, and it's perfect valid to SELECT on it. I reported this as a bug, but because the ORM works with model definitions and the result set can contain any column, what the ORM is really supposed to do or not is murky, so it's WONTFIX. This is one instance where data doesn't map to an object and the ORM concept crumbles.

There are many other pain points with ORMs, and Django's in particular, but these are the highlights for me. For an elegant querying API, in my opinion, check out RethinkDB. It doesn't depend on schemas (therefore, ORMs) and it supports map/reduce semantics, which solves 100% of what you need to do with data.

Re: Lessons learned defying Joel Spolsky with Django

#116

Post by zzzeek » Sat, Apr 27, 2013, 6:37 PM UTC

white_devil wrote: post #86

zzzeek wrote: post #18

Earlier quoted context omitted.

The word "stupid" here is ambiguous, so can be interpreted as, "stupid", "it's stupid to use them" (I disagree), or "stupid", "they don't necessarily understand your intent" (this applies to all software), "stupid", "simple issues are made more complicated than they should be, to an unreasonable extent" (this is a problem that varies to a significant degree based on the ORM in use and cannot be generalized based on t…

The problem with ORMs seems to be that they're a leaky abstraction. I've been very happy using SQLAlchemy for quite a while now, but I'm intentionally only using the SQL Expression API. Based on my experience with Hibernate, there's always something you want to accomplish that necessitates circumventing your ORM. Not only that, but typically you end up balancing between stuff being unavailable because of lazy-loading…

> The problem with ORMs seems to be that they're a leaky abstraction.

I disagree it's a problem.

going back to Joel again (!) :

"All non-trivial abstractions, to some degree, are leaky."

I talk about this a lot in this particular talk: https://www.youtube.com/watch?v=E09qigk_hnY

Hibernate was a great influence on me but I like to think that it only introduced some ideas in rough form that we've all had many years to improve upon.

The ORM will of course introduce new issues to deal with but this is because it's taking care of a vast amount of persistence code you no longer have to write, and applies a consistency to that persistence logic that would be extremely difficult to achieve without using tools.

Re: Lessons learned defying Joel Spolsky with Django

#117

Post by gnaffle » Sat, Apr 27, 2013, 6:39 PM UTC

jroseattle wrote: post #91

macspoofing wrote: post #71

Earlier quoted context omitted.

>I've never bought into Spolsky's vision that re-writing code is poor strategy. I think it's something that's true in general but false in some specific cases. Rewriting involves spending enormous time and resources to at best standstill, and at worst move backwards ( chances are your re-written product will be poorer in features, and initially buggier than your old, stable, battle-tested version). For smaller compan…

Yes, very true. Although I'd like to add some context. > Rewriting involves spending enormous time and resources to at best standstill, and at worst move backwards ( chances are your re-written product will be poorer in features, and initially buggier than your old, stable, battle-tested version). There is plenty of evidence that this has happened in many places and with many companies. A very real scenario that's pl…

I think the big danger Spolsky warns about is the "stop working on the old code and spend a year doing a complete rewrite". Many very smart people have underestimated the risks in doing that and the time it will take, while overestimating the benefits.

A smarter approach is to rewrite parts of the code. Apple has done that with many apps (and even iOS is OSX with a new UI/API on top). Linux has had many patch-by-patch rewrites as well. With a webapp, a "complete rewrite" still might leave most of the heavy lifting to the web framework, and you might be able to keep most of your HTML, CSS and JS.

I think the well-executed rewrites usually include launching an entirely new product while still supporting the old one. Adobe did this with InDesign (replacing PageMaker), Apple did this with OSX (the initial releases included OS9) and Microsoft did it with NT while keeping Windows 9X around.

Re: Lessons learned defying Joel Spolsky with Django

#118

Post by beambot » Sat, Apr 27, 2013, 6:40 PM UTC

Is there a (publicly available) video that accompanies this slide deck?

Re: Lessons learned defying Joel Spolsky with Django

#119

Post by zzzeek » Sat, Apr 27, 2013, 6:40 PM UTC

mwsherman wrote: post #92

k4st wrote: post #29

Earlier quoted context omitted.

The SQL generated appears correct, unless the underlying database can guarantee that all foreign key constraints are met. That is, I consider this a failure of the user of the ORM to appreciate that the two invocations of filter are not identical. In the former case, the query is verifying that the object_id field cannot be used to find a foreign object--regardless of the value of object_id. This is exactly what it i…

Correct, these are semantically different. You’ve described it well. The ORM would be incorrect if it did not generate the SQL that it did. Part of the issue is confusing object data with metadata. The id is an implementation detail of the data store – metadata. What the user is trying to do here is “talk database” and “talk model” at the same time. Which is a perfectly good argument against an ORM. But one should co…

If the schema is known to have a FOREIGN KEY constraint on sometable.other_id, then the SQL is as close as can be to wrong without actually returning the wrong answer. In the presence of the FK, the two queries have the same intent. But the first produces a vastly complex query plan for no reason, and this has nothing at all to do with the application/model layer.

Edit: plus! even if you want the same answer assuming that FKs are not in place and that bogus values might be present in other_id, the query is still far less efficient than it should be. You should be doing this:

    select * from sometable where not exists 
    (select 1 from othertable where othertable.id=sometable.other_id)

compare the query plans on any reasonable database and see (and yes, SQLAlchemy produces the NOT EXISTS form when the relationship is a one-to-many versus many-to-one and you ask it for objects with empty collections).

Re: Lessons learned defying Joel Spolsky with Django

#120

Post by espeed » Sat, Apr 27, 2013, 6:42 PM UTC

zzzeek wrote: post #40

PommeDeTerre wrote: post #21

Earlier quoted context omitted.

You know, you could've just directly mentioned SQLAlchemy earlier...

I don't like making my posts look like plugs for my own stuff, though I guess it's unavoidable....

In these type of discussions, it's not a plug -- it provides examples and evidence useful for teaching, which can light a path to a better way.

As Abelson and Sussman point out in SICP, "Computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute" (http://mitpress.mit.edu/sicp/front/node3.html).

The software we write is a codified expression of what we think, distilled into a working example that can elevate a discussion from theoretical-based to evidenced-based. Those who have devoted time to think through an issue deeply and have codified their thinking into software would be doing a disservice by not referencing it.