Live data from Hacker News

Tips for Building High-Quality Django Apps at Scale

blog.doordash.com

61–70 of 125 posts

Re: Tips for Building High-Quality Django Apps at Scale

#61
post #34

I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat un…

Where app directories are incredibly useful is in Django library development. They're a really convenient way to hook in models, static files, template helpers, etc. to an existing Django site, without forcing users to do a bunch of setup work when integrating your library. It's not a coincidence that most Django libraries have a very similar setup: 1. Install the package 2. Add the package to settings.INSTALLED_APPS…

Yeah, I agree. Makes sense for libraries. So keep that separate for library writers, don't expose me to the complexity of that.

Re: Tips for Building High-Quality Django Apps at Scale

#62

I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat un…

Have you checkout out O'Reilly's Lightweight Django book? It basically goes through all of this, starting with a single file Django app.

No! I read a lot of books, but missed that. Thanks!

Re: Tips for Building High-Quality Django Apps at Scale

#63
post #46
post #41

Earlier quoted context omitted.

> I have been programming for almost 15 years, and have never once needed to use migrations. > Have I ever had to add a column to a database? Absolutely, ... then you've used a migration. A statement like "I've never once had to use a migration" is so obviously incorrect that it's comical. Your contradiction two lines down makes it more so. The rest of your comment is fairly misinformed about a few things, full of FU…

Oh please, tell me how I'm misinformed? If you need a migration library because you added a column to your database, then you have a serious problem. If you need a migration library because you don't know how to write SQL, then you have a serious problem. If your data structure is truly changing that often, then you shouldn't be using a SQL database in the first place, and you have a serious problem.

Again, FUD. If you really need me to spell out why then I will, but I doubt it will change your viewpoint.

Let me spell it out for you:

You build your awesome [app] for [client]. You get your schema bang on first time and it's all working absolutely fine. Everyone is happy.

Then [client] comes to you and says "[app] is awesome, but we need [x]". You think, and [x] needs some database modifications. Maybe it's a new column to store something, or perhaps it's a column that now shouldn't be nullable. Maybe it's a new table entirely. Perhaps what you need is already implemented in library [y], which needs database modifications of some kind like it's own tables. The possibilities are endless.

Great. So how do we do this. Obviously we need to run some SQL on the database. So you change all your code, press deploy, and really quickly log onto the production server and smash out some artisanal SQL to modify the database before any requests hit the server. Everyone is happy! You quickly write an email all your co-workers to do these migrations locally so they can run the app.

Oh no, but wait, you forgot to do operation Z! Now your app and everyone elses is broken. You resolve to ensure this doesn't happen again by writing a .sql file for each migration and running them in sequence. Great, you can even check them into your VCS and share them with your co-workers. Awesome! You can even make them part of your CI build (you do that, right?).

You do another deploy, but nuts, something went wrong! You need to roll a migration back! But you only added forward migrations in your .sql files! Ok, so lets add [migration]-backwards.sql and [migration]-forwards.sql from now on. Awesome! So you've got a forwards and backwards migrations. But... wait, we need to run some Python code to do something as part of the migration. Ok... lets add a [migration]-{forwards}.py as well.

Fantastic. But then you start to install package [z] because it does exactly what the client wants and you realize the benefits of using well tested, widely used libraries in your systems and not writing everything by hand. This requires some database modifications. The author of this package has the migrations it needs but it's in it's own home-grown format and it's only for MySQL and Oracle! Nuts, ok, so you translate them by hand and add them to your migrations. But you made a mistake somewhere in the translation and everything breaks. You fix it by hand.

Congratulations. You've just written a shit version of Django migrations that everyone hates and that doesn't really work.

tl;dr: schema changes are a natural part of development. You, me, and everyone else working on any kind development project has done them. They happen, this is a fact. If you need to:

1. Share the migrations in your team

2. Run them as part of your CI build

3. Use third party migrations for packages (including built in contenttypes or other core Django tables)

4. Have them in a format that works for any supported database

5. Have rollback and the ability to execute arbitrary code as part of the migration

Then you can either write your own shitty version or use a well supported, built in system that handles the complexity for you. This is not a bad thing.

I can't comprehend your viewpoint, which mostly boils down to "these damn kids are on my lawn", so I can only imagine your setup is some ad-hoc .sql files that you run by hand.

This is a bad thing.

Re: Tips for Building High-Quality Django Apps at Scale

#64
post #46
post #41

Earlier quoted context omitted.

> I have been programming for almost 15 years, and have never once needed to use migrations. > Have I ever had to add a column to a database? Absolutely, ... then you've used a migration. A statement like "I've never once had to use a migration" is so obviously incorrect that it's comical. Your contradiction two lines down makes it more so. The rest of your comment is fairly misinformed about a few things, full of FU…

Oh please, tell me how I'm misinformed? If you need a migration library because you added a column to your database, then you have a serious problem. If you need a migration library because you don't know how to write SQL, then you have a serious problem. If your data structure is truly changing that often, then you shouldn't be using a SQL database in the first place, and you have a serious problem.

Okay, so you changed the DB schema in your 'dev' environment to add a column.

1. In what manner/format do you define or commit the change to VCS so it is in sync with the corresponding code change (that uses the new column)?

2. What happens when a teammate comes to deploy that schema change to production?

3. What happens when we need to roll back production to some previous commit? The need to rollback may/may-not relate to the schema change (not suggesting you 'got it wrong').

If you can answer 1, 2, 3 by some convention ... you've created a migration system that should probably be codified.

Re: Tips for Building High-Quality Django Apps at Scale

#65
post #10

I've got more than 5 years of experience with Django on a number of teams and at a couple of companies and in my experience almost everything in this article is completely incorrect. The only things I would agree with is the point about project layout and avoiding django's squashmigrations for the truncate the migrations table, delete the migrations, and create a new initial migration. Practically everything else in…

I've got 10 years' experience on projects small and large and I have to agree. The title talks about building at scale but the article doesn't stress that which makes some of the advice downright weird. >If you don't really understand the point of apps, ignore them and stick with a single app for your backend. You can still organize a growing codebase without using separate apps. This is where the article lost me. If…

I agree the doordash article gets some stuff right and most stuff wrong, almost to the point where it's difficult to read. But (somewhat tangentially) I admit I have struggled in the past with separating out Django apps for reasons not mentioned in the article.

Specifically, say I have two apps, with a second more specific app heavily dependent on a first more general app. What I find in this scenario is that I sometimes need hooks into the general app from the specific app, which means that I wind up importing modules from the specific app into the general app. This hasn't generally been a showstopper in my experience, but it creates some friction because:

a) I would prefer for the general app to have no dependencies on the specific app

b) This results in circular imports (which can themselves be addressed, but this is an implementation detail I would prefer not to have to worry about)

I realize these issues can be mitigated with signals, but I try to use signals sparingly for various reasons (https://code.djangoproject.com/ticket/16547#comment:2). It also helps that foreign keys can be expressed using a string literal rather than the actual model, but in the end, I still occasionally run into situations I don't feel great about.

Please note that I'm not advising against separating out functionality into apps. Instead, I'm merely citing an issue about having multiple apps that bothers me.

Re: Tips for Building High-Quality Django Apps at Scale

#66
post #51

Earlier quoted context omitted.

How do you add columns to your db? Do your requirements never change? I am fully capable of doing everything done with migrations with SQL - but I always use migrations (even for custom sql outside of the ORM's capability). Why? because then we have a record of the schema state at any given moment (that is in sync with the application code at that point) and how it got to where it is now.

If you're leaning on migrations often, with SQL, then I think you should not be using SQL, obviously (or you're just doing something wrong) Or, one should use a SQL database that allows more abstract data types (like JSON blobs in Postgres)

And how do you keep your database schema under version control?

Edit: Do all you downvoters not keep track of your schema changes as you develop your application (via migrations, or even sql files), so you can easily roll back changes if needed?

Re: Tips for Building High-Quality Django Apps at Scale

#67
post #41
post #30

Earlier quoted context omitted.

> I'm not sure why you put quotes around migrations as if it's some alien, obscure or weird feature. If you've never written a web application that needs migrations then you're not writing the kinds of applications (or indeed any 'serious' application) that would benefit from Django IMO. I have been programming for almost 15 years, and have never once needed to use migrations. I have worked on projects for multiple y…

> I have been programming for almost 15 years, and have never once needed to use migrations. > Have I ever had to add a column to a database? Absolutely, ... then you've used a migration. A statement like "I've never once had to use a migration" is so obviously incorrect that it's comical. Your contradiction two lines down makes it more so. The rest of your comment is fairly misinformed about a few things, full of FU…

Please keep this kind of programming flamewar off HN. If someone is wrong and you want to explain how, do so civilly.

We've unfortunately had to warn you a bunch of times already about being uncivil on HN. This process isn't infinite and ends in an account getting banned, so would you please fix this?

Re: Tips for Building High-Quality Django Apps at Scale

#68
I find that the safest way to migrate DB schema is gradually, spreading out intermediate steps across a few deploys.

Migrate the schema creating the (initially unused) fields in advance. Step by step change your logic where it talks to the storage—start querying new field values with a fallback, use new fields for incoming data. Migrate the existing data. Remove the fallbacks. Ensure the old fields are not used at this point. Migrate the schema, finally deleting the old fields.

Don’t rush, let every step reach production. In deployment sequence, apply migrations after the new code is already running. Goes without saying, monitor error rates for spikes.

Yes, at some points in time your production DB schema won’t be fully normalized. In return for prolonged messiness you get smoother flow, no single fateful and stressful deployment that attempts to get it right all in one go.

Migration handling is the only part of the article I find a bit foggy and debatable. The rest roughly resonates with what I’ve arrived at through my years of experience with Django web apps.

Re: Tips for Building High-Quality Django Apps at Scale

#69
post #46
post #41

Earlier quoted context omitted.

> I have been programming for almost 15 years, and have never once needed to use migrations. > Have I ever had to add a column to a database? Absolutely, ... then you've used a migration. A statement like "I've never once had to use a migration" is so obviously incorrect that it's comical. Your contradiction two lines down makes it more so. The rest of your comment is fairly misinformed about a few things, full of FU…

Oh please, tell me how I'm misinformed? If you need a migration library because you added a column to your database, then you have a serious problem. If you need a migration library because you don't know how to write SQL, then you have a serious problem. If your data structure is truly changing that often, then you shouldn't be using a SQL database in the first place, and you have a serious problem.

Please don't get involved in technical flamewars on HN (or other flamewars). As agitation goes up, information goes down, and these discussions turn into back-and-forth spats that benefit no one, including the combatants.

Re: Tips for Building High-Quality Django Apps at Scale

#70
post #10

Earlier quoted context omitted.

I've got 10 years' experience on projects small and large and I have to agree. The title talks about building at scale but the article doesn't stress that which makes some of the advice downright weird. >If you don't really understand the point of apps, ignore them and stick with a single app for your backend. You can still organize a growing codebase without using separate apps. This is where the article lost me. If…

> Avoiding "fat models" is another place where it feels more like opinion than anything to do with performance or good design So in the Java world, the general pattern is that: Views: - Accept and sanitize query parameters - Call call one or more service methods. - Catch errors and return an appropriate error response - Render a JSON response based on the results of the service methods if nothing goes wrong. Service…

Is there a good place or pattern for service methods in Django? I've got some very fat models right now, and it's a DRY improvement over having the fat in the views, but like you say it takes a lot of effort to trace what's going on.
Post reply on HN