Tips for Building High-Quality Django Apps at Scale
91–100 of 125 posts
Re: Tips for Building High-Quality Django Apps at Scale
#92Earlier quoted context omitted.
If you're leaning on migrations often, with SQL, then I think you should not be using SQL, obviously (or you're just doing something wrong) Or, one should use a SQL database that allows more abstract data types (like JSON blobs in Postgres)
And how do you keep your database schema under version control? Edit: Do all you downvoters not keep track of your schema changes as you develop your application (via migrations, or even sql files), so you can easily roll back changes if needed?
The models live in version control. It's that simple.
If your data structure is changing so often in such a way that you need to keep a history, then I'm afraid you have some fundamental problems that no amount of tooling or process can solve. You will not be able to scale your development efforts with this type of constantly changing data model, even with a migration library.
If these are truly problems that you or your team has, then you should hire someone to help you organize your application, data, and workflow in a more effective manner.
Re: Tips for Building High-Quality Django Apps at Scale
#93Earlier quoted context omitted.
Oh please, tell me how I'm misinformed? If you need a migration library because you added a column to your database, then you have a serious problem. If you need a migration library because you don't know how to write SQL, then you have a serious problem. If your data structure is truly changing that often, then you shouldn't be using a SQL database in the first place, and you have a serious problem.
Again, FUD. If you really need me to spell out why then I will, but I doubt it will change your viewpoint. Let me spell it out for you: You build your awesome [app] for [client]. You get your schema bang on first time and it's all working absolutely fine. Everyone is happy. Then [client] comes to you and says "[app] is awesome, but we need [x]". You think, and [x] needs some database modifications. Maybe it's a new c…
You are setting up a comical hypothetical situation in which the developer is blundering through their job in a mindless inept manner.
I'll just stop you right there.
No amount of best practices, process, or tooling will fix an incompetent developer who is part of a disorganized team that are working on a pile of bad architecture.
Re: Tips for Building High-Quality Django Apps at Scale
#94Earlier quoted context omitted.
Oh please, tell me how I'm misinformed? If you need a migration library because you added a column to your database, then you have a serious problem. If you need a migration library because you don't know how to write SQL, then you have a serious problem. If your data structure is truly changing that often, then you shouldn't be using a SQL database in the first place, and you have a serious problem.
Okay, so you changed the DB schema in your 'dev' environment to add a column. 1. In what manner/format do you define or commit the change to VCS so it is in sync with the corresponding code change (that uses the new column)? 2. What happens when a teammate comes to deploy that schema change to production? 3. What happens when we need to roll back production to some previous commit? The need to rollback may/may-not re…
1. Change model code to add the new field, test, and commit it to version control. Then it's merged to staging, where it's tested in a production-like environment with a recent production data snapshot. After that's approved, it goes to master/production servers.
2. git pull then restart some wsgi/worker processes
3. git checkout to desired revision and restart some wsgi/worker processes
It's perfectly fine to have models comprised of only a subset of the data they interact with.
The models do not have to reflect all fields that exist in the database.
The models only reflect information that is necessary for the application in which said models are used.
Basically, if you clobber your database, no amount of migration tooling or process will help you. Ever. It will just get in the way, to be honest, and will prevent you from doing things properly in the first place.
Re: Tips for Building High-Quality Django Apps at Scale
#95Earlier quoted context omitted.
How does the ORM prevent horizontal scaling?
The ORM works against a relational database. The relational databases that Django supports are not distributed, right?
You can get horizontally scaled eventually consistent on reads just fine on postgres/mysql via replication. There are also write scaling options.
Re: Tips for Building High-Quality Django Apps at Scale
#96Earlier quoted context omitted.
I've got 10 years' experience on projects small and large and I have to agree. The title talks about building at scale but the article doesn't stress that which makes some of the advice downright weird. >If you don't really understand the point of apps, ignore them and stick with a single app for your backend. You can still organize a growing codebase without using separate apps. This is where the article lost me. If…
> Avoiding "fat models" is another place where it feels more like opinion than anything to do with performance or good design So in the Java world, the general pattern is that: Views: - Accept and sanitize query parameters - Call call one or more service methods. - Catch errors and return an appropriate error response - Render a JSON response based on the results of the service methods if nothing goes wrong. Service…
We do roughly this where I work, so I broadly agree.
That said, it's fairly common for the unit of reuse to be below service methods. Also, depending upon how exactly you manage transactions, another thing to look out for is making multiple non-idempotent service calls from the view - this will be an area ripe for race conditions you likely aren't testing.
Re: Tips for Building High-Quality Django Apps at Scale
#97Earlier quoted context omitted.
> why not make the reuse optional? Well, it is optional. You can just not divide you stuff and place everything inside the main app. But notice the article recreates the same layer of reuse inside the main app, and got some extra problems because of it (renaming the tables is probably caused by this). Django may need an extra paragraph at the tutorial, telling how it is ok to place everything on the main app when eve…
> Well, it is optional What I mean is, you still have to have a "main" app. Why? I don't want to reuse it. So I don't want an "app". I just want to build my website. Stop making me put things in an unnecessarily complicated directory hierarchy. If it were Java, I wouldn't think twice about it, but it doesn't seem Pythonic. > reverse Dunning-Kruger Good question! Probably... illusory inferiority AKA imposter syndrome.…
Sure. One of the Djangoesque opinions is that if you're building a site, you'd like it to be maintainable, and that you'll probably add on to it.
If you've been working in the Python ecosystem long enough to know that you're building a site that doesn't have those assumptions, you'll probably pick something different to start out with, like Flask or Bottle.
Re: Tips for Building High-Quality Django Apps at Scale
#98Earlier quoted context omitted.
> Avoiding "fat models" is another place where it feels more like opinion than anything to do with performance or good design So in the Java world, the general pattern is that: Views: - Accept and sanitize query parameters - Call call one or more service methods. - Catch errors and return an appropriate error response - Render a JSON response based on the results of the service methods if nothing goes wrong. Service…
We do roughly this where I work, so I broadly agree. That said, it's fairly common for the unit of reuse to be below service methods. Also, depending upon how exactly you manage transactions, another thing to look out for is making multiple non-idempotent service calls from the view - this will be an area ripe for race conditions you likely aren't testing.
> it's fairly common for the unit of reuse to be below service methods.
In terms of utility functions or serializers? What does that look like exactly?
E.g. in our codebase service methods can call helper methods (non-reusable), utility methods (reusable), and serializers (non-reusable).
Re: Tips for Building High-Quality Django Apps at Scale
#99Earlier quoted context omitted.
The ORM works against a relational database. The relational databases that Django supports are not distributed, right?
You scale your app server out and your database server up. You can have 100 django servers hitting one database with a read-replica, for example. Depending on your workload, this is often a great way to scale and pretty typical for Django. In most cases your app servers will saturate way before the database does.
Thanks for this. Yes this actually makes sense. I considered that a given, though, since scaling the Django app is trivial since it is stateless. Scaling the state is always the difficult thing.
Re: Tips for Building High-Quality Django Apps at Scale
#100I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat un…
It depends what you're doing. It shines on low level features like authentication and image cropping, where I need ways to do these things, and I can use them across the board.
It breaks down when you have a lot of views and urls that are trying to stay close to their related models, but are too interconnected to have any real kind of separation of concerns.
I find myself using all of the above. There's nothing wrong with building a small site in a single app, or nesting apps to provide a little more structure for how they're connected or separated.