Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for Building High-Quality Django Apps at Scale

blog.doordash.com

Post reply on HN
Print view View original

71–80 of 125 posts

Re: Tips for Building High-Quality Django Apps at Scale

#71

Post by linkmotif »

benwilber0 wrote: post #59
linkmotif wrote: post #35

Django is not meant to be run at scale. At least not horizontal scale. If you're not going to use the ORM, which prevents you from scaling horizontally, why are you using Django? Django can be scaled by getting large instances with lots of RAM, which is quite affordable these days. It can also be scaled with caching. But if you want horizontal scaling, don't use Django. That's not what it's for.

There's nothing really inherent to Django that prevents it from scaling horizontally, not even the ORM. For the typical SQL read-heavy workload it will scale as well as anything else depending on the scalability of the backing RDBMS. Should always use proper HTTP response caching with something like Varnish or via 3rd-party CDN where possible.

> depending on the scalability of the backing RDBMS

As far as I know, none of the backing RDBMSs that the ORM supports are horizontally distributable. At least not officially, right? CockroachDB with its Postgres interface sounds good, or Citus? But as of today, are any of the supported DBs distributed?

Re: Tips for Building High-Quality Django Apps at Scale

#72

Post by linkmotif »

ensignavenger wrote: post #53
linkmotif wrote: post #35

Django is not meant to be run at scale. At least not horizontal scale. If you're not going to use the ORM, which prevents you from scaling horizontally, why are you using Django? Django can be scaled by getting large instances with lots of RAM, which is quite affordable these days. It can also be scaled with caching. But if you want horizontal scaling, don't use Django. That's not what it's for.

How does the ORM prevent horizontal scaling?

The ORM works against a relational database. The relational databases that Django supports are not distributed, right?

Re: Tips for Building High-Quality Django Apps at Scale

#73

Post by mythrwy »

tinix wrote: post #27
Alex3917 wrote: post #24

Earlier quoted context omitted.

> it's admin interface is by far the best thing since sliced bread for internal/backoffice applications. By the way, some best practices here that I've discovered: - Use the Django admin for performing actions closely tied to specific models that don't involve any business logic. E.g. changing which one of a user's email addresses is their primary address. - For admin business logic that spans multiple tables, (e.g.…

> The benefit of having your admin business logic wrapped in REST endpoints is that you are writing and testing all your admin logic the same way as you write and test all your other endpoints. You write all your business logic in REST endpoints? That's insanity... Are you even doing RESTful things with all those endpoints? Have you even considered the impact of HTTP overhead? This is a thread about scalability, afte…

I don't see exactly what the mentioned problem has to do with writing REST endpoints though.

Agree with OP. REST endpoint in many cases are the way to go. Particularly today when the front end might undergo who knows what transition and not really have much to do with Django at all.

Re: Tips for Building High-Quality Django Apps at Scale

#74

Post by ralmeida »

mabcat wrote: post #70
Alex3917 wrote: post #28

Earlier quoted context omitted.

> Avoiding "fat models" is another place where it feels more like opinion than anything to do with performance or good design So in the Java world, the general pattern is that: Views: - Accept and sanitize query parameters - Call call one or more service methods. - Catch errors and return an appropriate error response - Render a JSON response based on the results of the service methods if nothing goes wrong. Service…

Is there a good place or pattern for service methods in Django? I've got some very fat models right now, and it's a DRY improvement over having the fat in the views, but like you say it takes a lot of effort to trace what's going on.

I'm not aware of architectural patterns for service methods in Django (would like to find some as well), but what I did was to somewhat mimic a Java structure.

All the project is in one single app, which I unimaginatively called "core", and inside this app there's a "services" Python package (i.e. a folder with a __init__.py file inside). These have roughly one Python module (.py file) for each "category" of services. For example, there's thin layers like "user_service.py" (basically passes through to the relevant models), to more complex services like "dependency_x_integration_service.py", which connects to external service "X" and pulls some relevant data (say, user interaction datapoints), and bridges them to the models in the system.

Re: Tips for Building High-Quality Django Apps at Scale

#75

Post by mythrwy »

tobltobs wrote: post #32
tinix wrote: post #30

Earlier quoted context omitted.

> I'm not sure why you put quotes around migrations as if it's some alien, obscure or weird feature. If you've never written a web application that needs migrations then you're not writing the kinds of applications (or indeed any 'serious' application) that would benefit from Django IMO. I have been programming for almost 15 years, and have never once needed to use migrations. I have worked on projects for multiple y…

You get your DB model 100% correct the first time, always? Even years later your DB model is able to accommodate all those always changing business requirements?

If you get your DB model 100% correct the first time I want to work for your clients!

Re: Tips for Building High-Quality Django Apps at Scale

#76

Post by squeaky-clean »

linkmotif wrote: post #72
ensignavenger wrote: post #53

Earlier quoted context omitted.

How does the ORM prevent horizontal scaling?

The ORM works against a relational database. The relational databases that Django supports are not distributed, right?

You scale your app server out and your database server up. You can have 100 django servers hitting one database with a read-replica, for example.

Depending on your workload, this is often a great way to scale and pretty typical for Django. In most cases your app servers will saturate way before the database does.

Re: Tips for Building High-Quality Django Apps at Scale

#77

Post by vosper »

mi100hael wrote: post #5
randomsearch wrote: post #2

I'm so glad to see the mention of "app" directories here. I've only dabbled in Django development, but I've always thought the desire to divide things into apps didn't really make any sense. It felt like the developers of Django had said "well, intuitively, there must be some unit of reuse at this level" and then stuck the notion of apps in there in an attempt to provide that reuse. This seems to me to be somewhat un…

Part of it is that Django has been around for quite a while now. It's maybe _the_ most succesful Python framework out there, so there are some paradigms that aren't common anymore but are hold-overs from years past. In particular, it pre-dates the current "microservice" trend and assumes a fairly monolithic environment. And I think "apps" _do_ make sense in certain contexts. Consider this situation: - I start a "poll…

Is it actually possible to find and drop in apps like you suggest? And have it just work? Is a repository of various blog apps out there? What if one of them does something nasty to your database?

It seems like it'd be easier to spin up a Wordpress instance...

Re: Tips for Building High-Quality Django Apps at Scale

#78

Post by ensignavenger »

linkmotif wrote: post #72
ensignavenger wrote: post #53

Earlier quoted context omitted.

How does the ORM prevent horizontal scaling?

The ORM works against a relational database. The relational databases that Django supports are not distributed, right?

May depend on your workload and what you mean by 'distributed'... there are some scalablleity options for postrgres. There are backend drivers for Oracle and MS SQL as well, but I have never used either and I'm not sure just how good they are.

But there is nothing that I am aware of inherent to the ORM itself that prevent horizontal scale.

Re: Tips for Building High-Quality Django Apps at Scale

#79

Post by benwilber0 »

linkmotif wrote: post #71
benwilber0 wrote: post #59

Earlier quoted context omitted.

There's nothing really inherent to Django that prevents it from scaling horizontally, not even the ORM. For the typical SQL read-heavy workload it will scale as well as anything else depending on the scalability of the backing RDBMS. Should always use proper HTTP response caching with something like Varnish or via 3rd-party CDN where possible.

> depending on the scalability of the backing RDBMS As far as I know, none of the backing RDBMSs that the ORM supports are horizontally distributable. At least not officially, right? CockroachDB with its Postgres interface sounds good, or Citus? But as of today, are any of the supported DBs distributed?

MySQL Cluster [1] has been around a long time for applications that need linear scalability with ACID guarantees

[1] https://www.mysql.com/products/cluster/

Re: Tips for Building High-Quality Django Apps at Scale

#80

Post by arctangent »

tinix wrote: post #30
orf wrote: post #19

Earlier quoted context omitted.

I agree with all of your points about the article, but... > However, they should take it a step further, and just avoid Django in the first place. Django is a tool, and like most tools it has a use. I find Django indispensable for writing specific kinds of applications, and it's admin interface is by far the best thing since sliced bread for internal/backoffice applications. It's not perfect by any means but it's ama…

> I'm not sure why you put quotes around migrations as if it's some alien, obscure or weird feature. If you've never written a web application that needs migrations then you're not writing the kinds of applications (or indeed any 'serious' application) that would benefit from Django IMO. I have been programming for almost 15 years, and have never once needed to use migrations. I have worked on projects for multiple y…

You've caught a lot of flak for your statement.

With the knowledge I have now it would easy for me to say "and rightly so", but I think I can understand where you're coming from.

Django didn't always have schema migrations built in. I think they appeared in version 1.7 about 2 and a bit years ago. Before then there were separate add-on apps that could do migrations for you.

When Django added built-in support for migrations I still didn't use them for a while because at the time I was primarily a solo developer and wanted to have full control over what happened in the DB. And because I was a solo developer it didn't matter. So in that sense you're comment is in some ways correct.

However, in recent times I've started to work as part of a team. And that's where migrations really start to shine, because there might be several different people making app changes that result in DB schema changes. Django stores migration files with information about which other migrations each one is dependent upon.

Without that kind of migrations system it would simply be impossible for developers working on combinations of different branches of a development tree to build a working version of the code.

Post reply on HN