Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some notes on starting to use Django

jvns.ca

Post reply on HN
Print view View original

61–70 of 145 posts

Re: Some notes on starting to use Django

#61

Post by otherme123 »

sgt wrote: post #51
compounding_it wrote: post #43

Earlier quoted context omitted.

Significantly complex means when ORM starts to become bigger and bigger and you need multiple threads and more complex processes that run in workers. When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience. Then as a programmer, you have to find workarounds in Django instead of workarounds with programming. PS: Dealing with a lot of sca…

You can absolutely scale Django. The framework itself is not the limiting factor. The main constraint of performance usually comes from Python itself (really slow). And possibly I/O. There are well established ways to work around that. In practice, lots of heavy lifting happens in the DB, can you can offload workloads to separate processes as well (whether those are Python, Go, Rust, Java etc). You need to identify t…

> And the obvious for any Django dev; select_related, prefetch_related, annotate

And sometimes not so obvious, I have been bitten by forgetting one select_related while inadvertedly joining 5 tables but using only 4 select_related: the tests work OK, but the real data has a number of records that cause a N+1. A request that used to take 100ms now issues "30 seconds timeout" from time to time.

Once we added the missing select_related we went back to sub-second request, but it was very easy to start blaming Django itself because the number of records to join was getting high.

The cases that we usually walk out of the Django path is for serializations and representations, trying to avoid the creation of intermediate objects when we only need the "values()" return.

Re: Some notes on starting to use Django

#62

Post by jgavris »

ndr wrote: post #57
jgavris wrote: post #10

The Django ORM / migrations are still basically unmatched in happiness factor.

I found it very lacking in how to do CD with no downtime. It requires a particular dance if you ever want to add/delete a field and make sure both new-code and old-code work with both new-schema and old-schema. The workaround I found was to run tests with new-schema+old-code in CI when I have schema changes, and then `makemigrations` before deploying new-code. Are there better patterns beyond "oh you can just be care…

The general approach is to do multiple migrations (add first and make new-code work with both, deploy, remove old-code, then delete old-schema) and this is not specific to Django's ORM in any way, the same goes for any database schema deployment. Take a peek at https://medium.com/@pranavdixit20/zero-downtime-migrations-i... for some ideas.

Re: Some notes on starting to use Django

#63

Post by jgavris »

senko wrote: post #58
ndr wrote: post #57

Earlier quoted context omitted.

I found it very lacking in how to do CD with no downtime. It requires a particular dance if you ever want to add/delete a field and make sure both new-code and old-code work with both new-schema and old-schema. The workaround I found was to run tests with new-schema+old-code in CI when I have schema changes, and then `makemigrations` before deploying new-code. Are there better patterns beyond "oh you can just be care…

You can do three stage: 1. Make a schema migration that will work both with old and new code 2. Make a code change 3. Clean up schema migration Example: deleting a field: 1. Schema migration to make the column optional 2. Remove the field in the code 3. Schema migration to remove the column Yes, it's more complex than creating one schema migration, but that's the price you pay for zero-downtime. If you can relax that…

I was just in the middle of writing something similar above, thanks!

Re: Some notes on starting to use Django

#64

Post by rorylaitila »

ndr wrote: post #57
jgavris wrote: post #10

The Django ORM / migrations are still basically unmatched in happiness factor.

I found it very lacking in how to do CD with no downtime. It requires a particular dance if you ever want to add/delete a field and make sure both new-code and old-code work with both new-schema and old-schema. The workaround I found was to run tests with new-schema+old-code in CI when I have schema changes, and then `makemigrations` before deploying new-code. Are there better patterns beyond "oh you can just be care…

I simplify it this way. I don't delete fields or tables in migrations once an app is in production. Only manually clean them up after they are impossible to be used by any production version. I treat the database schema as-if it were "append only" - Only add new fields. This means you always "roll-forward", a database. Rollback migrations are 'not a thing' to me. I don't rename physical columns in production. If you need an old field and a new field to be running simultaneously that represent the same datum, a trigger keeps them in sync.

Re: Some notes on starting to use Django

#65

Post by aljarry »

ndr wrote: post #57
jgavris wrote: post #10

The Django ORM / migrations are still basically unmatched in happiness factor.

I found it very lacking in how to do CD with no downtime. It requires a particular dance if you ever want to add/delete a field and make sure both new-code and old-code work with both new-schema and old-schema. The workaround I found was to run tests with new-schema+old-code in CI when I have schema changes, and then `makemigrations` before deploying new-code. Are there better patterns beyond "oh you can just be care…

One option is to do multi-stage rollout of your database schema and code, over some time windows. I recall a blog post here (I think) lately from some Big Company (tm) that would run one step from the below plan every week:

1. Create new fields in the DB.

2. Make the code fill in the old fields and the new fields.

3. Make the code read from new fields.

4. Stop the code from filling old fields.

5. Remove the old fields.

Personally, I wouldn't use it until I really need it. But a simpler form is good: do the required schema changes (additive) iteratively, 1 iteration earlier than code changes. Do the destructive changes 1 iteration after your code stops using parts of the schema. There's opposite handling of things like "make non-nullable field nullable" and "make nullable field non-nullable", but that's part of the price of smooth operations.

Re: Some notes on starting to use Django

#66

Post by halfcat »

compounding_it wrote: post #43
sgt wrote: post #33

Earlier quoted context omitted.

What does significantly complex mean though? You have to make sure you understand the queries made by the ORM, avoid pitfalls like SELECT N+1 queries and so on. If you don't do this, it'll be slow but it's not the ORM's fault - it's that of the programmer.

Significantly complex means when ORM starts to become bigger and bigger and you need multiple threads and more complex processes that run in workers. When you start to run into scaling problems, your solution is within that framework and that becomes a limiting factor from my experience. Then as a programmer, you have to find workarounds in Django instead of workarounds with programming. PS: Dealing with a lot of sca…

> Then as a programmer, you have to find workarounds in Django instead of workarounds with programming.

The mental unlock here is: Django is only a convention, not strictly enforced. It’s just Python. You can change how it works.

See the Instagram playbook. They didn’t reach a point where Django stopped scaling and move away from Django. They started modifying Django because it’s pluggable.

As an example, if you’re dealing with complex background tasks, at some point you need something more architecturally robust, like a message bus feeding a pool of workers. One simple example could be, Django gets a request, you stick a message on Azure Service Bus (or AWS SQS, GCP PubSub, etc), and return HTTP 202 Accepted to the client with a URL they can poll for the result. Then you have a pool of workers in Azure Container Apps (or AWS/GCP thing that runs containers) that can scale to zero, and gets woken up when there’s a message on the service bus. Usually I’d implement the worker as a Django management command, so it can write back results to Django models.

Or if your background tasks have complex workflow dependencies then you need an orchestrator that can run DAGs (directed acyclic graph) like Airflow or Dagster or similar.

These are patterns you’d need to reach for regardless of tech stack, but Django makes it sane to do the plumbing.

The lesson from Instagram is that you don’t have to hit a wall and do a rewrite. You can just keep modifying Django until it’s almost unrecognizable as a Django project. Django just starts you with a good convention that (mostly) prevents you from doing things that you’ll regret later (except for untangling cross-app foreign keys, this part requires curse words and throwing things).

Re: Some notes on starting to use Django

#67

Post by m000 »

ndr wrote: post #57
jgavris wrote: post #10

The Django ORM / migrations are still basically unmatched in happiness factor.

I found it very lacking in how to do CD with no downtime. It requires a particular dance if you ever want to add/delete a field and make sure both new-code and old-code work with both new-schema and old-schema. The workaround I found was to run tests with new-schema+old-code in CI when I have schema changes, and then `makemigrations` before deploying new-code. Are there better patterns beyond "oh you can just be care…

Deploying on Kubernetes using Helm solves a lot of these cases: Migrations are run at the init stage of the pods. If successful, pods of the new version are started one by one, while the pods of the new version are shutdown. For a short period, you have pods of both versions running.

When you add new stuff or make benign modifications to the schema (e.g. add an index somewhere), you won't notice a thing.

If the introduced schema changes are not compatible with the old code, you may get a few ProgramingErrors raised from the old pods, before they are replaced. Which is usually acceptable.

There are still some changes that may require planning for downtime, or some other sort of special handling. E.g. upgrading a SmallIntegerField to an IntegerField in a frequently written table with millions of rows.

Re: Some notes on starting to use Django

#68

Post by senko »

pbreit wrote: post #26

I much prefer Python but am not really seeing any point to doing anything other than JavaScript for web projects at this point. I also do not see much reason to do more than emit JSON on the server side.

You still need clear separation between frontend and backend (react server components notwithstanding), so nothing's stopping you from using Python on the backend if you prefer it.

Django with DRF or django-ninja works really nice for that use case.

Re: Some notes on starting to use Django

#69

Post by brianwawok »

wiredfool wrote: post #56
cuu508 wrote: post #53

Earlier quoted context omitted.

Can you give an example how this would happen?

Ok, from memory -- There's a pre, do and post phase for the migrations. When you run a single migration, it's: pre, do, post. When you run 2 migrations, it's: pre [1,2], do: [1,2], post: [1,2]. So, if you have a migration that depends on a previous migration's post phase, then it will fail if it is run in a batch with the previous migration. When I've run into this is with data migrations, or if you're adding/assigin…

There’s like an atomic flag you can pull it out of the transaction . Solves a lot of these issues.

Re: Some notes on starting to use Django

#70

Post by Nextgrid »

tecoholic wrote: post #28
scott_w wrote: post #27

The author makes a great last point about Settings and it’s something I’ve not considered… ever! I wonder if there’s a feature request for this because having a pre-configured object would be nice for the ability to verify correctness on startup.

I use a project generator tool for a Django project. One of the things it does is generate setting file using string manipulation. I have been trying to think of a more sane way to do this. leverage something like dataclass or Pydantic models to have the typing information available and render a typed and validated Python object. If Django ever made that possible, it would be amazing for dev ex.

https://docs.pydantic.dev/latest/concepts/pydantic_settings/
Post reply on HN