Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Django Styleguide

github.com

Post reply on HN
Print view View original

41–50 of 141 posts

Re: Django Styleguide

#41

Post by roflyear »

69Represente wrote: post #38
andybak wrote: post #5

> We use Celery for the following general cases: > > Communicating with 3rd party services (sending emails, notifications, etc.) > Offloading heavier computational tasks outside the HTTP cycle. > Periodic tasks (using Celery beat) Sigh. No mention of the trade-offs. There's simpler ways to do all these things. Celery is a big complex beast and it always pains me to see it as the default suggestion for simple tasks.

I have a bug with celery i can't solve. when I send an async job that get data from various APIs and write all in a DB, in the case of lot there is lot of data, the celery task finish properly to but my flask app becomes unresponsive. I have to restart flask to get back to a normal state. Anyone would know where I should check?

sounds like a resource issue, maybe you are opening application contexts and not handling them properly?

Re: Django Styleguide

#42

Post by roflyear »

emptysea wrote: post #34
nickjj wrote: post #10

Earlier quoted context omitted.

A long time ago I wrote a blog post about Celery use cases at https://nickjanetakis.com/blog/4-use-cases-for-when-to-use-c... . It applies to Django, Flask or any Python system using it. All of it still applies today. It covers a few use cases on the before vs after of using Celery and touches base on why I'd consider using Celery over other solutions such as async / await. The TL;DR is Celery brings a lot to the tab…

I think as a concept a task queue with some workers makes a lot of sense but having used Celery in production, it leaves a lot to be desired We’ve run into various bugs and weird performance gotchas (like the workers prefetch jobs which is terrible if they aren’t all the same size)

Agreed. I have had more luck writing my own "jobs" engine that does stuff that I need that celery doesn't have anyway (retries, some record of execution, rate limiting).

I'm sure if I really tried, I can get celery to be very reliable... but I never really got there.

Also for whatever reason I have NEVER been able to get celery to be reliable for its scheduling/cron stuff. It just starts to fail. I use this library for that, which I have never had problems with: https://schedule.readthedocs.io/en/stable/

Re: Django Styleguide

#43

Post by rowanseymour »

randlet wrote: post #37
olau wrote: post #17

I think there are too many concepts in this, but rather than being negative, here are some tips: Always keep your models slim. Don't stuff template related stuff in there. You need to look at those models often, so compact is a win. course_has_finished(course) is not much longer than course.has_finished(), and will allow you to expand the functionality as time goes on. Do precomputation if you need the information in…

"Also be wary of the formerly South, now built-in migrations stuff. It's built around a fragile model (perfect history representation), so has lots of foot guns." My experience has been opposite so I'd be interested in hearing your experiences if you are willing. I have been using the built in migrations since day 1 on a medium sized Django project with 350+ migrations and migration issues for our project have been e…

Same here. If I was starting a non-Python project tomorrow I'd consider using Django to manage the database schema - especially now that we can describe custom indexes and constraints in migrations. Our project has gone thru over 1000 migrations so far tho we squash them down to 50 or so about once a year.

Re: Django Styleguide

#44

Post by roflyear »

giancarlostoro wrote: post #14
notdarkyet wrote: post #7

Earlier quoted context omitted.

Do you have additional examples of those simpler ways? I totally understand how Celery can be a hammer and everything is a nail type situation.

I would just use serverless functions to achieve the same thing personally, if you're in the cloud already, chances are high you can trigger functions based on new records in a database or new file uploaded, or what have you. Then you don't need to import much outside of the serverless SDK which should typically be pretty minimal. That's how I did a timed function for a Django project we were hosting in Azure anyway.

Serverless functions are good for some stuff: clear this cache on this schedule, run this task when XYZ happens, etc..

But that is not the popular usecase for celery. Often you want "some code" (that you likely already have written) to be executed async. Sure, you can create a public interface for "some code" then write a record, that the serverless function is looking for, that then calls back to that interface (but is it a web interface???? then you have a problem where if the job takes too long to complete, what about http timeouts ... ) and now you're really creating a big circle for something that should be simple: execute some code outside of the request (send an email, hit an api, whatever).

Serverless functions really shouldn't contain much logic either, because it's too complicated to test.

Re: Django Styleguide

#45

Post by roflyear »

hdjjhhvvhga wrote: post #6
andybak wrote: post #5

> We use Celery for the following general cases: > > Communicating with 3rd party services (sending emails, notifications, etc.) > Offloading heavier computational tasks outside the HTTP cycle. > Periodic tasks (using Celery beat) Sigh. No mention of the trade-offs. There's simpler ways to do all these things. Celery is a big complex beast and it always pains me to see it as the default suggestion for simple tasks.

What would be your first choices for each of the above?

This is good for really simple scheduling of stuff: https://schedule.readthedocs.io/en/stable/

Re: Django Styleguide

#46

Post by switch007 »

olau wrote: post #17

I think there are too many concepts in this, but rather than being negative, here are some tips: Always keep your models slim. Don't stuff template related stuff in there. You need to look at those models often, so compact is a win. course_has_finished(course) is not much longer than course.has_finished(), and will allow you to expand the functionality as time goes on. Do precomputation if you need the information in…

Apps should be buried deep in the documentation in some “super advanced” section and not advocated much at all

Re: Django Styleguide

#47

Post by hbrn »

roflyear wrote: post #40
hbrn wrote: post #39

Ah, the typical "where to put business logic in Django". M in ActiveRecord MVC web frameworks is deeply misunderstood. M is not "data model" (it would be called DVC if that was the case). M is your domain model. It's the model of your business, model of the real world. It's the core of your application. Another thing that I never understood, why are functions called services? Is it a subconscious desire to go back to…

I never understood why this was so hard or why people complicate this so much. You have a segment of your application that "does stuff" - some mix of classes and functions. This stuff has its own API. Then you have your web views call that code through that API (which is probably just calling functions...). No, instead, it is that "does stuff" hast to be its own library, or god forbid, its own service that lives some…

Typically it goes like this:

1. You found one case where complexity is essential.

2. That one case is not consistent with the rest of your app, and you were taught that inconsistency is bad.

3. Since you can't remove complexity from that case, for the sake of consistency you add complexity to all other cases.

Class-based views is a typical example. You found a place where CBVs are useful. Now some parts of your app use functions, some use classes, that's inconsistent. Edit your style guide to enforce CBV everywhere. Now a simple healthcheck endpoint that returns "OK" has to be a class.

As some folks used to say, you can write Java in any language.

The right approach, of course, is to say "I'd rather have inconsistency than complexity". The challenge is that perception of complexity is subjective, but inconsistency is objective. So the right approach eventually loses, and every organization turns into a bureaucratic hell.

Re: Django Styleguide

#48

Post by miketery »

j4mie wrote: post #28

I've been heavily inspired by this styleguide over the years, but I still think it's a bit too complex. A few random thoughts: - I think "services" is too much of a loaded term. I prefer "actions", and I always use the function-based style. - I hate the naming of "APIs" in this document. They use the term "API" when they mean "endpoint" or "view". - "Reuse serializers as little as possible" is the single best piece o…

> "Reuse serializers as little as possible" is the single best piece of advice when using DRF. The inline InputSerializer thing is brilliant.

Can you expand on this? What is the InputSerializer as opposed to custom rest serializers?

Re: Django Styleguide

#49

Post by danjac »

VWWHFSfQ wrote: post #13
true_religion wrote: post #8

Earlier quoted context omitted.

My choice for periodic jobs is cronjobs.

Do you mean a cronjob that calls a Django manager command to do the work? Or invokes an API method? From my experience cronjobs have a lot of downsides as well. They're great for doing tasks local to the server the job is running on. Not so great for the kinds of tasks (periodic or transactional) that Celery/a real queue is designed for.

Crons are fine for running Django commands that are just talking to the same services as your web views (databases, cache, email services etc). It should be fine to let them run in their dedicated VM/containers, using Ansible etc to keep the crontab configurations in the repo.

Celerybeat has the advantage of better visibility e.g. you can configure them in the Django admin and check when they are running. If you are not using Celery though and your needs are simple, it's easier to just use plain crons.

Re: Django Styleguide

#50

Post by rowanseymour »

olau wrote: post #17

I think there are too many concepts in this, but rather than being negative, here are some tips: Always keep your models slim. Don't stuff template related stuff in there. You need to look at those models often, so compact is a win. course_has_finished(course) is not much longer than course.has_finished(), and will allow you to expand the functionality as time goes on. Do precomputation if you need the information in…

> You can make subdirectories without making apps, and thus avoid dependency hell.

It's hard to get 100% right and thus our projects always have a few lazy foreign key relationships and inline imports to avoid cyclical imports... but I think our code is easier to manage because we try to model the dependency relationships by having things in separate apps.

> Also be wary of the formerly South, now built-in migrations stuff

Our experience from South to 4.x has been that the models/migrations system has matured significantly and is probably now the main selling point for Django for us.

Post reply on HN