Live data from Hacker News

Scaling Rails and Postgres to millions of users at Microsoft

stepchange.work

91–96 of 96 posts

Re: Scaling Rails and Postgres to millions of users at Microsoft

#91

Earlier quoted context omitted.

Sorry on the slow reply. But yea, I can complain at length. - Model validations aren't run automatically. Need to call full_clean manually. - EXCEPT when you're in a form! Forms have their own clean, which IS run automatically because is_valid() is run. - This also happens to run the model's full_clean. - DRF has its own version of create which is separate and also does not run full_clean. - Validation errors in DRF'…

Thanks for your reply. I'm currently in a stage of falling out of love with Django and trying to get my thoughts together on why that is. I think Django seems confused on the issue of clean/validation. On the one hand, it could say the "model" is just a database table and any validation should live in the business logic of your application. This would be a standard way of architecting a system where the persistence l…

Yeah definitely understand the frustration. I've been there and while I don't think we've found _the_ solution, we've settled into a flow that we're generally happy with.

For us we separate validations in two. Business and Data validations, which are generally defined as:

- Business: The Invoice in Country X is needs to ensure Y and Z taxes are applied at Billing T+3 days otherwise throw an error.

- Data Validation: The company's currency must match the country it operates in.

Business validations and logic always go inside services where as data validations are on the model. Data validations apply to 100% of all inserts. Once there's an IF statement segmenting a group it becomes business validation.

I could see an argument as to why the above is bad because sometimes it's a qualitative decision. Once in a while the lines get blurry, a data validation becomes _slightly_ too complex and an arguement ensues as to whether it's data vs business logic.

Our team really adheres to services and not fat models, sorry DHH.

To me, it's all so controversial whatever you pick will work out just fine - just stick to it and don't get lazy about it.

Re: Scaling Rails and Postgres to millions of users at Microsoft

#92

Earlier quoted context omitted.

Thanks for your reply. I'm currently in a stage of falling out of love with Django and trying to get my thoughts together on why that is. I think Django seems confused on the issue of clean/validation. On the one hand, it could say the "model" is just a database table and any validation should live in the business logic of your application. This would be a standard way of architecting a system where the persistence l…

Yeah definitely understand the frustration. I've been there and while I don't think we've found _the_ solution, we've settled into a flow that we're generally happy with. For us we separate validations in two. Business and Data validations, which are generally defined as: - Business: The Invoice in Country X is needs to ensure Y and Z taxes are applied at Billing T+3 days otherwise throw an error. - Data Validation:…

Services are definitely better and a solid part of a domain-driven design. The trouble is with Django I think it's a bandaid on a fundamentally broken architecture. The models end up anaemic because they're trying to be two things at once. It's super common to see things like services directly mutating model attributes and set up relationships manually by creating foreign keys etc. All of that should be hidden far away from services.

The ultimate I think is Domain-Driven Design (or Clean Architecture). This gives you a true core domain model that isn't constrained by frameworks etc. It's as powerful as it can be in whatever language you use (which in the case of Python is very powerful indeed). Some people have tried to get it to work with Django but it fights against you. It's probably more up front work as you won't get things like Django admin, but unless you really, truly are doing CRUD, then admin shouldn't be considered a good thing (it's like doing updates directly on the database, undermining any semblance of business rules).

Re: Scaling Rails and Postgres to millions of users at Microsoft

#93
post #7

Postgres can be scaled vertically like Stackoverflow did. With cache on edge for popular reads if you absolutely must (but you most likely dont). No need to microservice or sync read replicas even (unless you are making a game). No load balancers. Just up the RAM and CPU up to TB levels for heavy real world apps (99% of you wont ever run into this issue) Seriously its so create scalable backend services with postgres…

Thank you for sharing this! I have been diving into it.

How do you manage transactions with PostgREST? Is there a way to do it inside it? Or does it need to be in a good old endpoint/microservice? I can’t find anything in their documentation about complex business logic beyond CRUD operations.

Re: Scaling Rails and Postgres to millions of users at Microsoft

#94
post #93
post #7

Postgres can be scaled vertically like Stackoverflow did. With cache on edge for popular reads if you absolutely must (but you most likely dont). No need to microservice or sync read replicas even (unless you are making a game). No load balancers. Just up the RAM and CPU up to TB levels for heavy real world apps (99% of you wont ever run into this issue) Seriously its so create scalable backend services with postgres…

Thank you for sharing this! I have been diving into it. How do you manage transactions with PostgREST? Is there a way to do it inside it? Or does it need to be in a good old endpoint/microservice? I can’t find anything in their documentation about complex business logic beyond CRUD operations.

Transactions are done using database functions https://docs.postgrest.org/en/v12/references/api/functions.h....

Re: Scaling Rails and Postgres to millions of users at Microsoft

#95
post #93

Earlier quoted context omitted.

Thank you for sharing this! I have been diving into it. How do you manage transactions with PostgREST? Is there a way to do it inside it? Or does it need to be in a good old endpoint/microservice? I can’t find anything in their documentation about complex business logic beyond CRUD operations.

Transactions are done using database functions https://docs.postgrest.org/en/v12/references/api/functions.h... .

Ah ok awesome thank you!

Re: Scaling Rails and Postgres to millions of users at Microsoft

#96
post #72
post #25

Earlier quoted context omitted.

Our backend at work does use a read replica purely for websockets. I always wondered if it was overkill, I’m not a backend developer, though.

not sure what you are building but i hope that was for a real time multiplayer game otherwise doesn't make sense to have bi-directional communication when you only need reads making read replicas function also as writes is needed for such cases but already when you have more than one place to write you run into edge cases and complexities in debugging

I think the reason is pushes are sent out regularly in batches by some cron system and rather than reading from the main database it reads from the replica before it pushes them out. I didn't really explain the context properly in my comment.
Post reply on HN