331–340 of 369 posts
Earlier quoted context omitted.
This is true, and I’ve worked on systems that use this, but it’s a lot more work than just a rename. I’d recommend that, if you have a Postgres database already, definitely use that instead. Your queues will be transactional and they will get backed up when the rest of your database does.
>> but it’s a lot more work than just a rename Such as?
I'm always surprised that when I see people talk about queues I never see anyone mention beanstalkd. I've been using it for basically everything for 10 years and it's solid as a rock, incredibly simple and requires basically no maintenance. It Just Works™
During my tenure as CTO at a fintech company I built a banking engine using postgres backed queue system using Elixir / Phoenix. It's still in use today. The company processed large volumes of transactions and we were able to do things in real-time in terms of payments. Our system reached a point where I realized that we can scale almost infinitely just using a 2 tier architecture (Elixir / Phoenix / Oban and Postgre…
Earlier quoted context omitted.
Python, Flask, SQLAlchemy, and Postgres all have great documentation individually, but if I am building an application at the intersection often a guide on exactly how to join them all up is much faster than using each individually and trying to figure out the interactions in four places. AWS white papers and engineering blogs tend to give me everything I need in one place, and I don't think there are any for apps bu…
SQLAlchemy is an extra abstraction blocking your path here. While you probably should still use an ORM for your regular relation queries, you are not gaining anything significant by trying to use SQLAlchemy for implementing a queue backend. You can write raw SQL with psycopg2 (which is already a dependency in your project thanks to SQLAlchemy), and wrap these raw queue management SQL in a nice little Python module wh…
I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article. Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with…
Earlier quoted context omitted.
> This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases. If you mess up the tables in one database it doesn't affect others, but if you lock up the server where it can't respond to queries, that affects every database running on that server. > Also, if you run one postgres, you won't have of an issue running another if…
> If you mess up the tables in one database it doesn't affect others, but if you lock up the server where it can't respond to queries, that affects every database running on that server. How is it different from: putting multiple queues on same redis, when one queue is locked up, others queue are affected? If that's a real risk, you can always put them into different instances. The solution is exactly the same for re…
Queues by nature tend to be for tasks that you can tolerate delaying a bit. If your queues can impact your "live"/online processing, that's worse than just impacting other queues.
Also something like redis tends to be a lot simpler and less prone to locking up than the monster that is postgresql.
Earlier quoted context omitted.
SQLAlchemy is an extra abstraction blocking your path here. While you probably should still use an ORM for your regular relation queries, you are not gaining anything significant by trying to use SQLAlchemy for implementing a queue backend. You can write raw SQL with psycopg2 (which is already a dependency in your project thanks to SQLAlchemy), and wrap these raw queue management SQL in a nice little Python module wh…
You can write raw SQL with SA, while keeping the other nice features it has.
Earlier quoted context omitted.
You can write raw SQL with SA, while keeping the other nice features it has.
Without being rude, what are the nice features? I've worked with it a bit and constantly found myself wishing it was just SQL whenever I've bumped into it
Look up the features of Core if interested. No ORM needed, as it says in the docs.