A lightweight, high-performance, language-independent job queue system
1–10 of 41 posts
Re: A lightweight, high-performance, language-independent job queue system
#2Similar to the AppEngine taskqueue. Nice job! https://cloud.google.com/appengine/docs/standard/java/taskqu...
Re: A lightweight, high-performance, language-independent job queue system
#3When would a push queue like this be preferable over a pull queue?
Re: A lightweight, high-performance, language-independent job queue system
#4Honest question, how is using MySQL lightweight? Many job queues I've seen use e.g. Redis
Re: A lightweight, high-performance, language-independent job queue system
#5Similar to the AppEngine taskqueue. Nice job! https://cloud.google.com/appengine/docs/standard/java/taskqu...
FYI: Tasks are finally being decoupled from App Engine into their own standalone service (similar to Datastore before it).
Currently Alpha: https://cloud.google.com/sdk/gcloud/reference/alpha/tasks/
(I work for GCP)
Re: A lightweight, high-performance, language-independent job queue system
#6Similar to the AppEngine taskqueue. Nice job! https://cloud.google.com/appengine/docs/standard/java/taskqu...
FYI: Tasks are finally being decoupled from App Engine into their own standalone service (similar to Datastore before it). Currently Alpha: https://cloud.google.com/sdk/gcloud/reference/alpha/tasks/ (I work for GCP)
A bit off topic, but do you know if there has been any discussion to bring the search API to a standalone service?
https://cloud.google.com/appengine/docs/standard/java/search...
Re: A lightweight, high-performance, language-independent job queue system
#7Earlier quoted context omitted.
FYI: Tasks are finally being decoupled from App Engine into their own standalone service (similar to Datastore before it). Currently Alpha: https://cloud.google.com/sdk/gcloud/reference/alpha/tasks/ (I work for GCP)
A bit off topic, but do you know if there has been any discussion to bring the search API to a standalone service? https://cloud.google.com/appengine/docs/standard/java/search...
Not sure, and I can't talk about things that aren't public anyway :)
GCP and Elastic did partner to offer hosted Elasticsearch though it's not a fully managed service: https://www.elastic.co/about/partners/google-cloud-platform
Re: A lightweight, high-performance, language-independent job queue system
#8> It is built on top of RDBMS (MySQL),
Can I hear more about the rationale behind this?
Re: A lightweight, high-performance, language-independent job queue system
#9> It is built on top of RDBMS (MySQL), Can I hear more about the rationale behind this?
I'm not involved in the project, but you do this if you want the same robustness as an rdbms, e.g. if a client gets an acknowledgement back from the queue then the data is definitely captured and won't be lost if the queue process then crashes.
Re: A lightweight, high-performance, language-independent job queue system
#10Honest question, how is using MySQL lightweight? Many job queues I've seen use e.g. Redis
If the requirement is for persistence, i.e. no data loss if the queue process dies, then Redis won't fit.
EDIT: TIL Redis has the option to turn on fsync-to-disk on every write. Probably not what people are thinking of when they suggest Redis as lightweight.