Support non-local queue without redis

[djay]

p.a.async supports multiserver queues without requiring additional software like redis. with it's own pros and cons. It would be nice to have a solution should someone already be using p.a.async or don't want to install redis.

[datakurre]

Supporting p.a.async and persisting queue onto ZODB are two different things.

My main point in c.taskqueue is to re-use Zope2's existing event loop, worker thread management and all the magic in ZPublisher in executing tasks. That way I don't need to support that part and can still trust that tasks are executed in "known environment". I don't know, how to do that "re-using part" in p.a.async's or Celery's task execution code (besides how David executed view code in his gist).

Providing support for zc.queue, on the other hand, would be possible. Maybe even trivial. Probably zc.queues could be stored in a such place that it would be easy to mount them from a separate ZODB to avoid them bloating the main database.

[djay]

You are right. I've updated the title of the issue.

[gforcada]

Late to the party but that would be really nice to have, if time allows I will see if I can do something about it, not sure if I'm smart enough though :)

[datakurre]

RabbitMQ support has been long for due, but redis support already left the server module a bit dirty (asyncore socket map is a bit tricky) and I haven't been able to abatract it better (and select my new favorite amqp lib).

What did you have in your mind? Redis was chosen, because It was possible to do persistent queue with quarenteed delivery with it.

Gil Forcada Codinachs notifications@github.com kirjoitti 11.1.2016 kello 15.38:

Late to the party but that would be really nice to have, if time allows I will see if I can do something about it, not sure if I'm smart enough though :)

—
Reply to this email directly or view it on GitHub.

[gforcada]

First of all I want to have it running. I was mostly interested in the idea of having a Plone built-in solution so that it would be easier to integrate, just like zcatalog and solr, the first one is a basic but does its job perfectly for small scale and then you can plug in something enterprise if you wish.

So something like that would be really nice to have for async stuff as well (IMHO).

[datakurre]

@gforcada The "basic" version is the local volatile queue (it even uses Python's Queue class), which consumes only the other of the default two Zope threads. We are using it for non-critical and re-runnable stuff like updating RSS-feeds on the background.

Anything else would need some place to store the queue. If ZODB would have been a good place, we would all be happy with plone.app.async.

[gforcada]

@datakurre I guess you already tried it but I was suggested to use zc.queue, that's supposed to fill that gap, isn't it?

At any rate, the upside of having an in-plone solution would be that without much extra effort nor extra dependencies you get async jobs for free, and as usual, if you want to go enterprise or have bells and whistles then you can do the extra mileage and add redis/rabbitMQ/whatnot.

[datakurre]

@gforcada No, haven't tried it yet. I've just trusted others that zc.queue is not such a good choice. It would be very similar to LocalVolatileTaskQueue, but it would open a new ZODB connection to access the persisted queue (actually queue + consumer specific processing queue). Probably queues should be stored so that they could be easily mounted from a separate ZODB. Also, similarly to local queue, in the worst case (separate worker) the queue would be polled only once in 30 seconds (ZServer asyncore poll interval when there's no incoming socket data - could be set smaller with a monkey patch). (Redis is always instant, because of the special code in server to hook Redis connection into asyncore socket map.)