Switch to amqplib

[nathan-muir]

• Major version bump
• performance testing.
• Review channel management
  • Publish tasks on a single channel (?)
  • Consume results on a single channel (?)
• fail fast when connection buffer is full
• Support reconnects
• re-integrate support for onStarted
• Test failure modes
• Connection.close() doesn't wait for pending rpc calls to finish.
• Support RPC backend
• Support Task v2 format.

[nathan-muir]

Performance

• Seems reasonable
• Much faster without confirms
• Much faster publishing on a single channel (until throughput exceeds buffers)
• Consuming from a single queue/channel would be better

Considerations

• An error on a channel causes the whole channel to enter an "error" state [1]
• Celery doesn't use "confirm"'s by default [2][3], to enable, set broker_transport_options={"confirm_publish":True}
• Celery v3.1.25 / v4 introduces a new task message protocol[4][5]
• Celery v4 introduces the RPC results backend [6] which involves using a single results queue per process (instead of creating a queue per invocation)

[1] http://www.squaremobius.net/amqp.node/channel_api.html#dealing-with-failure
[2] http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
[3] https://tech.labs.oliverwyman.com/blog/2015/04/30/making-celery-play-nice-with-rabbitmq-and-bigwig/
[4] http://docs.celeryproject.org/en/latest/_modules/celery/app/amqp.html#AMQP.create_task_message
[5] http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#new-task-message-protocol
[6] http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#rpc-result-backend-matured

[nathan-muir]

Good reference for implementing AMQP