13. Job Queue¶
eduMFA workflows often entail some time-consuming tasks, such as sending mails or SMS or saving usage statistics. Executing such tasks during the handling of API requests negatively affects performance. eduMFA allows to delegate certain tasks to external worker processes by using a job queue.
As an example, assume that eduMFA receives an authentication request by a user with an email token (see Email) via HTTP. eduMFA will send a one-time password via E-Mail. In order to do so, it communicates with a SMTP server. Normally, eduMFA handles all communication during the processing of the original authentication request, which increases the response time for the HTTP request, especially if the SMTP server is at a remote location.
A job queue can help to reduce the response time as follows. Instead of communicating with the SMTP server during request handling, eduMFA stores a so-called job in a job queue which says “Send an E-Mail to xyz@example.com with content ‘…’”. eduMFA does not wait for the E-Mail to be actually sent, but already sends an HTTP response. An external worker process then retrieves the job from the queue and actually sends the corresponding E-Mail.
Using a job queue may improve the performance of your eduMFA server in case of a flaky connection to the SMTP server. Authentication requests that send E-Mails are then handled faster (because the eduMFA server does not actually communicate with the SMTP server), which means that the corresponding web server worker thread can handle the next request faster.
eduMFA implements a job queue based on huey which uses a Redis server to store jobs. eduMFA allows to offload sending mails to the queue. Other jobs will be implemented in future versions.
13.1. Configuration¶
The job queue is disabled by default. In order to enable it, add the following configuration option to edumfa.cfg
:
EDUMFA_JOB_QUEUE_CLASS = 'edumfa.lib.queues.huey_queue.HueyQueue'
After a server restart, you will be able to instruct individual SMTP servers to send all mails via the job queue by checking a corresponding box in the SMTP server configuration (see SMTP server configuration). This means that you can have separate SMTP server configurations, some of which send mails via the job queue, some of which send mails during the request processing.
Note that you need to run a Redis server which is reachable for the eduMFA server. By default, huey assumes a locally running Redis server. You can use a configuration option to provide a different URL (see here for information on the URL format):
EDUMFA_JOB_QUEUE_URL = 'redis://somehost'
In addition to the eduMFA server, you will have to run a worker process which fetches jobs from the queue and executes them. You can start it as follows:
edumfa-queue-huey
By default, the worker process logs to edumfa-queue.log
in the current working directory. You can pass a different logfile by using the -l
option:
edumfa-queue-huey -l /var/log/queue.log
As the script is heavily based on the huey consumer script, you can find information about additional options in the huey documentation.
Note that a side-effect of the queue is that the eduMFA server will not throw or log errors if a mail could not be sent. Hence, it is important to monitor the queue log file for errors.