AWX: A server error has occurred and Redis

Sometimes in the container based deployments of AWX, you get errors. As with any computer system with many moving parts in a distributed architecture.

AWX has 3 containers when you run it:

  • awx-web - a django applicaiton
  • awx-task - a task runner
  • awx-redis - the message queue / broker
  • postgresql - the long term database

I had an issue that when ever a project or template was updated there would be an error:

Error!

Failed updating job undefined with variables. POST returned: 500 A server error has occurred.

awx-an-error-has-occured

Diving a bit deeper and looking at the logs - awx-web lost connection to redis with this error:

awx send_packed_command (errno, errmsg)) redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.

So if you look at the logs of the redis container you see an error:

1:M 02 Mar 2021 08:09:04.122 # Background saving error
1:M 02 Mar 2021 08:09:10.049 * 100 changes in 300 seconds. Saving...
1:M 02 Mar 2021 08:09:10.055 * Background saving started by pid 536
536:C 02 Mar 2021 08:09:10.059 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Permission denied

This it does not have permissions to the data file.

What I did - as a quick fix was execute shell for the container and run:

redis-server /usr/local/etc/redis/redis.conf

Redis started up ok and the errors were gone on AWX.

Problem now is that the job output is empty.
Also this is not a long term fix

Longer Term Fix

The longer term fix for the awx issue of Redis not having permissions to data/dump.rdb

Was to add this to you inventory:

redis_security_context_enabled=False