Categories
awx

AWX: A server error has occurred and Redis

Sometimes in the container based deployments of AWX, you get errors. As with any computer system with many moving parts in a distributed architecture.

AWX has 4 containers when you run it:

  • awx-web – a django applicaiton
  • awx-task – a task runner
  • awx-redis – the message queue / broker
  • postgresql – the persistent storage database

I had an issue that when ever a project or template was updated there would be an error:

Error!

Failed updating job undefined with variables. POST returned: 500 A server error has occurred.

awx-an-error-has-occured

Diving a bit deeper and looking at the logs – awx-web lost connection to redis with this error:

awx send_packed_command (errno, errmsg)) redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.

So if you look at the logs of the redis container you see an error:

1:M 02 Mar 2021 08:09:04.122 # Background saving error
1:M 02 Mar 2021 08:09:10.049 * 100 changes in 300 seconds. Saving...
1:M 02 Mar 2021 08:09:10.055 * Background saving started by pid 536
536:C 02 Mar 2021 08:09:10.059 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Permission denied

This it does not have permissions to the data file.

What I did – as a quick fix was execute shell for the container and run:

redis-server /usr/local/etc/redis/redis.conf

Redis started up ok and the errors were gone on AWX.

Problem now is that the job output is empty.
Also this is not a long term fix

Longer Term Fix

The longer term fix for the awx issue of Redis not having permissions to data/dump.rdb

Was to add this to you inventory:

redis_security_context_enabled=False