Sometimes in the container based deployments of AWX, you get errors. As with any computer system with many moving parts in a distributed architecture.
AWX has 3 containers when you run it:
awx-web– a django applicaiton
awx-task– a task runner
awx-redis– the message queue / broker
postgresql– the long term database
I had an issue that when ever a project or template was updated there would be an error:
Error! Failed updating job undefined with variables. POST returned: 500 A server error has occurred.
Diving a bit deeper and looking at the logs –
awx-web lost connection to redis with this error:
awx send_packed_command (errno, errmsg)) redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.
So if you look at the logs of the redis container you see an error:
1:M 02 Mar 2021 08:09:04.122 # Background saving error 1:M 02 Mar 2021 08:09:10.049 * 100 changes in 300 seconds. Saving... 1:M 02 Mar 2021 08:09:10.055 * Background saving started by pid 536 536:C 02 Mar 2021 08:09:10.059 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: Permission denied
This it does not have permissions to the data file.
What I did – as a quick fix was execute shell for the container and run:
Redis started up ok and the errors were gone on AWX.
Problem now is that the job output is empty.
Also this is not a long term fix
Longer Term Fix
The longer term fix for the awx issue of Redis not having permissions to
Was to add this to you inventory: