Category: ansible

Ansible SSH Errors when connecting to local vagrant juniper vqfx10k box

When recently running ansible on a local virtual switch from juniper vqfx10k, I got the following error:

$ ansible local -m ping
[WARNING]: sftp transfer mechanism failed on [127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: scp transfer mechanism failed on [127.0.0.1]. Use ANSIBLE_DEBUG=1 to see detailed information
vqfx1 | FAILED! => {
    "changed": false,
    "module_stderr": "Connection to 127.0.0.1 closed.\r\n",
    "module_stdout": "\r\nerror: unknown command: /bin/sh\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 0
}

That error was not good enough to figure out what the hell was going on. So I ran the command in verbose mode with ANSIBLE_DEBUG=1:

$ ANSIBLE_DEBUG=1 ansible local -m ping -vvvv
...
 99919 1615908883.42088: _low_level_execute_command() done: rc=0, stdout=
error: unknown command: /bin/sh
, stderr=OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/stephen/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 98168
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0

So we have this annoying logout of the shared session.
Github had an issue and solution

Which I tried by setting in ansible.cfg:

[ssh_connection]
ssh_args =

It was successful in moving on to connecting via SSH, but now I got another issue:

debug3: Ignored env _
debug1: Sending command: /bin/sh -c '/usr/local/bin/python '"'"'error: unknown command: /bin/sh/AnsiballZ_ping.py'"'"' && sleep 0'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

debug3: send packet: type 1
debug1: fd 1 clearing O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Connection to 127.0.0.1 closed.
Transferred: sent 2888, received 2360 bytes, in 0.2 seconds
Bytes per second: sent 15168.1, received 12395.0
debug1: Exit status 0

Boiling down to:

So I found the issue on juniper and realised my issue:

I forgot to specify netconf as the connection:

vqfx1 ansible_host=127.0.0.1 ansible_port=2222 ansible_user='vagrant' ansible_ssh_private_key_file='/Users/stephen/.vagrant.d/insecure_private_key' ansible_network_os=junos ansible_connection=netconf

You can also remove the above ssh_args as now it is connecting with netconf which now does not need that connection story.

Using Keycloak as the Identity Provider for AWX

Good day, in this post I will show you how to use Keycloak (Open source Redhat SSO) as the identity provider for AWX.

I am basing this tutorial on the post on red hat SSO integration on ansible.com and from the AWX docs on SAML authentication

Prerequisites

You need the following set up:

  • AWX instance
  • Keycloak Instance
  • A realm with users or linked to another user source (identity brokering or user federation with LDAP)

Creating a Key Pair

Using OpenSSL or LibreSSL create a public-private key pair:

openssl req -new -x509 -days 365 -nodes -out saml.crt -keyout saml.key

On your keycloak realm -> keys -> providers -> Add (RSA)

Then upload your private key and cert you created.

Steps

  1. Log into AWX as an admin user
  2. Go to Settings -> Authentication
  3. At the top select the SAML button

    On this page the SAML Assertion Consumer Service (ACS) URL and SAML Service Provider Metadata URL are provided for you to enter on keycloak side

At this stage we can create the client for our realm

  1. Log into keycloak as realm admin
  2. Go to Clients -> Create client

    Set Client Protocol to SAML

    Go to /api/v2/settings/system on AWX to find your TOWER_URL_BASE. Add that as the Entity ID on Keycloak side.

    Example: https://dev-automation.fixes.co.za

    Then add the SAML Assertion Consumer Service (ACS) URL from AWX as the CLient SAML Endpoint on Keycloak.

keycloak-awx-client-saml-endpoint

Now we are done with the client creation we need to set the settings on keycloak side. Fill in the entity ID , SAML Service Provider Public Certificate and SAML Service Provider Private Key you created previously.

Now fill in any additional information:

Under SAML Service Provider Organization Info (This is just information about the identity provider):

{
  "en-US": {
    "url": "http://keycloak.fixes.co.za",
    "displayname": "Keycloak",
    "name": "keycloak"
  }
}

Under SAML Service Provider Technical Contact:

{
    "givenName": "Some User",
    "emailAddress": "suser@example.com"
}

Under SAML Service Provider Support Contact:

{
    "givenName": "Some User",
    "emailAddress": "suser@example.com"
}

Under SAML Enabled Identity Providers (Info on how to connect to the provider):

{
   "RHSSO": {
      "attr_last_name": "last_name",
      "attr_username": "username",
      "entity_id": "https://rhsso.usersys.redhat.com:8443/auth/realms/tower",
      "attr_user_permanent_id": "name_id",
      "url": "https://rhsso.usersys.redhat.com:8443/auth/realms/tower/protocol/saml",
      "attr_email": "email",
      "x509cert": "",
      "attr_first_name": "first_name",
      "attr_groups": "groups"
   }
}

Under SAML Organization Map:

{
   "Default": {
      "users": true
   },
   "Systems Engineering": {
      "admins": [
         "acheron@redhat.com",
         "jparrill@redhat.com",
         "covenant@redhat.com",
         "olympia@redhat.com
      ],
      "remove_admins": false,
      "remove_users": false,
      "users": true
   }
}

Add the Mappers on Keycloak

Errors

Keycloak Error: “We’re sorry, failed to process response”

Check your Keycloak log. If the log displays

failed: org.keycloak.common.VerificationException: Client does not have a public key

set Encrypt Assertions to OFF in your Keycloak client.

Logs in to Keycloak and Redirects - but does not Login to AWX

In the AWX logs you will find this line:

social Found an Attribute element with duplicated Name

You have to Your Realm -> Client Scopes (left side) -> role_list -> Mappers -> role list -> Set Single Role Attribute to On

Boom - it works!

Sources

AWX: ERROR! No inventory was parsed, please check your configuration and options.

Ensure that your playbook sets hosts: all as awx manages your host definition

As per AWX: Troubleshooting no inventory parsed

Well that didn't work still got the same issue.

I found 4 relevant issues on github on the ERROR! No inventory was parsed, please check your configuration and options. issue:

Although there is no clarity there.

I then found this red hat article (behind a login wall) that tells you that you need to set the inventory plugin as an environment variable:

Solution

It looks like this has to do with the custom venv.

I did not follow the documentation properly and teh venv needs a few base dependencies that i did not install - namely psutils and ANSIBLE - Einstein!!! What a tool I am.

Anyway actviate the virtualenv and run:

pip install psutil
pip install -U "ansible == 2.9.5"

Ensure it has the correct permissions

sudo chmod 0755 /opt/awx_venvs/

After doing that running my playbook I still got an error:

No such file or directory: b'/opt/awx_venvs/provisioning/bin/ansible-playbook'

So I looked at the awx custom venv docs.

I followed the steps but received the same error. The virtualenv setup on the host was not correctly setup within the container.

So I needed to look at the Container based custom venv documentation

Wait scratch that it looks too tough

Custom VirtualEnv as Vars

I looked at Updating venv in additional vars, according to ryanpetrollo on this github issue.

So I added my venv_vars.yaml:

And ran the isntall again but the venvs were not created

So I stopped and removed the containers:

sudo docker stop awx_task awx_web awx_memcached awx_redis awx_postgres

and removed them:

sudo docker rm awx_task awx_web awx_memcached awx_redis awx_postgres

then ran:

ansible-playbook -i inventory install.yml --extra-vars "@venv_vars.yaml"

Still the venv was not created.

Looking through the code you need to be deploying the kubernetes way version for this to work.

So I used kubernetes and it worked (it's also better this way as requirements are in code)

I had to add an extra variable to venv_vars.yml;

---

custom_venvs_path: "/opt/awx_venvs"

custom_venvs:
  - name: provisioning
    python: python3
    python_ansible_version: 2.9.5
    python_modules:
      - jmespath
      - netaddr
      - pyvcloud

Sources