Installing Hubot on CentOS

Provision a Centos 7 server


yum update

You will see your node and npm versions are old:


[root@st2 ~]# npm -v
3.10.10
[root@st2 ~]# node -v
v6.16.0

Use nodesource


curl -sL https://rpm.nodesource.com/setup_10.x | bash -
sudo yum install gcc-c++ make
sudo yum install -y nodejs

The node and npm versions are now updated:


[root@st2 ~]# node -v
v10.15.3
[root@st2 ~]# npm -v
6.4.1

Install Hubot

Use the hubot instructions


npm install -g yo generator-hubot
mkdir myhubot
cd myhubot

Importantly you need to create a local non-root user, otherwise you get this error:


Error: EACCES: permission denied, open '/root/.config/insight-nodejs/insight-yo.json'

Become that user then run:


yo hubot

Set up Hubot to use the telegram adapter

hubot-yeoman-telegram

Now hubot needs redis, so lets install that.


sudo yum install epel-release
sudo yum update
sudo yum install redis
sudo systemctl enable redis
sudo systemctl start redis

Now lets try run it:

bin/hubot

Which didn’t really work, so I tried with telgram:

bin/hubot -a telegram

Now errors pop up:


[Wed May 22 2019 07:23:05 GMT+0000 (Coordinated Universal Time)] ERROR Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

That error seems to be because of the polling rate which should be lowered, 3500 was the magic number for me

TELEGRAM_TOKEN=XXX:YYY TELEGRAM_INTERVAL=3500 bin/hubot -a telegram

 

Your response will be slower.

It is best to keep your bot name short, I changed mine:

telegram-hubot-fix

Add a simple custom action and action alias to stackstorm

In this post I will demonstrate adding a ping action to stackstorm. Then make that action available from chatops (slack) using an action alias.

The Scenario

On a team of network engineers often certain ip addresses need to be checked if they are accessible. This is done using the ping command. If an engineers would like to give visibility on the status of that ping command, she could make that ping command via her chat application. Then everyone in the same room would be able to see the result.

Create the Ping Action

Lets check what this action does, ssh into your stackstorm instance and run:


[cent@st2 packs]$ ping -c 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=3.15 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=2.99 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=2.69 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=2.73 ms

--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 2.696/2.894/3.157/0.201 ms

this is what we want to do, but instead of manually typing the command we want stackstorm to do the action.

Let us use the core.local action to run the command:


[cent@st2 packs]$ st2 run core.local -- ping -c 4 1.1.1.1
..
id: 5cdeb8bb52364c6d5cb1d90f
status: succeeded
parameters: 
  cmd: ping -c 4 1.1.1.1
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: 'PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
    64 bytes from 1.1.1.1: icmp_seq=1 ttl=59 time=3.17 ms
    64 bytes from 1.1.1.1: icmp_seq=2 ttl=59 time=3.09 ms
    64 bytes from 1.1.1.1: icmp_seq=3 ttl=59 time=2.89 ms
    64 bytes from 1.1.1.1: icmp_seq=4 ttl=59 time=2.75 ms

    --- 1.1.1.1 ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3003ms
    rtt min/avg/max/mdev = 2.759/2.981/3.171/0.167 ms'
  succeeded: true

So now stackstorm has run the action.

Now create a custom pack folder in /opt/stackstorm/packs and create a folder within that called actions and then create a file called ping.yaml within that. That file should contain:


---
description: Action that executes the Linux ping command on the localhost.
runner_type: "local-shell-cmd"
enabled: true
entry_point: ''
name: ping
parameters:
  ip:
    description: The ip address to ping
    type: string
    required: true
  cmd:
    description: Arbitrary Linux command to be executed on the local host.
    required: true
    type: string
    default: 'ping -c4 {{ip}}'
    immutable: true
  kwarg_op:
    immutable: true
  sudo:
    default: false
    immutable: true
  sudo_password:
    immutable: true

 

We are running a local shell command, I’m not 100% on the other paramters and if they are even needed but the cmd command is and is defaulted to ping -c4 {{ ip}} where we intepolate ip.

Now we reload stackstorm to pickup the action: st2ctl reload

Then we run the action: st2 action run my_pack.ping ip=1.1.1.1

Sorted.

Create the Action Alias

Now we are going to create the alias so that the ping can be called from slack.

In /opt/stackstorm/packs/my_pack/aliases/ping.yaml add:

 


---
name: "ping"
pack: "my_pack"
action_ref: "my_pack.ping"
description: "Execute a local ping."
formats:
  - "ping {{ ip }}"

Now you need to reload stackstorm: sudo sysctl reload.
The action should now be available on slack (if you have set chatops up).

Next thing, is the alias will not show up in help if you have not restarted the chatops service, so let us do that now: sudo systemctl restart chatops

when you do !help your alias will now be there:

st2-ping-custom-alias

So let’s run it (remember you can also @botname to run the command): @mybot ping 8.8.8.8

stackstorm-ping-alias-with-slack

So that is a good demo.

Is telegram good for ChatOps and DevOps?

I’ve been using telegram for some chat ops related activities and started integrating them with things like hubot and StackStorm ChatOps to figure out if it works seemlessly. This will also end up being a Telegram vs Slack vs insert chat client here if I get time.

telegram-app-for-chatops

I’ve picked up certain things that make things difficult with telegram:

  • Showing html raw and not as a string
  • Showing graphs and charts
  • The formatting is not different when coming from a bot or a real person

Unicode and UTF-8

Importantly, the python-telegram-bot expects utf-8 encoded strings. Not the normal python3 str which is unicode (I think)…nothing is clear with this.

There is a webpage for telegram icons so I used python to try send this message and it is still not working as expected…encodings…


chop = b'\xF0\x9F\x9A\x80'.decode(encoding='utf-8')
utf8_str_icon = bytes('\xF0\x9F\x9A\x80', 'utf-8').decode()
unicode_decoded = 'U+1F680'
unicode_icon = '🚀'

So I think I will try use ChatOps and integrate with Telegram somehow

As I don’t want to be dealing with this stuff.

Errbot and Telegram

I’ve been using stackstorm and asked the community what options there were when integrating with telegram and I got a response from the err-stackstorm maintainer

Installing errbot:


cd ~
python3 -m venv errbot-env
source errbot-env/bin/activate
pip install errbot

Ok, now we need a bit extra for the telegram backend configuration:

pip install python-telegram-bot

Then a bit on configuration, in config.py:


BACKEND = 'Telegram'

BOT_IDENTITY = {
    'token': '8043xxxx:xxxxxxxxxxxxxxxxxxxxx4',
}

CHATROOM_PRESENCE = ()

BOT_PREFIX = '/'

Boom…everything working well. Run your bot, and send some commands to it via telegram. The only thing that is not great is the formatting that you can test with the command

/render test

It isn’t beautifully formatted:

errbot-telegram-render-test

Errbot and Slack

I was dissapointed with the formatting, so I decided to try it with slack.

Check out the slack configuration

Be careful though, the latest stackclient has major changes. So ensure to:

pip install slackclient==1.3.1

Running the same thing on slack was much better in terms of markdown formatting – bold, italics, tables, showing images, dispalying code and links etc.

slack-errbot-render-test-1slack-errbot-render-test-2

There are other clients I want to try out in the future rocket.chat, microsoft teams, Cisco Spark and mattermost but I’ll try them when they cross my path.

Connecting Stackstorm to Errbot

So now let us connect the errbot to stackstorm chatops.

I am trying the errbot-stackstorm plugin. So it needs to be configured and then enable errbots webhook support.

Which is done by sending this command via chatops:

/plugin config Webserver {'HOST': '0.0.0.0', 'PORT': 3141, 'SSL': {'enabled': False, 'host': '0.0.0.0', 'port': 3142, 'certificate': '', 'key': ''}}

By speaking with the maintainer, I managed to get it working but it wasn’t easy. Errbot also logged alot of stuff and seemed like it was constantly restarting. Furthermore, it had bugs. For example it would repeat the output from a command 100 times.

It also wasn’t stable, it would just error out:

errbot-stackstorm-errors

Just to show you it did work:

stackstorm-chatops-telegramUsing Hubot with Telegram

from the docs:

If you installed StackStorm following the install docs, the st2chatops package will take care of almost everything for you. Hubot with the necessary adapters is already installed

It seems that stackstorm prefers hubot.

The first step would be using the telegram hubot adapter. Since stackstorm hubot doesn’t support telegram by default, follow the docs on how to create an external adapter


cd /opt/stackstorm/chatops
sudo npm install --save hubot-telegram

Then modify the chatops configuration in /opt/stackstorm/chatops/st2chatops.env:


export HUBOT_ADAPTER=telegram
export HUBOT_TELEGRAM_TOKEN="xxx"
export HUBOT_TELEGRAM_WEBHOOK=""
export HUBOT_TELEGRAM_INTERVAL=5000

Reload the config and restart chatops:


sudo systemctl restart st2chatops
sudo st2ctl reload

Make sure in the output that chatops is running, not like this:


st2chatops is not running.

So check what is going on with:


sudo journalctl --unit=st2chatops

# The error
Error: The environment variable "TELEGRAM_TOKEN" is required.

So I added the environment variable, without the HUBOT prefix:

That worked but still had a weird error in the logs:


ERROR Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

This error is related to the interval that needs to be increased to 3500.

I still wasn’t getting a response on telegram so I did a check:


[cent@st2 ~]$ bash self-check.sh 

Starting the Hubot Self-Check Program
===============================================

Step 1: Hubot is running.
Step 2: Hubot-stackstorm is installed (0.9.3).
Step 3: StackStorm has aliases that are registered and enabled.
Step 4: Chatops.notify rule is present.
Step 5: Chatops.notify rule is enabled.
Step 6: Hubot responds to the "help" command.
Step 7: Hubot loads commands from StackStorm.
Step 8 failed: chatops.post_message doesn't work.

I manually did a post_message and that worked. I ran the script again and:


[cent@st2 ~]$ bash self-check.sh 

Starting the Hubot Self-Check Program
===============================================

Step 1: Hubot is running.
Step 2: Hubot-stackstorm is installed (0.9.3).
Step 3: StackStorm has aliases that are registered and enabled.
Step 4: Chatops.notify rule is present.
Step 5: Chatops.notify rule is enabled.
Step 6: Hubot responds to the "help" command.
Step 7: Hubot loads commands from StackStorm.
Step 8: chatops.post_message execution succeeded.
Step 9: The hubot adapter token is ok
Step 10: chatops.post_message has been received.
End to end test failed: Hubot not responding to "st2 list" command.

    Try reinstalling the st2chatops package. This error shouldn't
    happen unless the Hubot installation wasn't successful.
    It's also possible you changed the bot's name; this script
    assumes that "hubot" is something the bot will respond to.

I fixed this by doing:


cd /opt/stackstorm/chatops/
sudo npm install --save hubot-telegram
sudo systemctl restart st2chatops
st2ctl reload

that fixed the checker, but the commands were still not showing up.

So looked at the logs with journalctl -u st2chatops

and saw this error:

This is solved by the following issue on the hubot-stackstorm package. It has not been published to the npm package though.

There was another issue of this error


ERROR Error: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset

Which seems to be coming from the coffescript based telegram adapter

FFS!

It is discussed in these github issues: Can’t parse message text bad entities

This is a problem with the telegram api, so the message needs to be parsed beforehand.

Using Slack by Default

Getting a slack token

You then just add the api key to: /opt/stackstorm/chatops/st2chatops.env

and enable the slack adapter.

Then restart chatops:

sudo service st2chatops restart

I had an issue where the st2chatops service was inactive.


st2chatops is not running.

I installed with the ansible playbook so I had to remove st2chatops and then reinstall it with yum.

Then ran: journalctl --unit=st2chatops

You have to update the token in /opt/stackstorm/chatops/st2chatops.env

The bot should now be running (online on slack) and you can run:

 

slack-stackstorm-hubot-chatops

But none of the st2 commands are showing. The reason for that (damn, always issues) the API key is wrong.

So create a new key:

st2 apikey create -k -m '{"used_by": "dotty"}'

# StackStorm API key
export ST2_API_KEY=XXX

Restart the service and you will now see the st2 commands:

sudo service st2chatops restart

Everything should work now. Ie. the message should be sent to slack.

If you need to troubleshoot, check the chatops troubleshooting guide

Ensure the stream is accessible by doing: http https://123.123.123.123/stream/v1/stream --verify=False

And you get a response in quick time, if it 504‘s then you should restart the st2stream

Also ensure ST2_HOSTNAME is set correctly. Ensure to restart the service: sudo st2ctl reload --register-all and sudo service st2chatops restart