Categories
ChatOps DevOps

Using Telegram for ChatOps

I’ve been using telegram for some chat ops related activities and started integrating them with things like hubot and StackStorm ChatOps to test it out.

telegram-app-for-chatops

I’ve picked up certain things that make things difficult with telegram:

  • Showing html raw and not as a string
  • Showing graphs and charts
  • The formatting is not different when coming from a bot or a real person

Unicode and UTF-8

The python-telegram-bot expects utf-8 encoded strings. Not the normal python3 str which is unicode.

Use mystr.encode('utf-8) to encode the unicode string into a bytestring (bytes object) with utf-8 encoding.

There is a webpage for telegram icons so I used python to try send this message and it is still not working as expected…encodings…


chop = b'\xF0\x9F\x9A\x80'.decode(encoding='utf-8')
utf8_str_icon = bytes('\xF0\x9F\x9A\x80', 'utf-8').decode()
unicode_decoded = 'U+1F680'
unicode_icon = '?'

So I think I will try use ChatOps and integrate with Telegram somehow…

As I don’t want to be dealing with this stuff.

Errbot and Telegram

I’ve been using stackstorm and asked the community what options there were when integrating with telegram and I got a response from the err-stackstorm maintainer

Installing errbot:


cd ~
python3 -m venv errbot-env
source errbot-env/bin/activate
pip install errbot

Ok, now we need a bit extra for the telegram backend configuration:

pip install python-telegram-bot

Then a bit on configuration, in config.py:


BACKEND = 'Telegram'

BOT_IDENTITY = {
    'token': '8043xxxx:xxxxxxxxxxxxxxxxxxxxx4',
}

CHATROOM_PRESENCE = ()

BOT_PREFIX = '/'

Boom…everything working well. Run your bot, and send some commands to it via telegram. The only thing that is not great is the formatting that you can test with the command

/render test

It isn’t beautifully formatted:

errbot-telegram-render-test

Errbot and Slack

I was dissapointed with the formatting, so I decided to try it with slack.

Check out the slack configuration

Be careful though, the latest stackclient has major changes. So ensure to:

pip install slackclient==1.3.1

Running the same thing on slack was much better in terms of markdown formatting – bold, italics, tables, showing images, dispalying code and links etc.

slack-errbot-render-test-1slack-errbot-render-test-2

There are other clients I want to try out in the future rocket.chat, microsoft teams, Cisco Spark and mattermost but I’ll try them when they cross my path.

Connecting Stackstorm to Errbot

So now let us connect the errbot to stackstorm chatops.

I am trying the errbot-stackstorm plugin. So it needs to be configured and then enable errbots webhook support.

Which is done by sending this command via chatops:

/plugin config Webserver {'HOST': '0.0.0.0', 'PORT': 3141, 'SSL': {'enabled': False, 'host': '0.0.0.0', 'port': 3142, 'certificate': '', 'key': ''}}

By speaking with the maintainer, I managed to get it working but it wasn’t easy. Errbot also logged alot of stuff and seemed like it was constantly restarting. Furthermore, it had bugs. For example it would repeat the output from a command 100 times.

It also wasn’t stable, it would just error out:

errbot-stackstorm-errors

Just to show you it did work:

stackstorm-chatops-telegramUsing Hubot with Telegram

from the docs:

If you installed StackStorm following the install docs, the st2chatops package will take care of almost everything for you. Hubot with the necessary adapters is already installed

It seems that stackstorm prefers hubot.

The first step would be using the telegram hubot adapter. Since stackstorm hubot doesn’t support telegram by default, follow the docs on how to create an external adapter


cd /opt/stackstorm/chatops
sudo npm install --save hubot-telegram

Then modify the chatops configuration in /opt/stackstorm/chatops/st2chatops.env:


export HUBOT_ADAPTER=telegram
export HUBOT_TELEGRAM_TOKEN="xxx"
export HUBOT_TELEGRAM_WEBHOOK=""
export HUBOT_TELEGRAM_INTERVAL=5000

Reload the config and restart chatops:


sudo systemctl restart st2chatops
sudo st2ctl reload

Make sure in the output that chatops is running, not like this:


st2chatops is not running.

So check what is going on with:


sudo journalctl --unit=st2chatops

# The error
Error: The environment variable "TELEGRAM_TOKEN" is required.

So I added the environment variable, without the HUBOT prefix:

That worked but still had a weird error in the logs:


ERROR Error: Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

This error is related to the interval that needs to be increased to 3500.

I still wasn’t getting a response on telegram so I did a check:


[cent@st2 ~]$ bash self-check.sh 

Starting the Hubot Self-Check Program
===============================================

Step 1: Hubot is running.
Step 2: Hubot-stackstorm is installed (0.9.3).
Step 3: StackStorm has aliases that are registered and enabled.
Step 4: Chatops.notify rule is present.
Step 5: Chatops.notify rule is enabled.
Step 6: Hubot responds to the "help" command.
Step 7: Hubot loads commands from StackStorm.
Step 8 failed: chatops.post_message doesn't work.

I manually did a post_message and that worked. I ran the script again and:


[cent@st2 ~]$ bash self-check.sh 

Starting the Hubot Self-Check Program
===============================================

Step 1: Hubot is running.
Step 2: Hubot-stackstorm is installed (0.9.3).
Step 3: StackStorm has aliases that are registered and enabled.
Step 4: Chatops.notify rule is present.
Step 5: Chatops.notify rule is enabled.
Step 6: Hubot responds to the "help" command.
Step 7: Hubot loads commands from StackStorm.
Step 8: chatops.post_message execution succeeded.
Step 9: The hubot adapter token is ok
Step 10: chatops.post_message has been received.
End to end test failed: Hubot not responding to "st2 list" command.

    Try reinstalling the st2chatops package. This error shouldn't
    happen unless the Hubot installation wasn't successful.
    It's also possible you changed the bot's name; this script
    assumes that "hubot" is something the bot will respond to.

I fixed this by doing:


cd /opt/stackstorm/chatops/
sudo npm install --save hubot-telegram
sudo systemctl restart st2chatops
st2ctl reload

that fixed the checker, but the commands were still not showing up.

So looked at the logs with journalctl -u st2chatops

and saw this error:

This is solved by the following issue on the hubot-stackstorm package. It has not been published to the npm package though.

There was another issue of this error


ERROR Error: Bad Request: can't parse entities: Can't find end of the entity starting at byte offset

Which seems to be coming from the coffescript based telegram adapter

FFS!

It is discussed in these github issues: Can’t parse message text bad entities

This is a problem with the telegram api, so the message needs to be parsed beforehand.

Using Slack by Default

Getting a slack token

You then just add the api key to: /opt/stackstorm/chatops/st2chatops.env

and enable the slack adapter.

Then restart chatops:

sudo service st2chatops restart

I had an issue where the st2chatops service was inactive.


st2chatops is not running.

I installed with the ansible playbook so I had to remove st2chatops and then reinstall it with yum.

Then ran: journalctl --unit=st2chatops

You have to update the token in /opt/stackstorm/chatops/st2chatops.env

The bot should now be running (online on slack) and you can run:

 

slack-stackstorm-hubot-chatops

But none of the st2 commands are showing. The reason for that (damn, always issues) the API key is wrong.

So create a new key:

st2 apikey create -k -m '{"used_by": "dotty"}'

# StackStorm API key
export ST2_API_KEY=XXX

Restart the service and you will now see the st2 commands:

sudo service st2chatops restart

Everything should work now. Ie. the message should be sent to slack.

If you need to troubleshoot, check the chatops troubleshooting guide

Ensure the stream is accessible by doing: http https://123.123.123.123/stream/v1/stream --verify=False

And you get a response in quick time, if it 504‘s then you should restart the st2stream

Also ensure ST2_HOSTNAME is set correctly. Ensure to restart the service: sudo st2ctl reload --register-all and sudo service st2chatops restart

In your channel do: @dotty help

Advanced Troubleshooting

If you are getting a 502 or 308 when accessing the api, you need to change the nginx config.

If you are getting a 401 or 403, then you need to create a new user and password and token for chatops and add that to st2chatops.env. Also ensure to change your hostname to point to your actual host – espescially when you have setup server blocks / virtualhosts.