Introduction to Alerta: Open Source Aggregated Alerts

stephen auto-remediation, DevOps

There are a number of platforms available these days to assist operations in terms of dealing with alerts. Namely Pagerduty, VictorOps and OpsGenie. These are unfortunately pay for tools/

These tools are known as monitoring aggregation

I was looking through the integrations of elastalert and found that there is an integration for alerta.io, so I checked the website and it seemed to check all the boxed of monitoring aggregation.

I used the docker compose way of setting it up quickly, but if you want to set it up proper then follow the alerta.io deployment guide.

Update some config:


docker exec -u root -it alerta_web_1 /bin/bash
apt update
apt install vim
# Edit the config in /app/alertad.conf
# Restart the container

Add the housekeeping cron job:


echo "* * * * * root /venv/bin/alerta housekeeping" >/etc/cron.daily/alerta

The default timeout period for an alert is 86400 seconds, or one day.

Check out the alerta plugins

What popular alerting and monitoring tools does alerta.io integrate with?

Allowing unverified HTTP Post’s by Elastalert

stephen Uncategorized

I am using Praeco as a frontend for the Elastalert API which relies on Elastalert.

The problem I faced was that sending unverified requests failed, as elastalert didn’t allow verfiy=False.

I searched the Elastalert code and found the place the request is being made, it is in alerts.py file, the class HTTPPostAlerter.

I changed:


response = requests.post(
               url,
               data=json.dumps(payload, cls=DateTimeEncoder),
               headers=headers,
               proxies=proxies,
               timeout=self.timeout
)

to:


response = requests.post(
               url,
               data=json.dumps(payload, cls=DateTimeEncoder),
               headers=headers,
               proxies=proxies,
               timeout=self.timeout,
               verify=False
)

You will still get an insecure request warning, but the request is sent.

If you are using the Praeco docker containers, then you can use this post to figure out which version of the code to change.

Remember to first go into the container:


docker exec -it praeco_elastalert_1 sh

Testing this in reality

Outside of test function, the http post does not work.

In the alert log on the praeco frontend the slack alert shows, but the http post does not.

In the application logs the following error is shown:

 


  ProcessController:  ERROR:root:Error while running alert http_post: Error posting HTTP Post alert: HTTPSConnectionPool(host='api.voxcloud.co.za', port=443): Max retries exceeded with url: /st2/api/v1/webhooks/praeco (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

I had to restart the containers, docker container stop and start for it to read the updated files.