Hal is my Hubot chatbot .. He’s awesome !! He gets me beer !!
He also does things like restart app servers, deploy code, and show me pictures of grumpy cats .. He’s so cool, I’ve started making non-humans to talk to him .. “Greg, what do you mean ??” .. Well, let me show you ..
- I have a Nagios server
- It monitors (allthethings)
- When the “logged in users” alert is triggered, Nagios sends a message to my chat service using hipsaint
- “logged in users” is a monitor I have that alerts me when more than 3 users are logged into a server
- I see the alert and the server in question
- I SSH into the server
- I type who
- I then determine if I need to care
- If not, move on with my life
- If so, dig deeper
The thing is, I have more than
1,000 1,500 2,200 5,000 6,500 active monitors .. That means Nagios can and will send many, many messages to my chat service — depending on the day .. So how can I make my life easier ??
Here’s an easy one: ask Hal who’s on a server ..
My stack is HipChat -> Hubot -> Jenkins -> Ansible .. That means I can damn near do anything I want, all from my chat client ..
Remember what I said earlier — about making non-humans talk to Hal ?? What I did was create a Nagios event handler that sends a message to my chat service using HipChat CLI .. Therefore, I AM NOT asking Hal who’s on a server, it’s NAGIOS WHO IS doing it ..
It doesn’t stop there !! You can create scripted Splunk alerts as well .. Before you know it, you will be making (allthethings) talk to Hal ..
So one of our AEM nodes was freaking out the other day .. No, not the election results .. Some code was deployed to it that had runaway processes, thus gorging itself on CPU and memory .. EEEKK !! What to do ?? If you’ve been around AEM for awhile, you know how we love our raw thread dumps .. That being said, I really dislike the process on how to obtain them:
- Log into the node
- Get the Java PID
- Execute jstack and output to a log file
- Repeat every 10 seconds for at least one minute
- Compress the log file
- Share the compressed log file
- Typically via email
So how can I automate this ?? Easy !!
Create a Bash script that will generate the thread dumps for you ..
Create an Ansible playbook that will execute the script, compress the log file, and email it
Jenkins (thank you CloudBees !!) is my chosen automation platform, often triggering an Ansible playbook (thank you Michael !!) to automate some mundane task .. I also use HipChat and have implemented a Hubot (thank you GitHub !!) chatbot .. Wouldn’t it be nice to integrate it all ?? For example, in HipChat:
@hal restart tomcat in dev
- Which Hubot recognizes
- Who then builds a Jenkins job
- Which then executes some Ansible playbook
- Which then restarts Tomcat in the development environment
Actually, it’s not that difficult .. For this post, I won’t go into the details on how to setup Hubot (this might help) or how to restart the Tomcat service (this might help) .. What I am most interested in is how to:
- Create a reusable way to build any Jenkins job
- Since Hubot is a Node.js application, an event listener sounds logical
- Invoke this reusable way
- i.e. event trigger
Here is an example of a Hubot event listener that can build a Jenkins job with or without parameters ..
Here is an example of a Hubot command that can trigger the previously defined event listener ..
I hope this helps !!