Category: Bash

Qik-n-EZ: Nagios Plugins to Check AWS EC2 Images and Snapshots

Afraid of having too many AWS EC2 images and/or snapshots, thus running up your bill ?? Fear not !! I have you covered:

Nagios Plugins to Check AWS EC2 Images

Nagios Plugin to Check AWS EC2 Snapshots

Advertisements

Qik-n-EZ: Collect AEM Thread Dumps and Email via Ansible

So one of our AEM nodes was freaking out the other day .. No, not the election results .. Some code was deployed to it that had runaway processes, thus gorging itself on CPU and memory .. EEEKK !! What to do ?? If you’ve been around AEM for awhile, you know how we love our raw thread dumps .. That being said, I really dislike the process on how to obtain them:

  1. Log into the node
  2. Get the Java PID
  3. Execute jstack and output to a log file
    1. Repeat every 10 seconds for at least one minute
  4. Compress the log file
  5. Share the compressed log file
    1. Typically via email

So how can I automate this ?? Easy !!

Create a Bash script that will generate the thread dumps for you ..

Create an Ansible playbook that will execute the script, compress the log file, and email it

 

Scripting AWS EBS Volume Snapshots

So then, let’s talk about the use case ..

  • I have a Linux node
  • On said node, I have attached and mounted some EBS volumes
    • Think Jenkins (/var/lib/jenkins), BitBucket (/opt/bitbucket), OSSEC (/var/ossec), etc ..
  • I need to be able to create EBS snapshots of said EBS volumes on a regular basis, without majorly disrupting the service
    • I don’t want to have to stop, snapshot, and start the related service
  • I don’t always know what the EBS volume ID is, nor do I want to know
  • Sometimes I want to wait until the EBS snapshot is done before I continue doing other things
  • I want to tag the EBS snapshot something meaningful
    • jenkins.example.net_var_lib_jenkins

So how am I going to solve this ??

  1. Find a way to snapshot an EBS volume
    1. Hello CREATE-SNAPSHOT
  2. Find a way to determine the state of an EBS snapshot
    1. Hello DESCRIBE-SNAPSHOTS
    2. This is so I can wait until the EBS snapshot is done
  3. Find a way to “pause” the service without major disruption
    1. Hello FSFREEZE
  4. Find a way to map a device to an EBS volume
    1. Hello DESCRIBE-INSTANCES
    2. It’s a little more complicated than that, but this is where the data is
  5. Find a way to tag an EBS snapshot
    1. Hello CREATE-TAGS

Putting it all together ..

  • I created a Bash script that is managed by Ansible and utilizes the AWS CLI and JQ
  • It should be ran ON THE NODE where the EBS volume is attached and mounted
    • /root/bin/snapshot_ebs_volume.sh makes sense to me
  • It allows you to pass as options:
    • -d <device path> — the device path of the EBS volume you will freeze, EBS snapshot, and unfreeze
    • -c “<EBS snapshot description>” — is what it says
    • -w [optional]  — if defined, script will not finish until EBS snapshot is done
  • Sequence of events is as follows:
    1. Get the EBS Volume ID of the provided device path
    2. Get the mounted file system path of the provided device path
    3. Freeze the mounted file system path
    4. Create an EBS snapshot of the discovered EBS Volume ID
    5. Unfreeze the mounted file system path
    6. Tag the EBS snapshot with a pretty name
    7. Wait until done [optional], otherwise takes about 5 seconds total

script output

script output