KickLogstash.sh: 7 Lines to Restart Logstash

It might well be my naive, beginner’s implementation of it, but it sure seems like Logstash is pretty easy to confuse. Elasticsearch temporarily unavailable? Hang. Some weird formatting sneaks in? Hang. Some as-yet-undiscovered circumstance that causes java to eat all your memory? Hang. And each time it hangs, Logstash becomes unresponsive, never recovering. So for the moment, I’ve given up, and I’m resorting to a dumb, sledgehammer-like solution:

#! /bin/sh
FILELENGTH=`stat --format=%s /var/log/logstash/logstash.err`

if [ $FILELENGTH != "0" ]
then
        killall -9 -u logstash
        service logstash restart
        logger Logstash Killed and Restarted Due To Non-Zero Error Log
fi

Again, I’ve only had about a week of actually sending a lot of input to Logstash, but so far, there’s always something in the error log when it hangs, and there’s rarely anything in there when things are going well. So this barely-a-script checks the size of the error log, and if it’s non-zero, it SIGKILLs every process belonging to the logstash user, and then tries to start Logstash. The restart part assumes you have a Logstash script in init.d, which I certainly hope will be included as part of a proper .deb distribution of Logstash someday soon. I run it every five minutes with cron:

*/5 * * * * /etc/logstash/kick_logstash.sh

Leave a comment

Your email address will not be published. Required fields are marked *