December 2010 – Jesse Bowes

This is a followup to this post. It seems there’s a more graceful option to reboot the server when there are problems.

These instructions are for debian

1. Add the following lines to /etc/sysctl.conf

  kernel.panic = 60
  kernel.panic_on_oops = 60

1 2	kernel.panic = 60 kernel.panic_on_oops = 60

2. server:~# apt-get install watchdog

3. In /etc/watchdog.conf, uncomment the following:

  file = /var/log/messages
  watchdog-device = /dev/watchdog

1 2	file = /var/log/messages watchdog-device = /dev/watchdog

[ You may want to play with some of the other settings, most
notibly ping. I didnt have luck with it, however. My server
constantly rebooted ]

4. server:~# /sbin/modprobe softdog

5. add the following to /etc/modules

  softdog

softdog

6. Reboot

[UPDATED INFORMATION]

Here’s a bash script I’m using on my server to restart the server in the event network goes down. I set this up because, during a large file transfer through Samba, the server’s network stopped responding. To be on the safe side, I’m doing a full restart.

The scripts attempt to ping HOST (192.168.1.1, my router), MAX_FAILURES (3) times. After the 3rd failure, the server will restart. If the network comes back before the restart, we’ll proceed as if nothing happened. I’m waiting 5 minutes between each attempt.

I’m running this through an hourly cronjob.

#!/bin/bash
####################################################################
# Shell script for monitoring ping to a certain host, and if the   #
# ping fails max_failures consecutive times, restart the machine.  #
#                                                                  #
# The main driver behind this script is that my server is located  #
# remotely and if something happens to kill the network adapter    #
# but not the whole system, I'd like the system to reboot to the   #
# nominal state so I can regain control.                           #
####################################################################
# Recommend calling this in a cronjob at a frequency less than     #
# MAX_FAILURES * SLEEP_TIME.                                       #
####################################################################
 
### EDIT THESE ###
HOST="192.168.1.1"
MAX_FAILURES=3
SLEEP_TIME=300  #300 seconds = 5 minutes
###  END EDIT  ###
 
# Put a timestamp in front of every echo
function echo {
  /bin/echo netfaildetect: `date`: $*
}
 
cur_failures=0
 
while [ $cur_failures -lt $MAX_FAILURES ]; do
    ping -c 1 $HOST > /dev/null
 
    #Try pinging, and if host is up, we bail
    if [ $? -eq 0 ]; then
        cur_failures=0
        break
    fi
    let cur_failures=cur_failures+1
    echo "Network failure to $HOST detected.  Attempt:$cur_failures/$MAX_FAILURES."
 
    if [ $cur_failures -lt $MAX_FAILURES ]; then
        sleep $SLEEP_TIME
    fi
done
 
if [ $cur_failures -ge $MAX_FAILURES ]; then
    echo "Too many failures occurred, restarting"
    reboot
fi
 
echo "Network is nominal"

#!/bin/bash

####################################################################

# Shell script for monitoring ping to a certain host, and if the #

# ping fails max_failures consecutive times, restart the machine. #

# #

# The main driver behind this script is that my server is located #

# remotely and if something happens to kill the network adapter #

# but not the whole system, I'd like the system to reboot to the #

# nominal state so I can regain control. #

####################################################################

# Recommend calling this in a cronjob at a frequency less than #

# MAX_FAILURES * SLEEP_TIME. #

####################################################################

### EDIT THESE ###

HOST="192.168.1.1"

MAX_FAILURES=3

SLEEP_TIME=300 #300 seconds = 5 minutes

### END EDIT ###

# Put a timestamp in front of every echo

function echo {

/bin/echo netfaildetect: `date`: $*

}

cur_failures=0

while [ $cur_failures -lt $MAX_FAILURES ]; do

ping -c 1 $HOST > /dev/null

#Try pinging, and if host is up, we bail

if [ $? -eq 0 ]; then

cur_failures=0

break

let cur_failures=cur_failures+1

echo "Network failure to $HOST detected. Attempt:$cur_failures/$MAX_FAILURES."

if [ $cur_failures -lt $MAX_FAILURES ]; then

sleep $SLEEP_TIME

done

if [ $cur_failures -ge $MAX_FAILURES ]; then

echo "Too many failures occurred, restarting"

reboot

echo "Network is nominal"

Month: December 2010

Server Watchdog

Bash Restart on Network Failure