F5 Networks Load Balancing Both Active

Had an interesting day today working with a pair of load balancing F5 Big-IP boxes.
They both thought they should be active at the sametime, which bumped all my connected servers.

I made the mistake of thinking the reboot from the console on one would cleanly reboot. Evidently this does NOT reboot the device drivers, which in my case was a disaster since the machine had been up for 97 days. There is a support notice to that effect. What you need to do is upgrade the firmware or reboot the device with full_box_reboot which calls a script which:

ping -c1 -q sccp &> /dev/null && touch /.sccp_hard_reboot; reboot

Second mistake was trying to halt from a serial console and the unit went in to run level 0.
which required a physical press of the power button.

[admin@f5box01:Standby] sbin # who -r
run-level 3 Aug 2 10:14 last=S

The BIG-IP is an amazing piece of hardware that provides high-availability load balancing