We had set up a bunch of lxc containers on two servers each with 16 core CPUs and 64 GB RAM(for reliability and loadbalancing). Both the servers are on same vlan. The servers need to have atleast one of their network interface in promiscuous mode so that it forwards all packets on vlan to the bridge(http://blogs.eskratch.com/2012/10/create-your-own-vms-i.html) which takes care of the routing to containers. If the packets are not addressed to the containers, the bridge drops the packet.
Having this setup, we moved all our platform maintenance services to these containers. They are fault tolerant as we used two host machines where each host machine has a replica of the containers on the other. The probability to crash for both the servers at the same time due to some hardware/software failure is less. But to my surprise both the servers are crashing exactly the same time with a mean life time 20 days. We had to wake up late nights(early mornings) to fix stuffs that gone down
The detective work started with latin kernel dumps. They proved futile. Our Hardware engineers upgraded the BIOS. On the Software side, I applied patches such that no memory leak or any sort of exceptions occur due to the applications running. But they were of no use and the crash continued.
I came across this blog http://codeascraft.etsy.com/2012/03/30/kernel-debugging-101/ which gave some insight. There is some stuff called NAPI (New API) introduced in network interfaces. Usually when data reaches physical interface, interrupt is sent upwards the network stack. NAPI kicks off when huge requests hit the interface, now the upper layers have to poll periodically for packets instead of interrupts. Since both the boxes are in promiscuous mode, they will switch to NAPI mode together when the traffic on VLAN is high.
This NAPI needs some data structure to be initialized before kicking off or else there will be kernel panic. The interface driver takes the responsibility of creating the datastructure. The mod tg3 driver (3.119) is installed on our boxes which supposedly have this bug. So upgraded the tg3 driver to 3.122 which claims this bug is fixed.
Today its a month since the last crash had happened(>mean lifetime). But still sleeping with one eye open
Having this setup, we moved all our platform maintenance services to these containers. They are fault tolerant as we used two host machines where each host machine has a replica of the containers on the other. The probability to crash for both the servers at the same time due to some hardware/software failure is less. But to my surprise both the servers are crashing exactly the same time with a mean life time 20 days. We had to wake up late nights(early mornings) to fix stuffs that gone down
The detective work started with latin kernel dumps. They proved futile. Our Hardware engineers upgraded the BIOS. On the Software side, I applied patches such that no memory leak or any sort of exceptions occur due to the applications running. But they were of no use and the crash continued.
I came across this blog http://codeascraft.etsy.com/2012/03/30/kernel-debugging-101/ which gave some insight. There is some stuff called NAPI (New API) introduced in network interfaces. Usually when data reaches physical interface, interrupt is sent upwards the network stack. NAPI kicks off when huge requests hit the interface, now the upper layers have to poll periodically for packets instead of interrupts. Since both the boxes are in promiscuous mode, they will switch to NAPI mode together when the traffic on VLAN is high.
This NAPI needs some data structure to be initialized before kicking off or else there will be kernel panic. The interface driver takes the responsibility of creating the datastructure. The mod tg3 driver (3.119) is installed on our boxes which supposedly have this bug. So upgraded the tg3 driver to 3.122 which claims this bug is fixed.
Today its a month since the last crash had happened(>mean lifetime). But still sleeping with one eye open
Comments
Post a Comment