Everyone knows that sheep like to flock together, so it should be no surprise that here at the internet sheep we have occasional requirements to use a load balancer to ensure that individual sheep servers are not overwhelmed with requests. Our usual weapon of choice to date has been CentOS 5, but we have been migrating to CentOS 6 as a base platform for server infrastructure. A recent new requirement needed a load balancer, and this was implemented using a fresh CentOS 6 install. There have been a number of changes to the various HA packages, but nothing that caused too many problems, and so we quickly had the load balancer configured to our standard setup and ready to test.
Unfortunately, initial testing quickly threw up a problem, and a fairly major one, performance through the load balancer was atrocious, with pages crawling through, images rendering almost line by line, and general all-round slowness. A quick test of the web-site performance directly was nice and quick, and it seemed obvious that somehow the load balancer was interfering. Network connections and cables were checked, configurations on the load balancer and web servers were checked and re-checked, all to no avail. It was not quite late in the day, so it was time to sleep on the matter.
The next morning, with a fresh mind (and a fresh cup of coffee), it was to Google we went, and after a quick read of a number of articles from others who had similar problems (and a few dead-ends), we found someone blaming an incompatibility between the ipvsadm tools and the latest kernel. A quick check of our boot configuration listed two kernels: 2.6.32-220 and 2.6.32-71 – we changed the default option to the older kernel, rebooted the server, and performance was restored. Presumably this issue will be fixed upstream soon enough, but for others with the same issue, simply edit /etc/grub.conf and change the “default=0” line to load a previous kernel (“default=1” in our case).