The time between Christmas and New Years is the slowest time of the year. Large portion of my team is on vacation and the customer activity is reduced to emergencies and routine maintenance.   For me it is an opportunity to spend more time with family, relax and… “geek out”.

As CEO large portion of my days consists with times of running the company taking all kind of high level decision managing projects and people, while I believe to be good at what I do I need to make sure to keep strong hands on skills and this slower period provides a lot of good opportunities for limited distractions while doing some hands on stuff.

Some previous years I would play with some PHP programming skills or setting up MySQL Zoo with MySQL Sandbox, while this year I was for real treat.

Few months ago a friend of mine send me bunch of old servers he was decommissioning from the data center. I got bunch of Dell PowerEdge 2850 and some smaller similarly aged beasts. I have and old rack in my basement so I was setting them up as a server farm to later install Percona XtraDB Cluster and run some evil experiments like unplugging network cables which are just not as fun to do in the Cloud.

The old hardware is really great if your purpose it training troubleshooting skills as chances are much higher it might develop some weird problems and so you can twist your brain about how exactly you can diagnose and troubleshoot them.   You tend to forget how weird hardware can be if all you deal with is hardware managed by someone else in remote data center and the cloud.

Replacing the CPUs and testing memory is not considered high value skills. Typically college kids with little training can do them,  yet I think it is very valuable to go through these activities every so often to maintain good knowledge about details which can be very valuable troubleshooting complex problems.

For example over last couple of days I learned what database IO stalls could be pointing to bad memory in the RAID controller (even though it does not report any problem).   Another puzzle I’m still to resolve is why I have 2 servers installed absolutely same way but sysbench is showing very different IO performance numbers on their RAID controllers.