Autmated distro testing for CentOS - Step 1
Everytime a new CentOS Release is built, it takes about 48 hrs before people can start testing these releases. What I would like to be able to do is run some install tests within the buildsystem / QA infrastructure so that we can catch major issues earlier and fix them faster. But I need some help from everyone out there in order to achieve this. Central to this idea is the need for kickstarts and test scripts that reflect usage out in the users systems and scenarios. Lets consider kickstarts first.
The idea here is that we have a couple of machines, close to the buildsystem, that can take a centos release, and run through installs in a loop, reporting success or failure after each install. Even if we only start with one machine for each primary arch, i386 and x86_64, we should be able to cycle through over 120 install cycles per arch within 24 hours with the right kind of automation. The automation is what I would like to focus on, I need your help with the kickstarts.
If you are able to write up some kickstarts, or even just go through an install and adapt the resulting anaconda-ks.cfg file - send them to me. Send me kickstarts for just about any role or deployment strategy you can think of - to get the ideas flowing I've made a short list of thing that would be good to have kickstarts for; dont, in any way feel restricted to these roles:
My initial wish list is something along these lines:
- General install, with no changes in the installer, just filling out whatever is the minimum required input
- A minimal install that only needs cd-1
- A LAMP server
- A Postgresql server
- GNOME Desktop
- KDE Desktop
- Xen Dom0 host
- KVM host
- DHCP and Static networked server installs
- Install from CDrom
- Install over http
- Install over NFS
- Install from local disk
- Office SME server, with samba and email
Here's how you could contribute the kickstarts:
- Post it online and leave a comment here with a url pointing to it
- Post it to the centos-devel list ( http://lists.centos.org )
- Drop into #centos-devel on irc.freenode.net and point us at it there
The kickstarts that we collect will be published online ( unless you specifically ask me not to ) along with results from each install cycle and details that came from it.
I know its not very organised at this time. But along with the automation I will build a web interface that lists the kickstarts along with a git repo that lets people directly pull from and push patches into.
If you need some help getting started with kickstarts, take a look at the wiki article at : http://wiki.centos.org/TipsAndTricks/KickStart and the docs at : http://www.centos.org/docs/5/html/5.2/Installation_Guide/s1-kickstart2-file.html . I would also highly recommend looking at the /root/anaconda-ks.cfg file that is left behind on any new centos install. Then there is always google.
Also, what about using qemu/virtualbox/kvm for simulating other archs/storage configurations?
Go ahead and send me some kickstarts which do these things and I will make sure they get added into the test pool.
the QA team does all the testing, there are some - very few - automated tests but the bulk of the testing is done by people.
What I would like to do is ease the pressure off the qa team and move to using more and more automation. Not to replace what people are already doing, but to bring in more test cases and extend the functionality trust as it cascades down from upstream down through our build services to the users.
the Automation is essentially a Continuous integration tool that has a set of 'sanity' tests which are run on each machine / vm instance that as its installed. The tests really depend on what the kickstarts contain. Ultimately being able to have a process that does both role and functional tests would be ideal but at the moment, its only a case of running through a set of sanity tests to make sure that system does - at a very basic level - what the user's default expectation would be.