What was the issue:

There was an invalid rpm ( eclipse-ecj ) in the x86_64/os tree. Premature EOF in rpm payload.

Report:

Early morning on the 7th James Hogarth, BSkyB Entertainment reported to the QA team that there was an issue with the eclipse-ecj package shipped in the CentOS-5.6/x86_64 os tree. Stephen Walsh and Manuel Wolfshant tracked the issue back to the primary seed machine and confirmed the issue was present in not only the os tree, but also on CD3 and DVD1 of the x86_64 distribution.

Reason for this issue:

The CentOS distribution is composed on the buildservices, but then transferred over to the distro-build machines where the installer is built. There are automated tests that run at both of these locations. The rpm content tests ( rpm -K ) for md5's as well as gpg key. The distro had passed this test at both locations. The output from the distro-build machines is the actual package tree, the installer code, isos for cd's and dvd. This is transferred using rsync to the staging machine from where we start the release process ( initially to QAMachines when in qa mode; or into the mirror.centos.org network when in release mode ). There are no tests done at the staging machine. Packages are transferred one at a time, rather than as a whole tree ( mostly for legacy reasons ). It seems that the transfer for eclipse-ecj did not complete ( driven by the fact that its OK on one side and not on the other ).

Impact:

We had to rebuild the torrents, DVD isos, CD isos and update the CentOS-5.6/x86_64 distribution. There was no impact to CentOS-5.6/i386.

What did we do to fix things:

To address this issue, we had to issue a new set of ISOs, and since the package content was changing, rebuild metadata. Which in turn needed a complete rebuild of the ISOS ( but not the install tree ). Over the course of the morning, Fabian and Manuel were able to test the new tree, and our automated tests ran through for the ISOs

There was also a lot of rollback work that needed to be done, including handling the torrent tracker, issuing new torrents, making sure the mirror network etc. Much of which is done manually; and the main reason things took almost 18 hrs to resolve.

Steps taken to ensure this problem does not happen again:

I've now moved a large number of tests to the staging machine as well, including the rpm tests. This adds an additional 3 hours to the process, but its a worthwhile safeguard.

I hope this helps clear up things. Also, md5sum and sha1sum for ISOs as well as torrent files are published along with the torrent files and available on all mirrors. They will also be mentioned in the actual CentOS-5.6 Release Announcement. Everyone should check to make sure they get the right ones.

If you get the new .torrent files and drop in into the same place as the older ones did, you should see most of your data be reused ( 30% on the DVD and 86% on the CD's ).

Couple of things that should answer most questions that people are asking about the 5.6 Release status:

• TimeLine: Stuff is syncing around, it could be upto another day before you see packages, tree and isos on your local mirrors.
• Updates: Yes, it includes all updates released into the distro and most updates from that point on ( exceptions: java packages, subversion and openoffice.org ). These 3 package updates will come through in the next 24 hrs, but due to the churn in the mirrors, might be a few more hours before they are publicly visible on a mirror near you.
• LiveCD: Yes, the LiveCD's for i386 and x86_64 are also included
• Plus Kernels: Yes, the plus kernels are included in the release with no delay.
• Torrents: Torrents will be published as soon as the release is announced. This is to work around the situation where people will download the torrent and use yum to hit mirrors that are not completely updated as yet. Causing non-trivial issues ( as they end up installing 5.5 pkgs that then need to be updated again ). So once the bit-flip happens, we use the mirror status monitors to workout the state-of-external-mirrors, once there is a reasonable number in sync, we will release torrents, publish md5sum and sha sum's for the isos and announce the release.
• Update Announcements: There will be individual announcements for all updates released since EL5.6 was released upstream. There will be no announcements for packages rolled into the distro. So in a nutshell: it will be the same as its always been for CentOS-5.
• Will you need to change anything on your machine to get 5.6 ? : No, once its released and publicly available everywhere a regular 'yum update' will move your machine from centos-5.x to 5.6

Please keep in mind that till the release is announced ( keep an eye on http://lists.centos.org/mailman/listinfo/centos-announce, things can still change.

And finally : If you dont run a public mirror listed as a centos mirror, please do not rsync from the centos.org machines for the next 3 to 5 days. Use a large pipe mirror near you instead.

Update: CentOS-5.6 is now released

Firstly, CentOS is super popular out there with the ruby guys and there are quite a few ruby based apps deployed on CentOS these days.

Secondly, every man - his dog - and kitchen sink seem to be building ruby rpms. Not apps or gems or anything, but Ruby itself. There must be over a dozen repos and over a few hundred private rpm builds floating around.

What needs to happen for these people to merge efforts ? CentOS is more than capable of hosting out-of-distro builds, either in Extras or CentOSPlus and there is plenty of precedence in the ecosystem to use namespace overloading to handle multiple build variants. Its not pretty, but it works. And using a sensible set of defaults also means that its mostly new-user friendly.

- KB

People who have been following the centos-devel list would have noticed a ressurection of the CentOS website Ver2 effort[1]. Which is a great sign, however, the intensity and the enthusiasm shown is about at par with the efforts that went into creating the original plan - a few years back. And the one thing that stands out from that effort was that almost always, we had communication issues between the various people doing the various tasks, there was a clear missing internal-customer, there were parallel efforts for some tasks where the teams didnt know someone else was also doing the same thing, at the same time. This time, it looks like we might be heading back into the same sort of a situation. And I say that mostly since one can already see people trying to solve problems we might not have, as well as trying to expand on the actual deliveryable proposal. Feature creep, from the outside in. Dont get me wrong, I am not being negative about whats going on, its great! The fact that this effort is getting off the ground is very good news, One thing that CentOS really does ned is a better, more functional and easy to navigate website.

Thinking about the task and communication side of things, I was wondering if perhaps we could adopt an agile sort of a development format. Are there any other open source distributed communities who have been able to adopt such a process ? I am sure there must be some. Perhaps we should look at how they did the coordination and how they worked with the idea of the 'development sprint'. Working on a delivery dates by fixed date is'nt going to work out too well, after all this is still people contributing their spare time and a best-effort situation. But on the other hand, we still need something in place that allows the other developers, admin and design people to keep track of each individual task thread. So the thing is, if there are other projects that have done something o this nature, what tools were used ?

I know that mingle ( http://www.thoughtworks-studios.com/mingle-agile-project-management ) has a license that allows us to use it for open source projects without any cost. Also, there is an OpenAtrium install within the .centos.org space, is that worth looking into ? I suspect getting OpenAtrium into a format we like, might need some php side development. And starting a project by developing on the management tool isnt the best way forward. There is also Pivotal Tracker ( http://www.pivotaltracker.com/ ) which is a hosted service. I know they recently announced that they were no longer going to be totally free anymore; but from what I can tell all public projects are still free to host, and memebers are not charged anything to join those projects.

Recently there has also been talk about AgileFant ( http://www.agilefant.org/ ), and there is xplanner ( http://www.xplanner.org/ ). Neither of which I've used. Does anyone have opinions ? Thoughts ?

- KB

[1] Maybe it should be called Website Ver2.1!

A while back, Red Hat started pushing out early-releases for some updates. I'm guessing it was due to customers and users asking for eary access to trivial fix's. These early releases were put into a repository ( channel for you RHN users! ) called fastrack, with the condition being that these packages had not been through the complete or proper QA process. Users were advised at the time they should be selective in picking packages from this repo, only to solve immediate issue and consider these packages to not really be 'recommended' by Red Hat. So, thats the background to the 'fastrack' entity.

These packages were pushed out at various intervals during the life of a point release, and then either rolled in or discarded completely when the next point release came around. Eg. Package-1.1-3 released into fastrack during the life of RHEL-5.2 might make it into the main distro when 5.3 is released. On the other hand, its also possible that by the time 5.3 came around this Package-1.1-3 had been updated to Package-1.1-4, making the Package-1.1-3 redundant ( unless one was on the EUS or the z-stream etc platform, but lets not get sidetracked here ).

That nicely brings me to the point of considering update and upgrade notifications. Red Hat publish release info at various places, and there are different mechanisms for one to get to those notifications. One can get them via email, on the web, within RHN, through oval notifications in some cases etc. But the interesting point is - announcements are not re-issued when packages from fastrack move into the main distro. There is a nice silent move from one to the other. So if you are not tracking announcements within the fastrack repo, you might see something sneak up on you that has no origin notification issued ( or so it would seem to you ).

Within CentOS we dont have this issue. Mainly since we don't build the fastrack rpms regularly. There are quite a few reasons for that, I'll blog about why we did'nt and how we are going to try and solve that problem with 5.6 and 6.0 going forward. But the main thing is that we dont build those fastrack rpms at this time and have'nt done so in the past. Which creates an semi-significant difference in the announcement stream, in that we will announce those packages which have moved from fastrack to the main distro - but we will announce them when they hit the distro. But there will be no announcements within CentOS for packages that are released upstream into fastrack and then never make it into the main distro. I havent done any work to quantify this, but there are examples of when this has happened in the past.

So, if you are going to compare announcements. Keep this one thing in mind. Also, to all the people scraping the centos-announce mailing list to get update details into your automation, please consider using an alternative route. There are quite a few other options, use one of these alternatives.

- KB

Trying to pull from a remote repo, just ran into this strange error being reported by gitosis.

ERROR:gitosis.serve.main:Arguments to command look dangerous


And in typical gitosis ways, its down to trawling the sources to workout what the problem might be. In this case it turns out that I had a / at the end of the reponame. I was trying to :

git clone ssh://git@remotehost/blah/repo/


Which made gitosis quite angry, it did not like that / at the end of the clone request. Changing that to 'ssh://git@remotehost/blah/repo' makes things work again.

- KB

Should open source projects, not focused on source or code at all but more on a service or a resource, really be called 'open source' projects ? I feel there might be a need to define the idea of an 'open service' or an 'open resource' project.

- KB

December 2010 was a crazy month. Extremely hectic, exceptionally busy. But I spent all that time doing stuff that was neither desirable nor pleasant. We had a bereavement in the family followed by me being put into jury duty to be interrupted by a fairly serious illness. 99% of the holiday season was spent in bed, under the covers, away from people I care about and away from the computers!

I did get 3 days into the jury duty before I fell ill. And it was a mostly wasted time experience; after being put into one case only to have it dismissed after 2 days and then spending a whole day hanging around in the terrible waiting rooms for a while day. If they make the experience a bit more comfortable and slightly more productive, it might be something more people find acceptable use of their time. But unless they do that - its all mostly just a waste of time.

But dec/2010 wasent all down and out. I had two very interesting phone conversations, both of which I shall blog about in the next few days. And I got lots of rest! lots and lots. Just would have preferred if it had been while I wasent under bed rest orders with massive antibiotics doses being shoveled into my system and a total beer deprivation. Guess I now need to plan out a get-some-beer-in agenda!

So, lets see what 2011 has in store.

During most of December I was offline, and so there are some serious serious mail queues and postponed work / open source stuff. I'm trying as hard as I can to catch up, but it might be a few days before I am there.

- KB

I've never been big on new years resolutions, but this year I have one. To Blog. Started off initially as a 'I should blog more often', then I realised that I didnt really blog much at all through 2009 and 2010. So the aim is to change that : 2011 will be a good year for blogging. For me anyway.

- KB

If you insert data using server side metadata like now(), think about the implications in a multi-machine database spread specially when you have load.

These sort of problems are somewhat reduced when using rowlevel / binary replication rather than just sql replay. But think about the implications this is going to have across the entire system, the application and specially think about how you expect data in those fields to be used in the future.

- KB

One fact of life with the sysadmin lifestyle is that most issues are task and event driven rather than project driven. For me, over the last 6 months, the split has been about 80% issue driven and 20% project driven. And most tasks tend to be short, event or incident driven. Short in this case might be 10 minute tasks or even 3 day long tasks, but not the sort of things one would handle in project scenarios : 3 to 4 month long tasks, split into multiple sprints.

With such a high rate of churn in the ToDo lists, one tends to often overlook the automate angle in return for short term win. On the other hand, its also easy to slip into a state where over aggressive automation leads to a case of seriously low on-the-day productivity. The way I've tried to get pragmatic about this is to adopt a policy of automating at the second instance, monitoring at the first.

So short tasks are fine to execute manually the first time they come up - but ensure there is monitoring around it. The second time a task is repeated, automate if possible. And if the same or similar issue comes up a third time, and its still not automated - reconsider the task. Either break it up into smaller chunks so it can be automated or reconsider what and how you are doing. Its only in exceptional corner cases that you will find automation fail, even for repeated tasks.

In highly repetitive environments, using a config and change management system should never be optional. Not only does one get a large portion of the admin-task automation win, but you also get an audit trail, state reproducibility and speed in deployment.

- KB

This question comes up often : how does one get started in the world of Linux sysadmin. And to be honest, I dont think there is a clear answer to that. The state of Linux certification is not ideal. There are a few courses one might get onto, like the RHCE. But doing those without any background info will leave you unable to really get the full benefit since they all makes assumptions : that you are already aware of the basics.

One way to start off in sysadmin, is to grab a good book about the topic. Then install and run through the various install options and get a couple of VM's setup with your favourite Linux distro. Start with one distro and stay with that one distro for the first few months atleast. But its also important that once you are familiar with the basics, you do move onto some other distro and see how things work there. One cant really put down into words or ever really express easily that the bottom line is Linux. No matter what distro you move onto, the only thing different would be how things are laid out and general communication around the platform - the main OS is Linux, and that never changes.

While you work with a new platform, its quite important that you really get involved with it and an easy way to do that is to look at what you are doing now, whatever platform that might be on, and try to do the same thing, in the same way on the Linux machines. Once you can get the basics in place, you should really switch to using that linux install as your main workstation. While this might not give you much in terms of low level admin abilities, it will give you a user perspective on things. And I have always thought that the best admins are those who consider the user pespective, the developer perspective and then the platform ( and admin ) perspective. End of the day, lets not forget that the computers are here to do a job, and the admins role is to make sure that the job is done to its best ability. But dont lose context : the aim is still to run that job.

In the early days of my Linux experience, I used to find it hard to relate to other people's applications and what they might be doing with their computers and their networks. It was hard since I wasent actually in those roles, so even coming up with situations was hard. And looking for experience-situations I realised that the best way to get one's hooks into an app was to join the mailing list for that app. And work on some of the issues that people brought up, ask questions about why certain things were being done in a specific way and to look at bug reports that people were posting about that app - since that clearly showed how that app was being used 'in the real world', and it gave me a very good foundation to build on. About 14 years later, I still think that the usergrounps for specific apps are the best way to really learn about the app, how its managed, what best practises around that might be and what the developer / user perspective is for those apps.

Finally, being able to program and write real code helps. Dont believe the people who go around saying that admin needs no coding. On the other hand, speak to some of the good sysadmin people around ( and there are plenty ): pretty much everyone will tell you that they spend between 40 to 60% of their time writing scripts and working with apps where knowing the basics of development help. I am not saying that a certification in Java is needed, but having a good understanding of the basics for bash and atleast one of python,perl or ruby should be considered essential. The traditional mindset of unix/c still exists, but not many sysadmins these days need to get down to driver level development, and most functional code that sysadmins need to work with are well handled in the bash, ruby, perl and python worlds.

- KB

Although its in the man page, most people don't know that you can specify schedules using one of these meta specs :

@reboot : Run once after reboot.
@yearly : Run once a year, ie. "0 0 1 1 *".
@annually : Run once a year, ie. "0 0 1 1 *".
@monthly : Run once a month, ie. "0 0 1 * *".
@weekly : Run once a week, ie. "0 0 * * 0".
@daily : Run once a day, ie. "0 0 * * *".
@hourly : Run once an hour, ie. "0 * * * *".


So if one of those fits the requirements, use it. At the very least, you end up with better readability. And because users can set these on their own, it allows them to create jobs and services that come online on machine reboot. Without there being any need for them to get access to the system initscripts. Easy win for a problem that comes up quite a few times and in many places.

Now someone needs to make puppet be aware of these!

- KB

Hi Guys,

A few days back I announced that there will be a 'week of rpm-metadata' hacks, starting from today. With great regret I haveto now announce that this must be postponed for a few weeks. Things are going to be extremely hectic for the next two weeks with work and personal issues, so I wont really have the 2hrs or so per day needed to make this happen right now.

I will come through with this, and promise to do so. Just not this week.

- KB

One of the most under utilised things around in any CentOS sysadmins toolset is the rpm-metadata repositories. Some part of it comes from the fact that most people consider it to be yum specific, and while thats true to quite an extent - its not the only use for the rpm-metadata info. The bottom line is that rpm-metadata is a collection of information extract from the rpms contained in the repository and contains a lot of content, state and depenencies.

Over the course of next week, starting on the 26th September I'm going to run through a week of rpm-metadata tricks. One new cool thing to do with the info contained in the repo metadata's every day.

Some of these things are perhaps good candidates for yum plugins, but I'll try and stay clear of any dependancy on yum and also try and cover as many languages as possible : atleast ruby, python, php, bash and lua! Stay tuned.

- KB

Given the recent developments on the opensolaris front, there is again the issue of what OS do people run on their existing sparc hardware ? And I wonder if now is a good time to consider bringing up the CentOS on Sparc conversations again.

I have some hardware that can be used for the builds and I know there is enough contributed / contribute-able resources around to setup some development instances as well. However, the real issue is : is there much desire out there to have CentOS running on these machines ?

Even though I don't have any equipment in production running off Sparc anymore, I know there are quite a few people who do. And it looks like Oracle wants to put these people in a position where they either run a paid for supported Solaris or nothing at all. If we can do something to create an option, that would be awesome. And given the huge commitment to open source that the CentOS upstream has, doing it here on this platform makes the most sense since efforts put in will stay in an open source model, with equal opportunity for anyone to join the effort and continue it. Knowing that there is no vendor preferences or vendor isolation / ownership issues being involved.

CentOS also makes for a good choice since the porting and development effort would need to only really be done at one point - when a major new release is announced upstream - and from that point on, its a case of maintaining the builds for the lifespan. Considering that the lifespan for a CentOS release would be 7 years or more, it makes the effort even more worthwhile.

Usual suspects still hold true : long life, stable releases, documentation that is shared with the i386/x86_64 world for most parts, userspace support in the regular media, a clueful community and an awesome infrastructure setup etc.

Maybe we can sync up with the Fedora-sparc guys and see if there is any synergy there. I know that they have been getting ready for a new release soon.

Something to think about.

- KB

I had to build a fresh set of rpms for the area_cli tools, and decided it was also a good time to rebase some of my local personal build tools to rhel6beta2 since its there and I need to do some tests with it anyway.

Firstly, there are a couple of cool things with rhel6 beta2 - rpmdevtools is included by default, which helps gets things started and helps a bit in managing spec files etc.

What isnt so cool is that the newer rpm in el6beta creates packages which then cause issues with some of the older CentOS releases. eg. this areca_cli package built on el6beta2/x86_64 for target i686 caused this to happen with yum:

Downloading Packages:
areca_cli-1.83_091103-1.e 100% |=========================| 493 kB    00:00     
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
rpmlib(FileDigests) is needed by areca_cli
rpmlib(PayloadIsXz) is needed by areca_cli
Complete!
(1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Which causes the package to not install. This is on a CentOS-5.2 machine. And I need to maintain that at CentOS-5.2 due to various issues ( no, machine is not available on the internet, and only hosts a single app in production ). Trying the install on a CentOS-5.3 machine I get the exact same issue. Upgrading rpm to 4.4.2.3-18.el5 ( the version in 5.4 and 5.5 ) makes no difference.

On the other hand, builds run on CentOS-5 have no problems installing on EL6Beta. So for the time being it looks like all buildhosts and all build stuff will need to stay on EL5. Specially since that allows you to target CentOS-3 and CentOS-4 as well.

- KB

Note: yeah, I see that url pointing at bugzilla isnt idea. I'll look into plumbing in a bugs.centos.org reference instead.

Seven hundred comment spam posts to my blog in the last 24 hours!

ZOMGWTF!!

- KB

A question that has come up a few times in the last week. So I thought a blog post about it is in order here. While some people are aware of the situation, most of the people are not.

In the early days of EL5, there was a perl.i386 included in the distro, which Red Hat then took away and made perl.x86_64 the only perl in the x86_64 tree. This caused a non-trivial issue to people who already had a CentOS install and wanted to maintain that install. So in order to avoid having to deal with potentially hundreds of people who would have had a broken perl setup as well as yum update issues, we decided to maintain a perl.i386 in the extras repo for c5/x86_64. This would mean that people who already had a perl.i386 on their machines would still be able to keep things the way they were - and new installs would only get the perl.x86_64 ( since the install is only run from the os/ repository, it does not come in contact with the perl.i386 in extras at all ). Yum has also become a lot smarter about its handling of arch specific packages, which would mean that people not looking for a perl.i386 by name, would not need to worry about it at all.

Think of it this way - when we had to make that decision, we asked around. Specially people who use perl in their day to day jobs and people who have non trivial contributions to cpan about the impact of perl.i386 on a x86_64 install. None of them came back to say anything that would lead us to believe perl.i386 was going to be a problem. So we took the approach of not breaking installs - and keeping the perl.i386 in a place where most people could access it.

So why the sudden interest in this issue again ? We had a perl update for CentOS-5 go out 2 days back. And this brought in some interest - since there are *still* people, three years down the road, who have perl.i386 installs! but it was a bit of our fault as well. In the specific case of perl-5.8.8-32.el5_5.1, the perl.i386 pkg into extras/x86_64 lagged by a few hours - and that resulted in *some* mirrors getting out of shape for a few hours. That was my fault completely. If you tried to do an update for perl and hit one of those mirrors, during those few hours, yum would have complained about conflicts between perl.x86_64 and perl.i386 ( of-course, only if you have perl.i386 in the first place ).

To make sure this does not happen again in the future I've plumbed in a test that checks for perl.i386 in extras/x86_64 matching perl in updates/i386/; This should ensure that we can do the update push in the future unless all the perl packages are in place first.

Keeping all this in mind, should we have taken the hit in the early CentOS-5 days and let users deal with the update issues they would almost certainly have run into ? I still think we did the right thing long term.

- KB

So when almost everyone had given up on IA64, it looks like there are a few people around who are still running this kit! And quite keen on getting it usable with CentOS. To that aim, I've restarted the IA64 builds and am inviting everyone interested in this platform to come drop into #centos-devel on irc.freenode.net on Wednesday 9th June at 2100 UTC for a catchup.

In terms of what agenda for the chat, we have four issues that need attention:

• Setup some goals for the effort
• Nominate some people who could be contact points
• Where we are right now
• How do we get to the targets - so that would include deciding on co-ordination and collaboration process

See you all then!

- KB

Spent a bit of time last evening and tried to get a rhel6 beta domU going on a CentOS 5.4 dom0 - everything seems to go fine except that the VM will not boot. The problem is that the rhel6 anaconda creates a /boot with ext4, which pygrub on CentOS and RHEL 5 will not be able to boot.

To Fix this, make sure you select the option to review partition and disk layout, and manually change /boot from ext4 to ext2. Everything should just work fine.

- KB

Just had a look at the Red Hat Enterprise Linux 6 beta package list, and it looks quite well put together. With a very interesting tilt towards developers and people building large scale platforms. Ofcourse there is the expected virtualisation, storage and cluster suite improvements.

Disapointed that Exim is going away, its become my mta of choice over the last few years. Also disapointed that ruby is at 1.8.6 and isnt going to make 1.9.x. Although, having rubygems in the distro is good. Same with all the major Version control systems are now included, svn, git, cvs, rcs, mercurial and bazaar

On the other hand, python 2.6 is cool. Along with the inclusion of ipython, turbogears and pylons. Like the fact that amqp via qpid ( my implementation of choice ) are also included in the base distro. Same with FCoE, been testing it over the last few months and would be really nice to see it in a supported format now.

Also amongst the interesting stuff : bacula replacing amanda, like it. Sysklogd replaced by rsyslog, like it. Vixie-cron replaced by cronie, like it. Al the system-config-* tools are gone, dont care - never used them myself anyway. Xfs is now in the mainstream supported mode along with ext4, like it. Completely Fair Scheduler in the kernel, like it. No drbd, odd given that its being used quite a lot.

This is just a first look reaction, over the next few weeks I'm going to try and poke around a bit more and will blog about more specific things.

There was quite a flurry of activity last evening when people wanted to get their hands on the rhel6 beta isos. I've stood up some torrents, you can download them here :

http://www.karan.org/stuff/rhel6-i386-beta-dvd.torrent
http://www.karan.org/stuff/rhel6-ppc64-beta-dvd.torrent
http://www.karan.org/stuff/rhel6-x86_64-beta-dvd.torrent 

There are a few seeds, on fairly high speed links, to get you started.

- KB

In case you missed it, RHEL-6 public beta got announced a short while back. You can download it at : ftp://ftp.redhat.com/pub/redhat/rhel/beta/6/ and read the announcement at : http://press.redhat.com/2010/04/21/red-hat-enterprise-linux-6-beta-available-today-for-public-download/

The first thing that everyone will want to know is : when will centos-6-beta come out. Well, it wont be for a while. The reason is - this is a public beta, you can get it directly from Red Hat, and we want you to do that. Beta test the Red Hat builds, report issues and bugs at http://bugzilla.redhat.com/ and make sure you follow their rhel-6-beta list. And do this because, we all want a better upstream product.

So where does this leave CentOS and CentOS-6-beta ? We will setup a build process for CentOS-6-Beta, and we will go through the process of putting out a new release, across a restricted number of machines so that people can also test our builds, however we want the focus of this testing around CentOS-6-beta to be the packages we need to change, modify and adapt for CentOS. The core packages and testing should still be focused on RHEL-6, since a better upstream product helps all of us, and a large number of issues reported against CentOS-6-beta will almost certainly be pushed upstream.

- KB

Everytime a new CentOS Release is built, it takes about 48 hrs before people can start testing these releases. What I would like to be able to do is run some install tests within the buildsystem / QA infrastructure so that we can catch major issues earlier and fix them faster. But I need some help from everyone out there in order to achieve this. Central to this idea is the need for kickstarts and test scripts that reflect usage out in the users systems and scenarios. Lets consider kickstarts first.

The idea here is that we have a couple of machines, close to the buildsystem, that can take a centos release, and run through installs in a loop, reporting success or failure after each install. Even if we only start with one machine for each primary arch, i386 and x86_64, we should be able to cycle through over 120 install cycles per arch within 24 hours with the right kind of automation. The automation is what I would like to focus on, I need your help with the kickstarts.

If you are able to write up some kickstarts, or even just go through an install and adapt the resulting anaconda-ks.cfg file - send them to me. Send me kickstarts for just about any role or deployment strategy you can think of - to get the ideas flowing I've made a short list of thing that would be good to have kickstarts for; dont, in any way feel restricted to these roles:

My initial wish list is something along these lines:

• General install, with no changes in the installer, just filling out whatever is the minimum required input
• A minimal install that only needs cd-1
• A LAMP server
• A Postgresql server
• GNOME Desktop
• KDE Desktop
• Xen Dom0 host
• KVM host
• DHCP and Static networked server installs
• Install from CDrom
• Install over http
• Install over NFS
• Install from local disk
• Office SME server, with samba and email

Here's how you could contribute the kickstarts:

• Post it online and leave a comment here with a url pointing to it
• Post it to the centos-devel list ( http://lists.centos.org )
• Drop into #centos-devel on irc.freenode.net and point us at it there

The kickstarts that we collect will be published online ( unless you specifically ask me not to ) along with results from each install cycle and details that came from it.

I know its not very organised at this time. But along with the automation I will build a web interface that lists the kickstarts along with a git repo that lets people directly pull from and push patches into.

If you need some help getting started with kickstarts, take a look at the wiki article at : http://wiki.centos.org/TipsAndTricks/KickStart and the docs at : http://www.centos.org/docs/5/html/5.2/Installation_Guide/s1-kickstart2-file.html . I would also highly recommend looking at the /root/anaconda-ks.cfg file that is left behind on any new centos install. Then there is always google.

- KB