I had to build a fresh set of rpms for the area_cli tools, and decided it was also a good time to rebase some of my local personal build tools to rhel6beta2 since its there and I need to do some tests with it anyway.

Firstly, there are a couple of cool things with rhel6 beta2 - rpmdevtools is included by default, which helps gets things started and helps a bit in managing spec files etc.

What isnt so cool is that the newer rpm in el6beta creates packages which then cause issues with some of the older CentOS releases. eg. this areca_cli package built on el6beta2/x86_64 for target i686 caused this to happen with yum:

Downloading Packages:
areca_cli-1.83_091103-1.e 100% |=========================| 493 kB    00:00     
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
rpmlib(FileDigests) is needed by areca_cli
rpmlib(PayloadIsXz) is needed by areca_cli
Complete!
(1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Which causes the package to not install. This is on a CentOS-5.2 machine. And I need to maintain that at CentOS-5.2 due to various issues ( no, machine is not available on the internet, and only hosts a single app in production ). Trying the install on a CentOS-5.3 machine I get the exact same issue. Upgrading rpm to 4.4.2.3-18.el5 ( the version in 5.4 and 5.5 ) makes no difference.

On the other hand, builds run on CentOS-5 have no problems installing on EL6Beta. So for the time being it looks like all buildhosts and all build stuff will need to stay on EL5. Specially since that allows you to target CentOS-3 and CentOS-4 as well.

- KB

Note: yeah, I see that url pointing at bugzilla isnt idea. I'll look into plumbing in a bugs.centos.org reference instead.

Seven hundred comment spam posts to my blog in the last 24 hours!

ZOMGWTF!!

- KB

A question that has come up a few times in the last week. So I thought a blog post about it is in order here. While some people are aware of the situation, most of the people are not.

In the early days of EL5, there was a perl.i386 included in the distro, which Red Hat then took away and made perl.x86_64 the only perl in the x86_64 tree. This caused a non-trivial issue to people who already had a CentOS install and wanted to maintain that install. So in order to avoid having to deal with potentially hundreds of people who would have had a broken perl setup as well as yum update issues, we decided to maintain a perl.i386 in the extras repo for c5/x86_64. This would mean that people who already had a perl.i386 on their machines would still be able to keep things the way they were - and new installs would only get the perl.x86_64 ( since the install is only run from the os/ repository, it does not come in contact with the perl.i386 in extras at all ). Yum has also become a lot smarter about its handling of arch specific packages, which would mean that people not looking for a perl.i386 by name, would not need to worry about it at all.

Think of it this way - when we had to make that decision, we asked around. Specially people who use perl in their day to day jobs and people who have non trivial contributions to cpan about the impact of perl.i386 on a x86_64 install. None of them came back to say anything that would lead us to believe perl.i386 was going to be a problem. So we took the approach of not breaking installs - and keeping the perl.i386 in a place where most people could access it.

So why the sudden interest in this issue again ? We had a perl update for CentOS-5 go out 2 days back. And this brought in some interest - since there are *still* people, three years down the road, who have perl.i386 installs! but it was a bit of our fault as well. In the specific case of perl-5.8.8-32.el5_5.1, the perl.i386 pkg into extras/x86_64 lagged by a few hours - and that resulted in *some* mirrors getting out of shape for a few hours. That was my fault completely. If you tried to do an update for perl and hit one of those mirrors, during those few hours, yum would have complained about conflicts between perl.x86_64 and perl.i386 ( of-course, only if you have perl.i386 in the first place ).

To make sure this does not happen again in the future I've plumbed in a test that checks for perl.i386 in extras/x86_64 matching perl in updates/i386/; This should ensure that we can do the update push in the future unless all the perl packages are in place first.

Keeping all this in mind, should we have taken the hit in the early CentOS-5 days and let users deal with the update issues they would almost certainly have run into ? I still think we did the right thing long term.

- KB

So when almost everyone had given up on IA64, it looks like there are a few people around who are still running this kit! And quite keen on getting it usable with CentOS. To that aim, I've restarted the IA64 builds and am inviting everyone interested in this platform to come drop into #centos-devel on irc.freenode.net on Wednesday 9th June at 2100 UTC for a catchup.

In terms of what agenda for the chat, we have four issues that need attention:

• Setup some goals for the effort
• Nominate some people who could be contact points
• Where we are right now
• How do we get to the targets - so that would include deciding on co-ordination and collaboration process

See you all then!

- KB

Spent a bit of time last evening and tried to get a rhel6 beta domU going on a CentOS 5.4 dom0 - everything seems to go fine except that the VM will not boot. The problem is that the rhel6 anaconda creates a /boot with ext4, which pygrub on CentOS and RHEL 5 will not be able to boot.

To Fix this, make sure you select the option to review partition and disk layout, and manually change /boot from ext4 to ext2. Everything should just work fine.

- KB

Just had a look at the Red Hat Enterprise Linux 6 beta package list, and it looks quite well put together. With a very interesting tilt towards developers and people building large scale platforms. Ofcourse there is the expected virtualisation, storage and cluster suite improvements.

Disapointed that Exim is going away, its become my mta of choice over the last few years. Also disapointed that ruby is at 1.8.6 and isnt going to make 1.9.x. Although, having rubygems in the distro is good. Same with all the major Version control systems are now included, svn, git, cvs, rcs, mercurial and bazaar

On the other hand, python 2.6 is cool. Along with the inclusion of ipython, turbogears and pylons. Like the fact that amqp via qpid ( my implementation of choice ) are also included in the base distro. Same with FCoE, been testing it over the last few months and would be really nice to see it in a supported format now.

Also amongst the interesting stuff : bacula replacing amanda, like it. Sysklogd replaced by rsyslog, like it. Vixie-cron replaced by cronie, like it. Al the system-config-* tools are gone, dont care - never used them myself anyway. Xfs is now in the mainstream supported mode along with ext4, like it. Completely Fair Scheduler in the kernel, like it. No drbd, odd given that its being used quite a lot.

This is just a first look reaction, over the next few weeks I'm going to try and poke around a bit more and will blog about more specific things.

There was quite a flurry of activity last evening when people wanted to get their hands on the rhel6 beta isos. I've stood up some torrents, you can download them here :

http://www.karan.org/stuff/rhel6-i386-beta-dvd.torrent
http://www.karan.org/stuff/rhel6-ppc64-beta-dvd.torrent
http://www.karan.org/stuff/rhel6-x86_64-beta-dvd.torrent 

There are a few seeds, on fairly high speed links, to get you started.

- KB

In case you missed it, RHEL-6 public beta got announced a short while back. You can download it at : ftp://ftp.redhat.com/pub/redhat/rhel/beta/6/ and read the announcement at : http://press.redhat.com/2010/04/21/red-hat-enterprise-linux-6-beta-available-today-for-public-download/

The first thing that everyone will want to know is : when will centos-6-beta come out. Well, it wont be for a while. The reason is - this is a public beta, you can get it directly from Red Hat, and we want you to do that. Beta test the Red Hat builds, report issues and bugs at http://bugzilla.redhat.com/ and make sure you follow their rhel-6-beta list. And do this because, we all want a better upstream product.

So where does this leave CentOS and CentOS-6-beta ? We will setup a build process for CentOS-6-Beta, and we will go through the process of putting out a new release, across a restricted number of machines so that people can also test our builds, however we want the focus of this testing around CentOS-6-beta to be the packages we need to change, modify and adapt for CentOS. The core packages and testing should still be focused on RHEL-6, since a better upstream product helps all of us, and a large number of issues reported against CentOS-6-beta will almost certainly be pushed upstream.

- KB

Everytime a new CentOS Release is built, it takes about 48 hrs before people can start testing these releases. What I would like to be able to do is run some install tests within the buildsystem / QA infrastructure so that we can catch major issues earlier and fix them faster. But I need some help from everyone out there in order to achieve this. Central to this idea is the need for kickstarts and test scripts that reflect usage out in the users systems and scenarios. Lets consider kickstarts first.

The idea here is that we have a couple of machines, close to the buildsystem, that can take a centos release, and run through installs in a loop, reporting success or failure after each install. Even if we only start with one machine for each primary arch, i386 and x86_64, we should be able to cycle through over 120 install cycles per arch within 24 hours with the right kind of automation. The automation is what I would like to focus on, I need your help with the kickstarts.

If you are able to write up some kickstarts, or even just go through an install and adapt the resulting anaconda-ks.cfg file - send them to me. Send me kickstarts for just about any role or deployment strategy you can think of - to get the ideas flowing I've made a short list of thing that would be good to have kickstarts for; dont, in any way feel restricted to these roles:

My initial wish list is something along these lines:

• General install, with no changes in the installer, just filling out whatever is the minimum required input
• A minimal install that only needs cd-1
• A LAMP server
• A Postgresql server
• GNOME Desktop
• KDE Desktop
• Xen Dom0 host
• KVM host
• DHCP and Static networked server installs
• Install from CDrom
• Install over http
• Install over NFS
• Install from local disk
• Office SME server, with samba and email

Here's how you could contribute the kickstarts:

• Post it online and leave a comment here with a url pointing to it
• Post it to the centos-devel list ( http://lists.centos.org )
• Drop into #centos-devel on irc.freenode.net and point us at it there

The kickstarts that we collect will be published online ( unless you specifically ask me not to ) along with results from each install cycle and details that came from it.

I know its not very organised at this time. But along with the automation I will build a web interface that lists the kickstarts along with a git repo that lets people directly pull from and push patches into.

If you need some help getting started with kickstarts, take a look at the wiki article at : http://wiki.centos.org/TipsAndTricks/KickStart and the docs at : http://www.centos.org/docs/5/html/5.2/Installation_Guide/s1-kickstart2-file.html . I would also highly recommend looking at the /root/anaconda-ks.cfg file that is left behind on any new centos install. Then there is always google.

- KB

Sony has announced that the next ps3 firmware update - 3.21 - will no longer support the ability to install and run linux on your ps3's. You can read the full announcement here : http://blog.us.playstation.com/2010/03/28/ps3-firmware-v3-21-update/

This does, ofcourse, only impact the older ps3 units. The newer slimline ps3 units never had support for linux. Keep in mind that this whole idea of new and old is a bit relative. Since its been possible to get the 'older' units at retail outlets all this while - ok, at a much higher prices ( £349 for a 120GB 'fat' ps3 as opposed to the £240 for a 250GB 'slim' ps3 ). But it has been possible to get these machines, and its been possible to use them as Linux hosts.

For a long time I believed these models were still being churned out, in much lower numbers and only retailed through specific vendors. But it seems there has been no 'fat' style ps3 units manufactured by SCE in 2010. Attempts to get info from Sony about this situation have been futile. I just get generic replies that they will get back in touch etc, and that I should get in touch with my local SCE vendor.

How much does this haveto do with IBM's recentish news that they are no longer interested in the Cell platform ( http://www.hpcwire.com/features/Will-Roadrunner-Be-the-Cells-Last-Hurrah-66707892.html ) ? Unless they plan on replacing the platform completely, surely it cant be much work keeping the linux / otheros option open. And I dont buy the argument that its cutting into their sales - if there was a market that large, Sony could just have an alternative box that they would put into the retail stores or even via a specialised vendor.

So where does this leave people who bought and run the ps3's only for the otheros support ? I guess they would need to not upgrade their firmwares for one!

- KB

Just signed upto the London Devops mailing list, and thought I'd just point out that one does not need a google account to join a google groups hosted list. Just send an email to the <list-name>+subscribe@googlegroups.com with 'subscribe' in the subject line of the email, and their list managment software will sign you up.

So in this case, sending an email with:

To: london-devops+subscribe@googlegroups.com
Subject: Subscribe

.. would be all thats needed to get you on there.

Google groups has a help page with details on what all services they host which dont need a google account to use.

A few days back I blog'd about my .screenrc and what it could do etc, something to get people interested in screen and start using it. But in order to really use 'screen' properly, dont forget the 'screen -xRR' in your .bash_profile.

That will make sure that when you ssh to a remote machine, it auto joins an existing screen - and if none exists then creates a new screen instance and drops you inside it. Try it, most people like it :)

- KB

Just upgraded to b2evo 3.3, lets see if this is any better than the b2evo 2.2 install I had on here. The db migration and content move went through fairly painlessly. Just need to tweak the skin back to my own liking a bit.

First thing though - not sure if I like this 'rich text' editing textbox area that now seems to be the default in b2evo 3.3; might live with it though.

- KB

Although, only for the 5th and 6th, will be back in London on the 6th evening.

- KB

Everyone uses screen right ? If not, what's wrong with you crazy person ?

Also everyone at some point or the other realises that its worth getting a decent screenrc in place. So did I, many years back. Essentially the screenrc lets you setup a few things about how you want screen to look and work. A config file, yes. And here is what I currently use:

caption string "%?%F%{= Bk}%? %C%A %D %d-%m-%Y %{= kB} %t%= %?%F%{= Bk}%:%{= wk}%? %n "
hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'

You can download it here : http://www.karan.org/stuff/MyScreenRc : remember to move it to ~/.screenrc .

What you see here is the result of what came from many iterations of changes. I remember in 2006 my screenrc put up a 3 line display with everything that anyone would ever need, including load on a machine, number of unread emails, days-for-present-sprint to end etc. And to be honest, while most of that is good to know I think the only things one needs on the console are : hostname, screen windows and labels for those and finally the time. On a nice dull background so its not in your face too much. Which is exactly what my existing screenrc is setup to be.

One interesting thing is that often when pairing with me or when people see my console shell they would ask why bother with the hostname, specially since you should know from the shell prompt what machine you are on. Well, yes - but then that does not work out too well when you start cascading shells between machines. Eg: machine 1 -> machine 2 -> machine 3. It can get tricky, but because I have my status bar setup on each shell - here is what my terminal looks like in that case:

So you can easily see how the screens are stacked up and on what machine with screen windows on each machine. Quite like that.

So do you have a screenrc setup ? tell me about it. If not - then well, this one here should be a good place to start from - and tell me if you like it, hate it or think it can be improved in some way.

- KB

Sometimes the CentOS-5/updates repository gets into a state wherein people can see the updated packages in the repo using a browser ( eg at http://mirror.centos.org/centos/5/updates/i386/RPMS/ ) but when they try and get the updates on their machine : yum is unable to 'see' these packages. The reason for this is that while the physical rpm packages have been pushed out to the mirrors, the yum metadata has not been updated. And yum relies on this metadata to workout what packages are available.

I'll try and briefly explain why this happens.

The CentOS mirror network is setup in layers, the first two levels of this network constitute the core - and are not available publicly. The third layer is what most people see at http://mirror.centos.org/ and the large mirror networks like http://mirrors.kernel.org/centos/ , http://ftp.heanet.ie/mirrors/centos/ , http://www.mirrorservice.org/sites/mirror.centos.org/ etc ( there are over 100 of them! ). The fourth layer in the mirror network are the smaller - but still very important - mirrors that sync from the third layer machines. The fifth and final layer is the private and internal company wide mirrors run by admins within their own networks.

When a new update is issued, the updated content is pushed into layer one. From there it makes its way down to layer two and then onto layer three. At this point, the content is now publicly visible, however it might not be on layer four and five machines. There are quite a few more complexities involved in the process, but two issues worth noting at this point are that (1) the whole process is automated and (2) the 'check and refresh' frequency is fairly high. eg. content moves from layer one to layer two in almost real time.

So why the metadata lag ? What we want to try and do is make sure the update does not 'break' any process. So we want to make sure that packages are visible and available to some relatively large number of mirrors before people and machines start requesting them. Therefore the metadata lag. Here is a snippet of code from the release-to-production script, which should make it easier to comprehend :

do_genMetadata
rsync -Pvar --include="*.rpm" --exclude="*" * $SeedHost:$SeedPath
do_seedCheck
rsync -Pvar * $SeedHost:$SeedPath


What happens in this case is that the metadata is generated, and only the rpms are pushed upto the layer one machines. the 'do_seedCheck' function will then block the process till such time as it can see the rpms publicly visible on a random cross section of mirrors ( looping every five minutes ). Once that mark is reached, it will return and the regular rsync which then includes the metadata will get run. And as soon as this metadata is visible, yum will start pulling the updated packages for users.

The other thing to keep in mind is that this is not a one-off occurrence. The yum metadata *always* lags the rpms by sometime. Lets say X seconds. The value of X now depends on how long it takes for those tests to pass + how long it takes the metadata to work its way down the mirror's chain. In a majority of the cases, the time lag is just a few minutes. eg the PyXML and gd updates from earlier today went through in less than 20 minutes. On the other hand there are times when the lag could reach many hours. eg an OpenOffice.org update could delay metadata for upto 8 - 12 hours since the mirrors need to shift almost 1.3GiB just for that one update, per machine.

Can we speed things up a bit ? Absolutely. One way, that we are hoping to trial in the next few weeks, is to move atleast some part of the core-mirror-machines to start using a push-style update process rather than the existing pull mechanism. That would reduce drastically the amount of time machines sit out-of-sync. But more on that in another post another day!

- KB

Have a good holiday season everyone!

2009 has been a great year on many fronts. I am sure 2010 will be better.

- KB

With every release, Thunderbird has become better and better as a personal email client. But over the last few releases, it definitely feels like as its getting better at personal email handling its getting worse for mailing list traffic.

I am on a few lists ( 32 to be exact ) and a couple of them get loads of emails ( 150+ a day ), but even so why is thunderbird at almost 90% cpu usage with over 800MiB of ram used ? If I take away the mailing lists account and only use it for my personal emails things are fine. Thunderbird will almost never corss 12 - 13% cpu usage and rarely ever cross 200MiB ram usage. Its not that I dont get personal emails; I do get a fair bit - not including spam - about 400/day.

Someone please fix it!

- KB

Just to change things around a bit, new blog skin!

- KB

Posting this, since a fair few people seem to have asked about it on the internet but no-one really had an answer.

If you are looking to add a optical media drive ( CDRom / DVDRom etc), just keep in mind that its a conventional atapi slimline drive that fits into the slot in the front. Its literally a case of flipping down the front cover, and sliding in a drive. I 'borrowed' a dvd drive from an old laptop, removed the mounting around the drive and pop'd it into the running machine. OpenFirmware was able to detect it right away and quite happily went through a 'boot cdrom:' process.

- KB

SIGSECTOOMUCH : The state of play when you over secure a machine to the extent that you can yourself no longer get onto the machine.

- KB

I use a proxy to get onto irc, which is good since I can get online from anywhere and do so over mostly a secure link. Another advantage of doing it this way is that my nickname is always online on irc and people can leave msgs or followup on conversations even when I am offline. Which has been a great way to stay in touch ( slowly ) with people who are in completely different time zones.

This saturday morning the machine I used for my irc proxy lost all internet access - which means 'I' went offline. Only having restored the link a few hours back I now realise that there were a few ongoing conversations that seem to have either finished in the time period or just gone away. Will try and catch up with people over the next few days for details, but if anyone is waiting on input from me - please ping again!

- KB

So, initially it was just a bunch of websites, it then became a part of Andover.net - soon to be merged with valinux. That went on for a bit before they changed over to being 'Open Source Developers Network (OSDN)', to only change over to being called ' Open Source Technology Group (OSTG)' soon after. Then it was 'Sourceforge, Inc and' now they are called 'GeekNet Inc.'

And this is the whole stack of slashdot.org freshmeat.net sourceforge.net etc.

Remember any other names they have been called over the years ?

- KB

At approx 16:45 the Coreix Data Center suffered a power loss, UPS took over the load for short while, however the Generators did not come online in time and it took till 7:35 for power to be restored. There are details posted on the coreix status page at http://status.coreix.net/

While power did get restored to the entire DC, only 1 of the machines came back online. All the rest have needed some level of manual intervention! I'm working with the support people ( who are a really good and effective bunch of guys ) to get the other machines online.

Services affected are:

• RPMForge svn repo
• RPMForge master mirror
• RPMForge mailing lists
• Karan.org Build services
• Karan.org testing services
• CentOS.org ipv6 test / qa setups
• CentOS.org Package and Automated testing development machine

Once all services are restored, I'll update this blog post with details. And apologies for this completely unplanned and avoidable outage.

Update: as of 23:45 26th Nov, all services are now restored.

In the overall scheme of things, I wonder if its Java that helped Linux get some level of traction in the large enterprise markets, of if its the other way around : Linux helped Java get into these shops and stay there.

Even though I've never been a big Java fan myself, its hard to argue against the fact that there is some level of hand and glove thing going on with Java and Linux in the enterprise setups.

- KB