Although, only for the 5th and 6th, will be back in London on the 6th evening.

- KB

Everyone uses screen right ? If not, what's wrong with you crazy person ?

Also everyone at some point or the other realises that its worth getting a decent screenrc in place. So did I, many years back. Essentially the screenrc lets you setup a few things about how you want screen to look and work. A config file, yes. And here is what I currently use:

caption string "%?%F%{= Bk}%? %C%A %D %d-%m-%Y %{= kB} %t%= %?%F%{= Bk}%:%{= wk}%? %n "
hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B} %d/%m %{W}%c %{g}]'

You can download it here : http://www.karan.org/stuff/MyScreenRc : remember to move it to ~/.screenrc .

What you see here is the result of what came from many iterations of changes. I remember in 2006 my screenrc put up a 3 line display with everything that anyone would ever need, including load on a machine, number of unread emails, days-for-present-sprint to end etc. And to be honest, while most of that is good to know I think the only things one needs on the console are : hostname, screen windows and labels for those and finally the time. On a nice dull background so its not in your face too much. Which is exactly what my existing screenrc is setup to be.

One interesting thing is that often when pairing with me or when people see my console shell they would ask why bother with the hostname, specially since you should know from the shell prompt what machine you are on. Well, yes - but then that does not work out too well when you start cascading shells between machines. Eg: machine 1 -> machine 2 -> machine 3. It can get tricky, but because I have my status bar setup on each shell - here is what my terminal looks like in that case:

So you can easily see how the screens are stacked up and on what machine with screen windows on each machine. Quite like that.

So do you have a screenrc setup ? tell me about it. If not - then well, this one here should be a good place to start from - and tell me if you like it, hate it or think it can be improved in some way.

- KB

Sometimes the CentOS-5/updates repository gets into a state wherein people can see the updated packages in the repo using a browser ( eg at http://mirror.centos.org/centos/5/updates/i386/RPMS/ ) but when they try and get the updates on their machine : yum is unable to 'see' these packages. The reason for this is that while the physical rpm packages have been pushed out to the mirrors, the yum metadata has not been updated. And yum relies on this metadata to workout what packages are available.

I'll try and briefly explain why this happens.

The CentOS mirror network is setup in layers, the first two levels of this network constitute the core - and are not available publicly. The third layer is what most people see at http://mirror.centos.org/ and the large mirror networks like http://mirrors.kernel.org/centos/ , http://ftp.heanet.ie/mirrors/centos/ , http://www.mirrorservice.org/sites/mirror.centos.org/ etc ( there are over 100 of them! ). The fourth layer in the mirror network are the smaller - but still very important - mirrors that sync from the third layer machines. The fifth and final layer is the private and internal company wide mirrors run by admins within their own networks.

When a new update is issued, the updated content is pushed into layer one. From there it makes its way down to layer two and then onto layer three. At this point, the content is now publicly visible, however it might not be on layer four and five machines. There are quite a few more complexities involved in the process, but two issues worth noting at this point are that (1) the whole process is automated and (2) the 'check and refresh' frequency is fairly high. eg. content moves from layer one to layer two in almost real time.

So why the metadata lag ? What we want to try and do is make sure the update does not 'break' any process. So we want to make sure that packages are visible and available to some relatively large number of mirrors before people and machines start requesting them. Therefore the metadata lag. Here is a snippet of code from the release-to-production script, which should make it easier to comprehend :

do_genMetadata
rsync -Pvar --include="*.rpm" --exclude="*" * $SeedHost:$SeedPath
do_seedCheck
rsync -Pvar * $SeedHost:$SeedPath


What happens in this case is that the metadata is generated, and only the rpms are pushed upto the layer one machines. the 'do_seedCheck' function will then block the process till such time as it can see the rpms publicly visible on a random cross section of mirrors ( looping every five minutes ). Once that mark is reached, it will return and the regular rsync which then includes the metadata will get run. And as soon as this metadata is visible, yum will start pulling the updated packages for users.

The other thing to keep in mind is that this is not a one-off occurrence. The yum metadata *always* lags the rpms by sometime. Lets say X seconds. The value of X now depends on how long it takes for those tests to pass + how long it takes the metadata to work its way down the mirror's chain. In a majority of the cases, the time lag is just a few minutes. eg the PyXML and gd updates from earlier today went through in less than 20 minutes. On the other hand there are times when the lag could reach many hours. eg an OpenOffice.org update could delay metadata for upto 8 - 12 hours since the mirrors need to shift almost 1.3GiB just for that one update, per machine.

Can we speed things up a bit ? Absolutely. One way, that we are hoping to trial in the next few weeks, is to move atleast some part of the core-mirror-machines to start using a push-style update process rather than the existing pull mechanism. That would reduce drastically the amount of time machines sit out-of-sync. But more on that in another post another day!

- KB

Have a good holiday season everyone!

2009 has been a great year on many fronts. I am sure 2010 will be better.

- KB

With every release, Thunderbird has become better and better as a personal email client. But over the last few releases, it definitely feels like as its getting better at personal email handling its getting worse for mailing list traffic.

I am on a few lists ( 32 to be exact ) and a couple of them get loads of emails ( 150+ a day ), but even so why is thunderbird at almost 90% cpu usage with over 800MiB of ram used ? If I take away the mailing lists account and only use it for my personal emails things are fine. Thunderbird will almost never corss 12 - 13% cpu usage and rarely ever cross 200MiB ram usage. Its not that I dont get personal emails; I do get a fair bit - not including spam - about 400/day.

Someone please fix it!

- KB

Just to change things around a bit, new blog skin!

- KB

Posting this, since a fair few people seem to have asked about it on the internet but no-one really had an answer.

If you are looking to add a optical media drive ( CDRom / DVDRom etc), just keep in mind that its a conventional atapi slimline drive that fits into the slot in the front. Its literally a case of flipping down the front cover, and sliding in a drive. I 'borrowed' a dvd drive from an old laptop, removed the mounting around the drive and pop'd it into the running machine. OpenFirmware was able to detect it right away and quite happily went through a 'boot cdrom:' process.

- KB

SIGSECTOOMUCH : The state of play when you over secure a machine to the extent that you can yourself no longer get onto the machine.

- KB

I use a proxy to get onto irc, which is good since I can get online from anywhere and do so over mostly a secure link. Another advantage of doing it this way is that my nickname is always online on irc and people can leave msgs or followup on conversations even when I am offline. Which has been a great way to stay in touch ( slowly ) with people who are in completely different time zones.

This saturday morning the machine I used for my irc proxy lost all internet access - which means 'I' went offline. Only having restored the link a few hours back I now realise that there were a few ongoing conversations that seem to have either finished in the time period or just gone away. Will try and catch up with people over the next few days for details, but if anyone is waiting on input from me - please ping again!

- KB

So, initially it was just a bunch of websites, it then became a part of Andover.net - soon to be merged with valinux. That went on for a bit before they changed over to being 'Open Source Developers Network (OSDN)', to only change over to being called ' Open Source Technology Group (OSTG)' soon after. Then it was 'Sourceforge, Inc and' now they are called 'GeekNet Inc.'

And this is the whole stack of slashdot.org freshmeat.net sourceforge.net etc.

Remember any other names they have been called over the years ?

- KB

At approx 16:45 the Coreix Data Center suffered a power loss, UPS took over the load for short while, however the Generators did not come online in time and it took till 7:35 for power to be restored. There are details posted on the coreix status page at http://status.coreix.net/

While power did get restored to the entire DC, only 1 of the machines came back online. All the rest have needed some level of manual intervention! I'm working with the support people ( who are a really good and effective bunch of guys ) to get the other machines online.

Services affected are:

• RPMForge svn repo
• RPMForge master mirror
• RPMForge mailing lists
• Karan.org Build services
• Karan.org testing services
• CentOS.org ipv6 test / qa setups
• CentOS.org Package and Automated testing development machine

Once all services are restored, I'll update this blog post with details. And apologies for this completely unplanned and avoidable outage.

Update: as of 23:45 26th Nov, all services are now restored.

In the overall scheme of things, I wonder if its Java that helped Linux get some level of traction in the large enterprise markets, of if its the other way around : Linux helped Java get into these shops and stay there.

Even though I've never been a big Java fan myself, its hard to argue against the fact that there is some level of hand and glove thing going on with Java and Linux in the enterprise setups.

- KB

If someone was to ask you why you run CentOS, what would your top 3 reasons be ?

- KB

I've been trying to automate the CentOS-5 updates system as much as possible - however, one thing that the system cant do at this time is be smart about packages that only exist on one arch and not on the others. eg. KVM on x86_64 only.

So for the time being, there will be updates announced for these package on all arch's. Ofcourse, this will be only for packages that are compatible, so in many cases only the .src.rpm will be announced!

I shall try and fix this soon'ish. But in the mean time, be sensible and look at the announcements completely if you do get them! The fix would be that the system does the right thing : only announce packages into the arch they are built for. So KVM will not get an announcement for i386, but only on x86_64.

- KB

A few of us are going to be getting together for drinks on the Tuesday 29th Sep 2009, everyone is welcome to come along. I'll get there for about 18:15hrs and plan on being around till about 20:00 - Depending on how many people are around and what the feeling is - we might nip around to Ragam ( mostly authentic South Indian food ), a few doors down.

There will be a demo for CentOS-5.4 as well! If there is anything specific you might want to see, let me know a bit in advance.

The full address is :

King & Queens,
1 Foley St,
London,
W1W 6DL‎

Here is a Google Street view of the place.

If you email me, I'll get back with my mobile number - although it should be mostly easy to spot the 'CentOS Guys'.

Hope to see you there, then!

If you ever want to get to the grub menu while using pygrub - there is a really simple way of doing that. Just add :

bootargs="-i"


into the /etc/xen/{domU config file}. And the next time you start the VM, it will bring up the grub menu. Quite handy when you need to recover the root passwords or to be able to use a different init script as a one off.

Essentially, that bootargs will pass in parameters to pygrub during the domU boot phase. To get a list of all the possible options you can pass in with bootargs, try this: pygrub --help. You should get output like this :

# pygrub --help
Usage: /usr/bin/pygrub [-q|--quiet] [-i|--interactive] [--output=] [--kernel=] [--ramdisk=] [--args=] [--entry=] image

- KB

By default md-raid will limit its operations to 200000k/sec - which is plenty for most desktop and 2 - 3 disk machines, but when you have more than 3 - 4 disks and there is enough cpu and i/o bandwith available, it makes sense to increase that limit.

to find out what the limit on your machine is :

$ cat /proc/sys/dev/raid/speed_limit_max
200000


Setting it to something higher :

echo 500000 >/proc/sys/dev/raid/speed_limit_max


So whats a good speed to set ? That depends on what it is that you are looking to achieve, eg: if you dont mind max'ing out your hardware platform ( cpu / io / disks ) then set it to something very high, like 2000000. On the other hand, if you want to keep some cpu and io resources back from md-raid ( like when doing a raid-1 rebuild on a production machine ) you might want to actually lower it down a bit.

The three main issues to consider when working out a raid max speed :

• Number of disks: for aggressive sync's I tend to go with 50 - 70 M/sec per disk, so on a 4 disk system the 200000 number is mostly ok, but on a 8 or 12 disk system I'd look to make that much higher. For conservative rates, or when machine resources are required elsewhere as well, 10 - 12M/sec per disk.
• Interface: What interface you use is also going to make a big difference. So consider the implications of using IDE / SATA / SCSI.
• CPU: the raid jobs,specially when run for large disks or over many disks, will be fairly cpu intensive. So workout what sort of speeds work best for the loads you have. Usually this isnt something one needs to consider unless the machine is already under load or expected to be used during the raid operation. Over the last few years, AMD's have been able to deliver slightly better throughputs than Intel's - but in the recent past, much of that has changed. So dont just go with what you hear or opinions around the place : test it yourself.

Finally, while speed_limit_max sets the rates md-raid is going to try and reach, there is the speed_limit_min - which is the rate that md-raid will try and maintain as an 'atleast' limit. I tend to be a bit more conservative about that number. Usually aiming for 25 - 30 M/sec per disk for a very aggressive run. Or 10 - 15 M/sec for a more toned down run. If you have i/o intensive ops running on the machine you might need to reduce this even further - however the default of 1M/s for the whole machine, irrespective of disk count is something I feel too low for a modern machine.

I find many people are unaware of this small detail, hopefully this post will help.

Got a WDC WD50 disk, and a usb caddy, to use as a portable backup / mass storage unit. Got it setup, powered it up. First reaction : wow this thing is quiet. In a nice way.

The disk its replacing is a 3 year old Hitachi 60GB. While the Hitachi seems to have actually been a bit faster than this, quiet is not something that anyone would have ever associated with that disk. Plugging into the laptop usually meant someone leaning over from the table next to mine and wondering if all was well with the laptop!

Not too concerned about the fact that this disk seems to max out at 62MB/sec whereas the Hitachi seemed to get upto 70MB/sec, since this disk is going to spend its life on the other end of a USB line.

- KB

I wonder if the last 6 - 7 years of Linux ( almost 5 of that exclusively Linux ) has taken away some of the memories of pain that we all went through in the Windows years. Just the other day I saw someone going through an hour long session of suffering with Windows Vista, trying to get the wireless working. This is in his own office, somewhere he goes everyday, on a laptop he has had for three months. And it sort of struck me how mad things used to be, and brought back a few memories - which I then went onto share with him and his coworkers. Not amused they were. Even more so, not amused when my laptop just worked.

So why am I blogging this ? Well, in Jan this year I had to write down things that I had done over the years - a sort of technologies that I am working on and something I remember from the past, all a part of some paperwork thing I had to get sorted out with the Home Office here in the UK. I came across that bit of paper again today - its quite funny how things from the last 3 - 4 years are very clear and vivid ( as they would be ), and also quite diverse. Then there is this dark period for a few years, haunted by 'sqlserver' and 'visualstudio'. That is *all* I could put down for that 3 year period, and again before that in the last 90's there is a nice long list of very interesting things that I was doing - almost all on Linux.

One bit of info that was on my profile that I had submitted to the Home Office in 2001 was 'I enjoy programming'. Which was true, a the time. The years from 2002 to 2004 of the intensive coding for the windows platform changed much of that - I gave up on coding completely for a long time. Its only when I got into python later in 2005 that I started to get back into this, and really started enjoying it again. Even assembly, something I did lots of in the mid 90's, was something I was again doing in 2006. Have again given up on it, there are way too many things going on to really be able to sit back and spend the days required to write meaningful code in assembly these days.

Every now and again I come across someone struggling in Windows with a task that really should be trivial for an OS and platform that has been around for so long, been developed for and on, for so long and something that has such a wide user base. And with the horror memories it brings back it also does one more thing for me. It makes me realise that while Linux might not be perfect and while it might be something that needs a bit of tech skill and a moderately wired mindset to do complex tasks in Linux - for the people it works, it works. And the number of people it works for, is always increasing.

go Linux!

- KB

I will be at the cPanel Conference in the first week of October this year. Hope to meet lots of CentOS Users there! CentOS has a corner in the exhibitor area, and helping out over there will be Garry Dale and maybe Johnny would be able to come down as well.

On the 6th at 1:30pm I'll be doing a short 30 min talk on 'Rapid deployment & provisioning' for CentOS. Depending on how it works out for time, I'll try and get a demo / walk though as well for some of the common recommended methodologies. If you manage 2 or more machines even if they are Virtual Machines, there should be something in there for you.

If you are based in the area, but unable to make it for the conference, get in touch with me anyway - I plan on being in the city for a few days after, so we could still meet up.

- KB

I've been looking into the idea of collaborative mind mapping. Think wiki, but in a mind map. The aim being to create a knowledge pool around some very specific areas, that multiple people could contribute into. Specially areas where there might be a lot of content overlap in different zones or a workflow thats easy to define.

Early examples ( and the ones I want to start working with ) could include :

• Post-compromise content and system audit
• System lockdown for various roles, like home-server or home-nas
• setting up a uPnP server, including storage and performance considerations
• Two node, heartbeat based failover cluster for mysql

I guess its easy to see the theme here, all the tasks are almost things that could be reduced to a howto. I keep thinking there must be better ways to handle this at a small to medium sized team level than using a wiki. Say 3 to 7 active contributors with a few dozen occasional drive-by's - and general knowledge levels of each contributor being drastically different from one another.

One thing that has worked really well in the past, for me personally, is doing these based on and around an issue tracker like TicGit. Before you dismiss that idea completely, think about it. However, that does not scale to > 1 person very well. And its a bit of a pain since the only way to organise those down is into a FAQ or a list-of-things kind of way. I hate both those approaches to organisation.

Mind maps are a logical next step after the step based issue trackers and wiki - however, finding one that works well in a browser, and can have nodes outside the immediate map isnt easy. In a nutshell : I've not found any software that lets me do that. I know xmind and Free Mind both have some ways to share the maps. But neither is optimal for mass public consumption. Pimki seems to have potential, but is too much single person centric. Wikka on the other hand, seems to set itself up as the perfect candidate - integrated wiki and mind mapping. But it needs a java plugin and the content it creates seems to not be openjdk friendly.

Are there any other options out there worth considering ?

- KB

So, the 20th Sept 2009 build for Thunderbird enabled Faceted search, by default. Quite nice. However, when you have nearly 17 GB of email- it can take a long long time to actually build the sqlite index's it needs to carry out this sort of search.

Late last night, my shreader installer updated to Gecko/20090910 build, and since then ( about 18 hrs back ) its been building the index. Which is itself upto 2 GB now. Lets see when it finishes. Although, to be honest I dont really care that much about a few gigs worth of data. Disk isnt that hard to come by and email is relatively important in the things that one uses on a day to day basis. The thing that concerns me is how and what the performance of something like this might end up being. A few initial peeps, and it looks and works great. However, will it still stay this cool with a many-gig index ?

Having said that - let there be no doubt about it : This is a *great* feature. Just the sort of thing that makes Thunderbird stand out miles from the other email clients out there. And having had it for just a few hours today, and even with a partial index - its already helping me get more productive and resolve / identify relevant threads faster!

There are some more details here : http://www.visophyte.org/blog/2009/09/03/sos-your-facet-faceted-global-search-for-mozilla-thunderbird/ and if you want to look at specifics of what faceted search is, look here http://en.wikipedia.org/wiki/Faceted_search

- KB

This via Hampus : HP seems to now have a support pack, specific to CentOS for their ProLiant servers. Look at : ProLiant Support Pack for CentOS 5 (i386 and x86_64) as an example.

That's one form of endorsement!

- KB

There will be about an hours downtime, starting at 1800 BST ( 1700 UTC ), for the entire karan.org build system and repository infrastructure. The public repo will not be affected.

I am hoping to achieve :

• Upgrade the local network within the rack to gigabit for both front end ( internet side ) and the private end ( 10.x.x.x ) network.
• Upgrading the storage box ( adding 2x 1 TiB disks ).
• Adding a 140GB disk into 'surya'.

Given the current financial setup, this might be the last upgrade for a while.

I still need to find an economical replacement for the VM/Code testing machine : which, after having been exceptionally unreliable, finally gave up the ghost a few days back. So if anyone knows or has a 1U machine, with 6 - 8 GB of ram, moderately fast cpu's ( old old opterons like 285's or Xeons like 3.0Ghz will do just fine ), let me know!

And I'd like to thank Peter Zwernia at Firmix Software gmbh ( http://www.firmix.at ), Austria for sponsoring the disk upgrade for Surya.

- KB

In quite a few situations its preferred to have ssh keys dedicated for a service or a specific role. Eg. a key to use for home / fun stuff and another one to use for Work things, and another one for Version Control access etc. Creating the keys is simple, just use

ssh-keygen -t rsa -f ~/.ssh/id_rsa.work -C "Key for Word stuff"

Use different file names for each key. Lets assume that there are 2 keys, ~/.ssh/id_rsa.work and ~/.ssh/id_rsa.misc . The simple way of making sure each of the keys works all the time is to now create config file for ssh:

touch ~/.ssh/config
chmod 600 ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.work" >> ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.misc" >> ~/.ssh/config

This would make sure that both the keys are always used whenever ssh makes a connection. However, ssh config lets you get down to a much finer level of control on keys and other per-connection setups. And I recommend, if you are able to, to use a key selection based on the Hostname. My ~/.ssh/config looks like this :

Host *.home.lan
  IdentityFile ~/.ssh/id_dsa.home
  User kbsingh

Host *.vpn
  IdentityFile ~/.ssh/id_rsa.work
  User karanbir
  Port 44787

Host *.d0.karan.org
  IdentityFile ~/.ssh/id_rsa.d0
  User admin
  Port 21871

Ofcourse, if I am connecting to a remote host that does not match any of these selections, ssh will default back to checking for and using the 'usual' key, ~/.ssh/id_dsa or ~/.ssh/id_rsa