Karanbir Singh - Thinkability

I am often surprised by the sort of questions asked in the forums or on irc around open source projects - it just feels as if people are going out of their way to inflict pain and suffering upon themselves by trying to find the most awkward and most complicated way to do things. So how can we better help these people ? We dont need to save them or anything as drastic like that, its just a case of being able to show or explain that there might be a better way.

The first thing that I've started now doing, when asked a strange question is ask the person 'What are you really trying to achieve?'. You might be amazed how many times the answer has nothing to do with the question being asked. Try to establish what the end goal is, and in many cases its clear that the person has been lead astray by random posts on the internet, some of which are perfectly fine in their own context, but can be quite a kludge outside that context.

Establishing, clearly what the goal is before advice or opinion is thrown at people will always result in a better overall experience. And to the people spending their time in the irc channels, web forums and mailing lists helping others out : must respect. You guys are the ones making the idea of Communities and Open Source work.

- KB

Looks like I will need to get a Visa again to visit Belgium for Fosdem 2012. This is starting to get a bit irritating now, six times I've been to Fosdem and every time they have asked me to come in for an interview before they give me a visa; once again ? Surely by now it should be possible to get onto the visa-by-post process.

- KB

So HP is getting in on the high density compute node game. And looking at http://www.theregister.co.uk/2011/11/01/hp_redstone_calxeda_servers/ they are doing it with a twist : use ARM chips. That being beside the point, I think these would make for fantastic machines where individual thread performance isnt that much of a deal, but being able to get lots of threads going is. PostGres-XC as an example. Rails apps are another great example where a cluster-in-a-box can be an attractive option.

Might be worth noting that while its getting most of its attention based on the ARM buzzword, they seem to be saying that Intel as well as AMD cpus might make it into these machines as well. So while OS options might be somewhat limited at the moment, that problem should solve itself fairly easily.

On the other hand, getting 128 cores into a 2U AMD machine these days isnt that hard - its a slightly different set of mechanics to this cluster-in-a-box solution, but in many cases solves similar problems - and you get the advantage of a single hyper visor, should one need it. Then there are also near-commodity like specialist kit which goto 512 Cores in a 2U machine ( x86 example: http://www.qsscit.com/en/01_product/02_detail.php?mid=27&sid=95&id=135&qs=56 )

I've now had the SSD in my laptop for about 10 days. Its made a massive difference to the way I work.

Its striking as to how much of a difference having this extra performance in the laptop would make. In march I upgraded the memory on this laptop from 2GB to 8GB - which also made a massive difference, specially since I almost never reboot the device and the filesystem cache get very good at handling just the right kind of stuff - but what kills them is my email ( ~ 30 gb ) and VMs ( upto 5 running at any given time ). Having the SSD now means that I no longer need to drop back to 10 seconds for jedit startup after I've been running a couple of VMs.

One thing that hasn't gone quite to expectation is the battery life. The HP 2540p had ~ 4 hrs or so, doing what I do, when I got it new. That had dropped to just over 3 hrs with the 250gb sata disk in. With the SSD its now gone to 2 hrs ~30 min or so. Initially that felt quite strange, I was expecting it to go in the other direction. And while I havent been able to put a finger on exactly what this is, it seems like there are 2 interesting side effects from the SSD upgrade.

1) The four cores on this i5 laptop now run at full speed ( 2.53 Ghz ) a lot more often than they did in the past, trending this over the last 48 hrs and its averaged 1.87Ghz; Not sure what it was earlier but the cpu governors used to stay blue a lot more than they do now.

2) Heat. The cooling fan is on a lot more, and the heat vent seems a lot warmer than it ever did in the past. This might be due to the cpus running a lot faster, a lot more. The disk itself does not 'seem' to be any warmer. The bottom left side of the laptop which houses the disk feels cool.

The big win of-course, is performance of everything. Almost every app just starts in place ( even eclipse! ). Doing a search in large code projects is instantaneous. Git operations are visibly quicker. Even using svn isn't nearly as boring as it used to be, if I can stop adding -a to all my svn commits it would not get in my way.

The only thing that isn't quite as quick as it needs to be on this machine now is the graphics interface ( intel HD ).

Also worth keeping in mind is that use CentOS-6 for the SSD hosted content, and make sure you have 'discard' enabled as a mount option.

- KB

We are pleased to announce the immediate availability of CentOS-6.0 for 
i386 and x86_64 Architectures.

CentOS-6.0 is based on the upstream release EL 6.0 and includes packages 
from all variants. All upstream repositories have been combined into 
one, to make it easier for end users to work with.

There are some important changes to this release compared with the 
previous versions of CentOS and we highly recommend reading this 
announcement along with the Release Notes at 
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS6.0

There are no CD images being released with CentOS-6, however we have 
some CD variants in the pipeline. Details for these are mentioned below.

Since upstream has a 6.1 version already released, we will be using a 
Continous Release repository for 6.0 to bring all 6.1 and post 6.1 
security updates to all 6.0 users, till such time as CentOS-6.1 is 
released itself. There will be more details about this posted within the 
next 48 hours.

+++++++++++++++++++++++
Upgrading from CentOS-4 or CentOS-5:

We recommend everyone run through a reinstall rather than attempt an 
inplace upgrade from CentOS-4 or CentOS-5

+++++++++++++++++++++++
LiveCD and LiveDVD

LiveCDs and LiveDVDs for i386 and x86_64 will be released within the 
next few days. These will bring in the ability to directly install from 
the livemedia.

+++++++++++++++++++++++
Minimal Install CD

We have also created a minimal install CD, that would bring up a base 
machine with just enough content to have a usable platform. This CD 
image will be released in the next few days.

+++++++++++++++++++++++
The LightWeightServer (LWS) CD

In order to bring back the CentOS-4 Server CD style single iso image, we 
are creating a LWS varient of the main distro. Details for this will be 
posted in the next few days with release happening after the live media 
and the minimal cd editions.

+++++++++++++++++++++++
Downloading CentOS-6.0 for new installs:

When possible, consider using torrents to run the downloads. In most 
cases you will find its also the fastest means to download the distro. 
There are currently over a thousand  people seeding CentOS-6 and it's 
possible to get upto 100mbps downloads via these torrents.

Torrent files for the DVD's are avilable at :
http://mirror.centos.org/centos/6.0/isos/i386/CentOS-6.0-i386-bin-DVD.torrent
http://mirror.centos.org/centos/6.0/isos/x86_64/CentOS-6.0-x86_64-bin-DVD.torrent

You can also use a mirror close to you :
http://www.centos.org/modules/tinycontent/index.php?id=30

Most mirrors will allow direct DVD downloads over http, ftp and rsync.

Please keep in mind that not all mirrors are currently updated, some 
might take upto another 24 hours before they have all the content.

+++++++++++++++++++++++
sha1sum for the CentOS-6.0 ISOS:

i386:
fcf49e875cd4494f2af68cf257ab9e93523c9427  CentOS-6.0-i386-bin-DVD.iso
862815623d2e7990207dd78a281837c7eb719e83  CentOS-6.0-i386-netinstall.iso

x86_64:
9de87b0c696ebd72b952edb4cc06c24cbdc37d81  CentOS-6.0-x86_64-bin-DVD1.iso
5e3834621f11fbcca78cf7d70625c647045f45f5  CentOS-6.0-x86_64-bin-DVD2.iso
23f9e606cbcbd52d2e5df3716a85cdde336f7bfe  CentOS-6.0-x86_64-netinstall.iso

+++++++++++++++++++++++
Sources and Debuginfo packages:

SRPMS and debuginfo packages are still making their way to the CentOS 
mirrors and should be available within the next 24 to 48 hours. We are 
prioritising the centos modified packages.

+++++++++++++++++++++++
Getting Help:

The best place to start when looking for help with CentOS is at the wiki 
( http://wiki.centos.org/GettingHelp ) which lists various options and 
communities who might be able to help. If you think there is a bug in 
the system, do report it at http://bugs.centos.org/ - but keep in mind 
that the bugs system is *not* a support mechanism.

+++++++++++++++++++++++
Contributing and joining the project:

We are always looking for people to join and help with various things in 
the project. If you are keen to help out a good place to start is the 
wiki page at http://wiki.centos.org/Contribute . If you have questions 
or a specific area you would like to contribute towards that is not 
covered on that page, feel free to drop in on 
#centos-devel at irc.freenode.net for a chat or email the centos-devel list 
(http://lists.centos.org).

+++++++++++++++++++++++
Thanks to everyone who contributed towards making 6.0


Enjoy!

Earlier in the day today Red Hat released RHEL 6.1 ( http://www.redhat.com/about/news/prarchive/2011/Red-Hat-Delivers-Red-Hat-Enterprise-Linux-6-1 ). Congratulations to them, it looks like a great release with lots of cool new stuff in there.

Most people will want to know how this impacts CentOS and the CentOS-6 plans. We are, at this time, on course to deliver CentOS-6 within the next couple of weeks. We will carry on with those plans as is, and deliver a 6.0 release and then goto work on 6.1. I am fairly confident that we can get to a 6.1 release within a few weeks of the 6.0 set being finalised. Partially due to the automation and the testing process's being put into place to handle the entire CentOS-6 branch.

If you would like to follow progress of the QA and Release team, you are welcome to drop in at http://qaweb.dev.centos.org/qa/ . Jeff has been keeping the calendar as updated as possible and is doing a good job of keeping a fair bit of information flowing through there. At some point next week, we will try and get some dates in place for the 6.1 process as well.

So what happens if 5.7 comes along in the mean time ? Well, the CentOS-5 process is now completely disconnected from the CentOS-6 one, and a 5.7 release should have no impact on the progress of CentOS-6 and the release cycles. We have also been working on plans for an opt-in, by design process that would allow users to get early access to packages being built for a point release. More details on that soon.

Comments and feedback are always welcome!

- KB

Trying to sign a bunch of rpms usually means having to type in your password for the gpg key multiple times, once for each rpm. However, you can avoid doing that with this :

rpm --resign `find . -name *.rpm`

That will only prompt you once for the key passphrase, and sign all the packages it finds under that directory.

- KB

Hi Guys,

CentOS 5.6 is now out and available from all mirrors. In the next few hours, all yum operations will switch from 5.5 to 5.6 ( for people who run the default yum configs ).

All the Release Details : http://lists.centos.org/pipermail/centos-announce/2011-April/017282.html and make sure you skim through the Release Notes at http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.6

Enjoy!

- KB

What was the issue:

There was an invalid rpm ( eclipse-ecj ) in the x86_64/os tree. Premature EOF in rpm payload.

Report:

Early morning on the 7th James Hogarth, BSkyB Entertainment reported to the QA team that there was an issue with the eclipse-ecj package shipped in the CentOS-5.6/x86_64 os tree. Stephen Walsh and Manuel Wolfshant tracked the issue back to the primary seed machine and confirmed the issue was present in not only the os tree, but also on CD3 and DVD1 of the x86_64 distribution.

Reason for this issue:

The CentOS distribution is composed on the buildservices, but then transferred over to the distro-build machines where the installer is built. There are automated tests that run at both of these locations. The rpm content tests ( rpm -K ) for md5's as well as gpg key. The distro had passed this test at both locations. The output from the distro-build machines is the actual package tree, the installer code, isos for cd's and dvd. This is transferred using rsync to the staging machine from where we start the release process ( initially to QAMachines when in qa mode; or into the mirror.centos.org network when in release mode ). There are no tests done at the staging machine. Packages are transferred one at a time, rather than as a whole tree ( mostly for legacy reasons ). It seems that the transfer for eclipse-ecj did not complete ( driven by the fact that its OK on one side and not on the other ).

Impact:

We had to rebuild the torrents, DVD isos, CD isos and update the CentOS-5.6/x86_64 distribution. There was no impact to CentOS-5.6/i386.

What did we do to fix things:

To address this issue, we had to issue a new set of ISOs, and since the package content was changing, rebuild metadata. Which in turn needed a complete rebuild of the ISOS ( but not the install tree ). Over the course of the morning, Fabian and Manuel were able to test the new tree, and our automated tests ran through for the ISOs

There was also a lot of rollback work that needed to be done, including handling the torrent tracker, issuing new torrents, making sure the mirror network etc. Much of which is done manually; and the main reason things took almost 18 hrs to resolve.

Steps taken to ensure this problem does not happen again:

I've now moved a large number of tests to the staging machine as well, including the rpm tests. This adds an additional 3 hours to the process, but its a worthwhile safeguard.

I hope this helps clear up things. Also, md5sum and sha1sum for ISOs as well as torrent files are published along with the torrent files and available on all mirrors. They will also be mentioned in the actual CentOS-5.6 Release Announcement. Everyone should check to make sure they get the right ones.

If you get the new .torrent files and drop in into the same place as the older ones did, you should see most of your data be reused ( 30% on the DVD and 86% on the CD's ).

Couple of things that should answer most questions that people are asking about the 5.6 Release status:

• TimeLine: Stuff is syncing around, it could be upto another day before you see packages, tree and isos on your local mirrors.
• Updates: Yes, it includes all updates released into the distro and most updates from that point on ( exceptions: java packages, subversion and openoffice.org ). These 3 package updates will come through in the next 24 hrs, but due to the churn in the mirrors, might be a few more hours before they are publicly visible on a mirror near you.
• LiveCD: Yes, the LiveCD's for i386 and x86_64 are also included
• Plus Kernels: Yes, the plus kernels are included in the release with no delay.
• Torrents: Torrents will be published as soon as the release is announced. This is to work around the situation where people will download the torrent and use yum to hit mirrors that are not completely updated as yet. Causing non-trivial issues ( as they end up installing 5.5 pkgs that then need to be updated again ). So once the bit-flip happens, we use the mirror status monitors to workout the state-of-external-mirrors, once there is a reasonable number in sync, we will release torrents, publish md5sum and sha sum's for the isos and announce the release.
• Update Announcements: There will be individual announcements for all updates released since EL5.6 was released upstream. There will be no announcements for packages rolled into the distro. So in a nutshell: it will be the same as its always been for CentOS-5.
• Will you need to change anything on your machine to get 5.6 ? : No, once its released and publicly available everywhere a regular 'yum update' will move your machine from centos-5.x to 5.6

Please keep in mind that till the release is announced ( keep an eye on http://lists.centos.org/mailman/listinfo/centos-announce, things can still change.

And finally : If you dont run a public mirror listed as a centos mirror, please do not rsync from the centos.org machines for the next 3 to 5 days. Use a large pipe mirror near you instead.

Update: CentOS-5.6 is now released

Firstly, CentOS is super popular out there with the ruby guys and there are quite a few ruby based apps deployed on CentOS these days.

Secondly, every man - his dog - and kitchen sink seem to be building ruby rpms. Not apps or gems or anything, but Ruby itself. There must be over a dozen repos and over a few hundred private rpm builds floating around.

What needs to happen for these people to merge efforts ? CentOS is more than capable of hosting out-of-distro builds, either in Extras or CentOSPlus and there is plenty of precedence in the ecosystem to use namespace overloading to handle multiple build variants. Its not pretty, but it works. And using a sensible set of defaults also means that its mostly new-user friendly.

- KB

People who have been following the centos-devel list would have noticed a ressurection of the CentOS website Ver2 effort[1]. Which is a great sign, however, the intensity and the enthusiasm shown is about at par with the efforts that went into creating the original plan - a few years back. And the one thing that stands out from that effort was that almost always, we had communication issues between the various people doing the various tasks, there was a clear missing internal-customer, there were parallel efforts for some tasks where the teams didnt know someone else was also doing the same thing, at the same time. This time, it looks like we might be heading back into the same sort of a situation. And I say that mostly since one can already see people trying to solve problems we might not have, as well as trying to expand on the actual deliveryable proposal. Feature creep, from the outside in. Dont get me wrong, I am not being negative about whats going on, its great! The fact that this effort is getting off the ground is very good news, One thing that CentOS really does ned is a better, more functional and easy to navigate website.

Thinking about the task and communication side of things, I was wondering if perhaps we could adopt an agile sort of a development format. Are there any other open source distributed communities who have been able to adopt such a process ? I am sure there must be some. Perhaps we should look at how they did the coordination and how they worked with the idea of the 'development sprint'. Working on a delivery dates by fixed date is'nt going to work out too well, after all this is still people contributing their spare time and a best-effort situation. But on the other hand, we still need something in place that allows the other developers, admin and design people to keep track of each individual task thread. So the thing is, if there are other projects that have done something o this nature, what tools were used ?

I know that mingle ( http://www.thoughtworks-studios.com/mingle-agile-project-management ) has a license that allows us to use it for open source projects without any cost. Also, there is an OpenAtrium install within the .centos.org space, is that worth looking into ? I suspect getting OpenAtrium into a format we like, might need some php side development. And starting a project by developing on the management tool isnt the best way forward. There is also Pivotal Tracker ( http://www.pivotaltracker.com/ ) which is a hosted service. I know they recently announced that they were no longer going to be totally free anymore; but from what I can tell all public projects are still free to host, and memebers are not charged anything to join those projects.

Recently there has also been talk about AgileFant ( http://www.agilefant.org/ ), and there is xplanner ( http://www.xplanner.org/ ). Neither of which I've used. Does anyone have opinions ? Thoughts ?

- KB

[1] Maybe it should be called Website Ver2.1!

A while back, Red Hat started pushing out early-releases for some updates. I'm guessing it was due to customers and users asking for eary access to trivial fix's. These early releases were put into a repository ( channel for you RHN users! ) called fastrack, with the condition being that these packages had not been through the complete or proper QA process. Users were advised at the time they should be selective in picking packages from this repo, only to solve immediate issue and consider these packages to not really be 'recommended' by Red Hat. So, thats the background to the 'fastrack' entity.

These packages were pushed out at various intervals during the life of a point release, and then either rolled in or discarded completely when the next point release came around. Eg. Package-1.1-3 released into fastrack during the life of RHEL-5.2 might make it into the main distro when 5.3 is released. On the other hand, its also possible that by the time 5.3 came around this Package-1.1-3 had been updated to Package-1.1-4, making the Package-1.1-3 redundant ( unless one was on the EUS or the z-stream etc platform, but lets not get sidetracked here ).

That nicely brings me to the point of considering update and upgrade notifications. Red Hat publish release info at various places, and there are different mechanisms for one to get to those notifications. One can get them via email, on the web, within RHN, through oval notifications in some cases etc. But the interesting point is - announcements are not re-issued when packages from fastrack move into the main distro. There is a nice silent move from one to the other. So if you are not tracking announcements within the fastrack repo, you might see something sneak up on you that has no origin notification issued ( or so it would seem to you ).

Within CentOS we dont have this issue. Mainly since we don't build the fastrack rpms regularly. There are quite a few reasons for that, I'll blog about why we did'nt and how we are going to try and solve that problem with 5.6 and 6.0 going forward. But the main thing is that we dont build those fastrack rpms at this time and have'nt done so in the past. Which creates an semi-significant difference in the announcement stream, in that we will announce those packages which have moved from fastrack to the main distro - but we will announce them when they hit the distro. But there will be no announcements within CentOS for packages that are released upstream into fastrack and then never make it into the main distro. I havent done any work to quantify this, but there are examples of when this has happened in the past.

So, if you are going to compare announcements. Keep this one thing in mind. Also, to all the people scraping the centos-announce mailing list to get update details into your automation, please consider using an alternative route. There are quite a few other options, use one of these alternatives.

- KB

Trying to pull from a remote repo, just ran into this strange error being reported by gitosis.

ERROR:gitosis.serve.main:Arguments to command look dangerous


And in typical gitosis ways, its down to trawling the sources to workout what the problem might be. In this case it turns out that I had a / at the end of the reponame. I was trying to :

git clone ssh://git@remotehost/blah/repo/


Which made gitosis quite angry, it did not like that / at the end of the clone request. Changing that to 'ssh://git@remotehost/blah/repo' makes things work again.

- KB

Should open source projects, not focused on source or code at all but more on a service or a resource, really be called 'open source' projects ? I feel there might be a need to define the idea of an 'open service' or an 'open resource' project.

- KB

December 2010 was a crazy month. Extremely hectic, exceptionally busy. But I spent all that time doing stuff that was neither desirable nor pleasant. We had a bereavement in the family followed by me being put into jury duty to be interrupted by a fairly serious illness. 99% of the holiday season was spent in bed, under the covers, away from people I care about and away from the computers!

I did get 3 days into the jury duty before I fell ill. And it was a mostly wasted time experience; after being put into one case only to have it dismissed after 2 days and then spending a whole day hanging around in the terrible waiting rooms for a while day. If they make the experience a bit more comfortable and slightly more productive, it might be something more people find acceptable use of their time. But unless they do that - its all mostly just a waste of time.

But dec/2010 wasent all down and out. I had two very interesting phone conversations, both of which I shall blog about in the next few days. And I got lots of rest! lots and lots. Just would have preferred if it had been while I wasent under bed rest orders with massive antibiotics doses being shoveled into my system and a total beer deprivation. Guess I now need to plan out a get-some-beer-in agenda!

So, lets see what 2011 has in store.

During most of December I was offline, and so there are some serious serious mail queues and postponed work / open source stuff. I'm trying as hard as I can to catch up, but it might be a few days before I am there.

- KB

I've never been big on new years resolutions, but this year I have one. To Blog. Started off initially as a 'I should blog more often', then I realised that I didnt really blog much at all through 2009 and 2010. So the aim is to change that : 2011 will be a good year for blogging. For me anyway.

- KB

If you insert data using server side metadata like now(), think about the implications in a multi-machine database spread specially when you have load.

These sort of problems are somewhat reduced when using rowlevel / binary replication rather than just sql replay. But think about the implications this is going to have across the entire system, the application and specially think about how you expect data in those fields to be used in the future.

- KB

One fact of life with the sysadmin lifestyle is that most issues are task and event driven rather than project driven. For me, over the last 6 months, the split has been about 80% issue driven and 20% project driven. And most tasks tend to be short, event or incident driven. Short in this case might be 10 minute tasks or even 3 day long tasks, but not the sort of things one would handle in project scenarios : 3 to 4 month long tasks, split into multiple sprints.

With such a high rate of churn in the ToDo lists, one tends to often overlook the automate angle in return for short term win. On the other hand, its also easy to slip into a state where over aggressive automation leads to a case of seriously low on-the-day productivity. The way I've tried to get pragmatic about this is to adopt a policy of automating at the second instance, monitoring at the first.

So short tasks are fine to execute manually the first time they come up - but ensure there is monitoring around it. The second time a task is repeated, automate if possible. And if the same or similar issue comes up a third time, and its still not automated - reconsider the task. Either break it up into smaller chunks so it can be automated or reconsider what and how you are doing. Its only in exceptional corner cases that you will find automation fail, even for repeated tasks.

In highly repetitive environments, using a config and change management system should never be optional. Not only does one get a large portion of the admin-task automation win, but you also get an audit trail, state reproducibility and speed in deployment.

- KB

This question comes up often : how does one get started in the world of Linux sysadmin. And to be honest, I dont think there is a clear answer to that. The state of Linux certification is not ideal. There are a few courses one might get onto, like the RHCE. But doing those without any background info will leave you unable to really get the full benefit since they all makes assumptions : that you are already aware of the basics.

One way to start off in sysadmin, is to grab a good book about the topic. Then install and run through the various install options and get a couple of VM's setup with your favourite Linux distro. Start with one distro and stay with that one distro for the first few months atleast. But its also important that once you are familiar with the basics, you do move onto some other distro and see how things work there. One cant really put down into words or ever really express easily that the bottom line is Linux. No matter what distro you move onto, the only thing different would be how things are laid out and general communication around the platform - the main OS is Linux, and that never changes.

While you work with a new platform, its quite important that you really get involved with it and an easy way to do that is to look at what you are doing now, whatever platform that might be on, and try to do the same thing, in the same way on the Linux machines. Once you can get the basics in place, you should really switch to using that linux install as your main workstation. While this might not give you much in terms of low level admin abilities, it will give you a user perspective on things. And I have always thought that the best admins are those who consider the user pespective, the developer perspective and then the platform ( and admin ) perspective. End of the day, lets not forget that the computers are here to do a job, and the admins role is to make sure that the job is done to its best ability. But dont lose context : the aim is still to run that job.

In the early days of my Linux experience, I used to find it hard to relate to other people's applications and what they might be doing with their computers and their networks. It was hard since I wasent actually in those roles, so even coming up with situations was hard. And looking for experience-situations I realised that the best way to get one's hooks into an app was to join the mailing list for that app. And work on some of the issues that people brought up, ask questions about why certain things were being done in a specific way and to look at bug reports that people were posting about that app - since that clearly showed how that app was being used 'in the real world', and it gave me a very good foundation to build on. About 14 years later, I still think that the usergrounps for specific apps are the best way to really learn about the app, how its managed, what best practises around that might be and what the developer / user perspective is for those apps.

Finally, being able to program and write real code helps. Dont believe the people who go around saying that admin needs no coding. On the other hand, speak to some of the good sysadmin people around ( and there are plenty ): pretty much everyone will tell you that they spend between 40 to 60% of their time writing scripts and working with apps where knowing the basics of development help. I am not saying that a certification in Java is needed, but having a good understanding of the basics for bash and atleast one of python,perl or ruby should be considered essential. The traditional mindset of unix/c still exists, but not many sysadmins these days need to get down to driver level development, and most functional code that sysadmins need to work with are well handled in the bash, ruby, perl and python worlds.

- KB

Although its in the man page, most people don't know that you can specify schedules using one of these meta specs :

@reboot : Run once after reboot.
@yearly : Run once a year, ie. "0 0 1 1 *".
@annually : Run once a year, ie. "0 0 1 1 *".
@monthly : Run once a month, ie. "0 0 1 * *".
@weekly : Run once a week, ie. "0 0 * * 0".
@daily : Run once a day, ie. "0 0 * * *".
@hourly : Run once an hour, ie. "0 * * * *".


So if one of those fits the requirements, use it. At the very least, you end up with better readability. And because users can set these on their own, it allows them to create jobs and services that come online on machine reboot. Without there being any need for them to get access to the system initscripts. Easy win for a problem that comes up quite a few times and in many places.

Now someone needs to make puppet be aware of these!

- KB

Hi Guys,

A few days back I announced that there will be a 'week of rpm-metadata' hacks, starting from today. With great regret I haveto now announce that this must be postponed for a few weeks. Things are going to be extremely hectic for the next two weeks with work and personal issues, so I wont really have the 2hrs or so per day needed to make this happen right now.

I will come through with this, and promise to do so. Just not this week.

- KB

One of the most under utilised things around in any CentOS sysadmins toolset is the rpm-metadata repositories. Some part of it comes from the fact that most people consider it to be yum specific, and while thats true to quite an extent - its not the only use for the rpm-metadata info. The bottom line is that rpm-metadata is a collection of information extract from the rpms contained in the repository and contains a lot of content, state and depenencies.

Over the course of next week, starting on the 26th September I'm going to run through a week of rpm-metadata tricks. One new cool thing to do with the info contained in the repo metadata's every day.

Some of these things are perhaps good candidates for yum plugins, but I'll try and stay clear of any dependancy on yum and also try and cover as many languages as possible : atleast ruby, python, php, bash and lua! Stay tuned.

- KB

Given the recent developments on the opensolaris front, there is again the issue of what OS do people run on their existing sparc hardware ? And I wonder if now is a good time to consider bringing up the CentOS on Sparc conversations again.

I have some hardware that can be used for the builds and I know there is enough contributed / contribute-able resources around to setup some development instances as well. However, the real issue is : is there much desire out there to have CentOS running on these machines ?

Even though I don't have any equipment in production running off Sparc anymore, I know there are quite a few people who do. And it looks like Oracle wants to put these people in a position where they either run a paid for supported Solaris or nothing at all. If we can do something to create an option, that would be awesome. And given the huge commitment to open source that the CentOS upstream has, doing it here on this platform makes the most sense since efforts put in will stay in an open source model, with equal opportunity for anyone to join the effort and continue it. Knowing that there is no vendor preferences or vendor isolation / ownership issues being involved.

CentOS also makes for a good choice since the porting and development effort would need to only really be done at one point - when a major new release is announced upstream - and from that point on, its a case of maintaining the builds for the lifespan. Considering that the lifespan for a CentOS release would be 7 years or more, it makes the effort even more worthwhile.

Usual suspects still hold true : long life, stable releases, documentation that is shared with the i386/x86_64 world for most parts, userspace support in the regular media, a clueful community and an awesome infrastructure setup etc.

Maybe we can sync up with the Fedora-sparc guys and see if there is any synergy there. I know that they have been getting ready for a new release soon.

Something to think about.

- KB

I had to build a fresh set of rpms for the area_cli tools, and decided it was also a good time to rebase some of my local personal build tools to rhel6beta2 since its there and I need to do some tests with it anyway.

Firstly, there are a couple of cool things with rhel6 beta2 - rpmdevtools is included by default, which helps gets things started and helps a bit in managing spec files etc.

What isnt so cool is that the newer rpm in el6beta creates packages which then cause issues with some of the older CentOS releases. eg. this areca_cli package built on el6beta2/x86_64 for target i686 caused this to happen with yum:

Downloading Packages:
areca_cli-1.83_091103-1.e 100% |=========================| 493 kB    00:00     
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
rpmlib(FileDigests) is needed by areca_cli
rpmlib(PayloadIsXz) is needed by areca_cli
Complete!
(1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Which causes the package to not install. This is on a CentOS-5.2 machine. And I need to maintain that at CentOS-5.2 due to various issues ( no, machine is not available on the internet, and only hosts a single app in production ). Trying the install on a CentOS-5.3 machine I get the exact same issue. Upgrading rpm to 4.4.2.3-18.el5 ( the version in 5.4 and 5.5 ) makes no difference.

On the other hand, builds run on CentOS-5 have no problems installing on EL6Beta. So for the time being it looks like all buildhosts and all build stuff will need to stay on EL5. Specially since that allows you to target CentOS-3 and CentOS-4 as well.

- KB

Note: yeah, I see that url pointing at bugzilla isnt idea. I'll look into plumbing in a bugs.centos.org reference instead.