Getting Started with CentOS CI

We have been building out a CentOS Community CI infra, that is open to anyone working on infra code or related areas to CentOS Linux, and have now onboarded a few projects. You can see the web ui ( jenkins! ) at https://ci.centos.org/.

Dusty has also put together a basic getting started guide, that also goes into some of the specifics on how and why the CentOS CI infra works the way it does, check it out at http://dustymabe.com/2016/01/23/the-centos-ci-infrastructure-a-getting-started-guide/.

Regards,

Few changes in CentOS Atomic Host build scripts

hi,

If you use the CentOS atomic host downstream build scripts at https://github.com/CentOS/sig-atomic-buildscripts you will want to note a major change in the downstream branch. The older build_ostree_components.sh script has now been replaced with 3 scripts:
builds_stage1.sh, build_stage2.sh and build_sign.sh; Running build_stage1.sh followed by build_stage2.sh will give you exactly the same output as the old script used to.

The third script, build_sign.sh, now makes it easier to sign the ostree repo before any of the images are built. In order to use this, generate or import your gpg secure key, and drop the resulting .gpg file into /usr/share/ostree/trusted.gpg.d/ and edit the build_sign.sh script, edit the keyid at the end, and run the script after your build_stage1.sh is complete ( and before you run the build_stage2.sh ). You will notice a pinentry window popup, enter the password, and check for a 0 exit. Note that the gpg sign is a detached sign for the ostree commit.

regards,

CentOS Meetup in London 3rd Dec 2015

Hi,

We now have a CentOS Users and contributors group for the UK on meetup.com ( http://www.meetup.com/CentOS-UK/ ), and I hosted the inaugural meetup over beer a few days back. It was a great syncup, and lots of very interesting conversations. One thing that always comes through at these meetings and I really appreciate is the huge diversity in the userbase, and the very different viewpoints and value propositions that people focus on into the CentOS Linux platform, and the larger ecosystem around it.

The main points that stuck with me over the evening were the CentOS Atomic Host ( https://wiki.centos.org/SpecialInterestGroup/Atomic/Download ) and the CentOS on ARM devices ( and the general direction of where ARM devices are going ). Stay tuned for more info on that in the next few weeks.

Looking forward now to the next London meetup ( likely 2nd week of Jan ’16 ), and also joining some meetings in other parts of the UK. Everyone is welcome to join, and I could certainly use help in organising meetups in other places around the UK. See you at a CentOS meetup soon.

Regards,

The portable cloud

In late 2012 I constructed myself a bare bones cluster of a couple of motherboards, stacked up and powered, to be used as a dev cloud. It worked, but was a huge mess on the table, and it was certainly neither portable nor quiet. That didnt mean I would not carry it around – I did, across the atlantic a few times, over to Asia once. It worked. Then in 2014 I gave the stack away. Which caused a few issues, since living in a certain part of London means I must put up with a rather sad 3.5mbps adsl link from BT. Had I been living in a rural setting, government grants etc would ensure we get super high speed internet, but not in London.

I really needed ( since my work pattern had incorporated it ), my development and testing cluster back. Time to build a new one!

Late summer last year the folks at Protocase kindly built me a cloud box, to my specifications. This is a single case, that can accommodate upto 8 mini-itx (or 6 mini-ATX, which is what i am using ) motherboards, along with all the networking kit for them and a disk each. Its not yet left the UK, but the box is reasonably well traveled in the country. If you come along to the CentOS Dojo, Belgium or the CentOS table at Fosdem, you should see it there in 2016. Here you can see the machine standing on its side, with the built in trolley for mobility.

Things to note here : you can see the ‘back’ of the box, with the power switches, the psu with its 3 hot swap modules, the 3 large case cooling fans and the cutout for the external network cable to go into the box. While there is only 1 psu, the way things are cabled inside the box, its possible to power upto 4 channels individually. So with 8 boards, you’d be able to power manage each pair on its own.

Box-1

Here is the empty machine as it was delivered. The awesome guys at Protocase pre-plumbled in the psu, wired up the case fans ( there are 3 at the back, and 2 in the front. The ones in the front are wired from the psu so run all the time, where as the back 3 are connected as regular case-fan’s onto the motherboards, so they come up when the corresponding machine is running ) – I thought long and hard about moving the fans to the top/bottom but since the machine lives vertically, this position gives me the best airflow. On the right side, opposite from the psu, you can see 4 mounting points, this is where the network switch goes in.
Box-2

Close up of the PSU used in this machine, I’ve load tested this with 6x i5 4690K boards and it works fine. I did test with load, for a full 24 hrs. Next time I do that, I’ll get some wattage and amp readings as well. Its rated for 950w max. I suspect anything more than 6 boards will get pretty close to that mark. Also worth keeping in mind is that this is meant to be a cloud or mass infra testing machine, its not built for large storage. Each board has its own 256gb ssd, and if i need additional storage, that will come over the network from a ceph/gluster setup outside.
Box-3

The PSU output is split and managed in multiple channels, you an see 3 of the 4 here. Along with some of the spare case fan lines.
Box-4

Another shot of the back 3 fans, you can also see the motherboard mounting points built into the base of the box. They put these up for a mini-itx / mini-ATX as well as regular ATX. I suspect its possible to get 4 ATX boards in there, but its going to be seriously tight and the case fans might need an upgrade.
Box-5

Close up of the industrial trolley that is mounted onto the box ( its an easy remove for when its not needed, i just leave it on ).
Box-6

The right side of the box hosts the network switch, this allows me to put the power cables on the left and back, with the network cables on the right and front. Each board has its own network port ( as they do.. ), and i use a usb3 to gbit converter at the back to give me a second port. This then allows me to split public and private networks, or use one for storage and another for application traffic etc. Since this picture was taken, I’ve stuck another 8 port switch on the front of this switch’s cover, to give me the 16 ports i really need.
Box-7

Here is the rig with the first motherboard added in, with an intel i5 4960k cpu. The board can do 32 gb, i had 16 in it then, have upgraded since.
Box-8

Now with everything wired up. There is enough space under the board to drive the network cables through.
Box-9

And with a second board added in. This time an AMD fx-8350. Its the only AMD in the mix, and I wanted one to have the option to test with, the rest of the rig is all intels. The i5’s are a fewer cores, but overall with far better power usage patterns and run cooler. With the box fully populated, running a max load, things get warm in there.
Box-10

The boards layer up on top of each other, with an offset; In the picture above, the intel board is aligned to the top of box, the next tier board was aligned to the bottom side of the box. This gives the cpu fans a bit more head room, and has a massive impact on temperature inside the box. Initially, I had just stacked them up 3 on each side – ambient temperature under sustained load was easily touching 40 deg C in the box. Staggering them meant ambient temperature came down to 34 Deg C.

One key tip was Rich Jones discovering threaded rods, these fit right into the mother board mounting points, and run all the way through to the top of the box. You can then use nuts on the rod to hold the motherboard at whatever height you need.

If you fancy a box like this for yourself, give the guys at Protocase a call and ask for Stephen MacNeil, I highly recommend their work. The quality of the work is excellent. In a couple of years time, I am almost certainly going to be back talking to them about the cloudybox2. And yes, they are the same guys who build the 45drives storinator machine.

update: the box runs pretty quiet. I typically only have 2 or 3 machines running in there, but even with all 6 running a heavy sustained load, its not massively loud, the airflow is doing its thing. Key thing there is that the front fans are set to ingest air – and they line up perfectly with the cpu placements, blowing directly at the heat sinks. I suspect the top most tier boards only get about 50% of the airflow compared to the lower two tiers, but they also get the least utilisation of the lot.

enjoy!

CentOS Linux 5 Update batch rate

Hi,

We typically push updates in batch’s. This might be anywhere from 1 update rpm to 100’s ( for when there is a big update upstream ), however most batches are in the region of 5 to 20 rpms. So how many batches have we done in the last year in a bit ? Here is a graph depicting our update batch release rate since Jan 1st 2014 till today.

cl5-update-batch-rate

I’ve removed the numbers from the release rate, and left the dates in since its the trending that most interesting. In a few months time, once we hit new years I’ll update this to split by year so its easy to see how 2015 compared with 2014.

You can click the image above to get a better view. The blue segment represents batches built, and the orange represents batches released.

regards,

CentOS Atomic Host in AWS via Vagrant

Hi,

You may have seen the announcement that CentOS Atomic Host 15.10 is now available ( if not, go read the announcement here : http://seven.centos.org/2015/10/new-centos-atomic-host-release-available-now/ ).

You can get the Vagrant box’s for this image via the Atlas / VagrantCloud process or just via direct downloads from http://cloud.centos.org/centos/7/atomic/images/ )

What I’ve also done this time is create a vagrant_aws box that references the AMIs in the regions they are published. This is hand crafted and really just a PoC like effort, but if its something people find helpful I can plumb this into the main image generation process and ensure we get this done for every release.

QuickStart
Once you have vagrant running on your machine, you will need the vagrant_aws plugin. You can install this with:

vagrant plugin install aws

and check its there with a

vagrant plugin list“.

You can then add the box with “vagrant box add centos/atomic-host-aws“. Before we can instantiate the box, we need a local config with the aws credentials. So create a directory, and add the following into a Vagrantfile there :

Vagrant.configure(2) do |config|
  config.vm.box = "centos/atomic-host-aws"
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "Your AWS EC2 Key"
    aws.secret_access_key = "Your Secret Key"
    aws.keypair_name = "Your keypair name"
    override.ssh.private_key_path = "Path to key"
  end
end


Once you have those lines populated with your own information, you should now be able to run
vagrant up --provider aws

It takes a few minutes to spin up the instance. Once done you should be able to “vagrant ssh” and use the machine. Just keep in mind that you want to terminate any unused instances, since stopping will only suspend it. A real vagrant destroy is needed to lose the ec2 resources.

Note: this box is setup with the folder sync’ feature turned off. Also, the ami’s per region are specified in the box itself, if you want to use a specific region just add a aws.region = ““, into your local Vagrantfile, everything else should get taken care of.

You can read more about the aws provider for vagrant here : https://github.com/mitchellh/vagrant-aws

Let me know how you get on with this, if folks find it useful we can start generating these for all our vagrant images.

Progress on the Software Collections SIG

hi,

The software collections special interest group ( https://wiki.centos.org/SpecialInterestGroup/SCLo ) has been making great progress and have finished their initial bootstrap process. They are now getting ready to do a mass build for test and release. I’ve just delivered their rpm signing key, so we are pretty close to seeing content in mirror.centos.org.

As an initial goal, they are working on and delivering rpms – but in parallel efforts are on to get container images into the registries as well, so folks using containers today are able to consume the software collections in either format.

The effort is being co-ordinated by Honza Horak ( https://twitter.com/HorakHonza ), and he’s the best person to get in touch with to join and help.

Regards,

Our second stable Atomic Host release

Jason just announced our second stable CentOS Atomic Host release at http://seven.centos.org/2015/09/announcing-a-new-release-of-centos-atomic-host/

I’m very excited about this one, and its not only because I’ve helped make it happen – but this is also the first time a SIG in the CentOS Ecosystem has done a full release, from rpms, to images, to hosted vendor space ( AMI’s in 9 regions on Amazon’s EC2 ).

One of the other things that I’ve been really excited about is that this is the first time we’ve used the rpm-sign infra that I’ve been working on these past few days. It allows SIG built content ( rpms or images or ISOs or even text ) to be signed with pre-selected keys. And do this without having to compromise the key trust level. I will blog more around this process and how SIGs can consume these keys, and how this maps to the TAG model being used in cbs.centos.org

for now, go get started with the CentOS Atomic Host!

regards,

CentOS Dojo in Barcelona, 20th Oct 2016

Hi,

We have a dojo coming up in Barcelona, co-located with the OpenNebula conference in late October. The event is going to run from 1:30pm to 6:30pm ( but I suspect it wont really end till well into the early hours of the morning as people keep talking about CentOS things over drinks, dinner, more drinks etc ! ).

You can get the details, including howto register at https://wiki.centos.org/Events/Dojo/Barcelona2015.

Fabian is going to be there, and we are talking to a great set of potential speakers – the focus is going to be very much on hands on learning about technologies on and around CentOS Linux! And as in the past, we expect content to be sysadmin / operations folks specific rather than developers ( although, we highly encourage developers to come along as well, and talk to us and share their experiences with the sysadmin world! ).

regards,

timezone mangling

Because of what I do and how / where I do it, there are always online, realtime conversations going on ( irc or IM ); and its never really been a huge issue except for people in the US pacific coast. Its always a case of them starting work when I am finishing for the day, and even when i work late at night for the odd hours, its almost always whack in the middle of their lunch hours. And they finish work, even their late night sessions, just about when I am getting started for the day.

So to everyone on that TZ, just want to remind everyone that the best thing to do is stick with emails. I know its fashionable these days to complain about emails and all that, but by and large there is no other means of comms around these days that is easier to get to, mature and really very productive for async conversations. The other thing to keep in mind is that while there are other services and ideas floating around that help solve specific challenges that email isnt best suited for, none of them do a good enough job to remove the email process from the equation. So if we are still going to have email knocking about, lets just use it.

And I’m not ignoring people on irc :) but with 300+ panes in irssi, sometimes it can get hectic and I will often encourage you to ‘Lets Move to Mail’. Its not because I dont want to have the convo right now, its because I want to have the complete conversation!

Regards,