The portable cloud

In late 2012 I constructed myself a bare bones cluster of a couple of motherboards, stacked up and powered, to be used as a dev cloud. It worked, but was a huge mess on the table, and it was certainly neither portable nor quiet. That didnt mean I would not carry it around – I did, across the atlantic a few times, over to Asia once. It worked. Then in 2014 I gave the stack away. Which caused a few issues, since living in a certain part of London means I must put up with a rather sad 3.5mbps adsl link from BT. Had I been living in a rural setting, government grants etc would ensure we get super high speed internet, but not in London.

I really needed ( since my work pattern had incorporated it ), my development and testing cluster back. Time to build a new one!

Late summer last year the folks at Protocase kindly built me a cloud box, to my specifications. This is a single case, that can accommodate upto 8 mini-itx (or 6 mini-ATX, which is what i am using ) motherboards, along with all the networking kit for them and a disk each. Its not yet left the UK, but the box is reasonably well traveled in the country. If you come along to the CentOS Dojo, Belgium or the CentOS table at Fosdem, you should see it there in 2016. Here you can see the machine standing on its side, with the built in trolley for mobility.

Things to note here : you can see the ‘back’ of the box, with the power switches, the psu with its 3 hot swap modules, the 3 large case cooling fans and the cutout for the external network cable to go into the box. While there is only 1 psu, the way things are cabled inside the box, its possible to power upto 4 channels individually. So with 8 boards, you’d be able to power manage each pair on its own.


Here is the empty machine as it was delivered. The awesome guys at Protocase pre-plumbled in the psu, wired up the case fans ( there are 3 at the back, and 2 in the front. The ones in the front are wired from the psu so run all the time, where as the back 3 are connected as regular case-fan’s onto the motherboards, so they come up when the corresponding machine is running ) – I thought long and hard about moving the fans to the top/bottom but since the machine lives vertically, this position gives me the best airflow. On the right side, opposite from the psu, you can see 4 mounting points, this is where the network switch goes in.

Close up of the PSU used in this machine, I’ve load tested this with 6x i5 4690K boards and it works fine. I did test with load, for a full 24 hrs. Next time I do that, I’ll get some wattage and amp readings as well. Its rated for 950w max. I suspect anything more than 6 boards will get pretty close to that mark. Also worth keeping in mind is that this is meant to be a cloud or mass infra testing machine, its not built for large storage. Each board has its own 256gb ssd, and if i need additional storage, that will come over the network from a ceph/gluster setup outside.

The PSU output is split and managed in multiple channels, you an see 3 of the 4 here. Along with some of the spare case fan lines.

Another shot of the back 3 fans, you can also see the motherboard mounting points built into the base of the box. They put these up for a mini-itx / mini-ATX as well as regular ATX. I suspect its possible to get 4 ATX boards in there, but its going to be seriously tight and the case fans might need an upgrade.

Close up of the industrial trolley that is mounted onto the box ( its an easy remove for when its not needed, i just leave it on ).

The right side of the box hosts the network switch, this allows me to put the power cables on the left and back, with the network cables on the right and front. Each board has its own network port ( as they do.. ), and i use a usb3 to gbit converter at the back to give me a second port. This then allows me to split public and private networks, or use one for storage and another for application traffic etc. Since this picture was taken, I’ve stuck another 8 port switch on the front of this switch’s cover, to give me the 16 ports i really need.

Here is the rig with the first motherboard added in, with an intel i5 4960k cpu. The board can do 32 gb, i had 16 in it then, have upgraded since.

Now with everything wired up. There is enough space under the board to drive the network cables through.

And with a second board added in. This time an AMD fx-8350. Its the only AMD in the mix, and I wanted one to have the option to test with, the rest of the rig is all intels. The i5’s are a fewer cores, but overall with far better power usage patterns and run cooler. With the box fully populated, running a max load, things get warm in there.

The boards layer up on top of each other, with an offset; In the picture above, the intel board is aligned to the top of box, the next tier board was aligned to the bottom side of the box. This gives the cpu fans a bit more head room, and has a massive impact on temperature inside the box. Initially, I had just stacked them up 3 on each side – ambient temperature under sustained load was easily touching 40 deg C in the box. Staggering them meant ambient temperature came down to 34 Deg C.

One key tip was Rich Jones discovering threaded rods, these fit right into the mother board mounting points, and run all the way through to the top of the box. You can then use nuts on the rod to hold the motherboard at whatever height you need.

If you fancy a box like this for yourself, give the guys at Protocase a call and ask for Stephen MacNeil, I highly recommend their work. The quality of the work is excellent. In a couple of years time, I am almost certainly going to be back talking to them about the cloudybox2. And yes, they are the same guys who build the 45drives storinator machine.

update: the box runs pretty quiet. I typically only have 2 or 3 machines running in there, but even with all 6 running a heavy sustained load, its not massively loud, the airflow is doing its thing. Key thing there is that the front fans are set to ingest air – and they line up perfectly with the cpu placements, blowing directly at the heat sinks. I suspect the top most tier boards only get about 50% of the airflow compared to the lower two tiers, but they also get the least utilisation of the lot.


CentOS Linux 5 Update batch rate


We typically push updates in batch’s. This might be anywhere from 1 update rpm to 100’s ( for when there is a big update upstream ), however most batches are in the region of 5 to 20 rpms. So how many batches have we done in the last year in a bit ? Here is a graph depicting our update batch release rate since Jan 1st 2014 till today.


I’ve removed the numbers from the release rate, and left the dates in since its the trending that most interesting. In a few months time, once we hit new years I’ll update this to split by year so its easy to see how 2015 compared with 2014.

You can click the image above to get a better view. The blue segment represents batches built, and the orange represents batches released.


CentOS Atomic Host in AWS via Vagrant


You may have seen the announcement that CentOS Atomic Host 15.10 is now available ( if not, go read the announcement here : ).

You can get the Vagrant box’s for this image via the Atlas / VagrantCloud process or just via direct downloads from )

What I’ve also done this time is create a vagrant_aws box that references the AMIs in the regions they are published. This is hand crafted and really just a PoC like effort, but if its something people find helpful I can plumb this into the main image generation process and ensure we get this done for every release.

Once you have vagrant running on your machine, you will need the vagrant_aws plugin. You can install this with:

vagrant plugin install aws

and check its there with a

vagrant plugin list“.

You can then add the box with “vagrant box add centos/atomic-host-aws“. Before we can instantiate the box, we need a local config with the aws credentials. So create a directory, and add the following into a Vagrantfile there :

Vagrant.configure(2) do |config| = "centos/atomic-host-aws"
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "Your AWS EC2 Key"
    aws.secret_access_key = "Your Secret Key"
    aws.keypair_name = "Your keypair name"
    override.ssh.private_key_path = "Path to key"

Once you have those lines populated with your own information, you should now be able to run
vagrant up --provider aws

It takes a few minutes to spin up the instance. Once done you should be able to “vagrant ssh” and use the machine. Just keep in mind that you want to terminate any unused instances, since stopping will only suspend it. A real vagrant destroy is needed to lose the ec2 resources.

Note: this box is setup with the folder sync’ feature turned off. Also, the ami’s per region are specified in the box itself, if you want to use a specific region just add a aws.region = ““, into your local Vagrantfile, everything else should get taken care of.

You can read more about the aws provider for vagrant here :

Let me know how you get on with this, if folks find it useful we can start generating these for all our vagrant images.

Progress on the Software Collections SIG


The software collections special interest group ( ) has been making great progress and have finished their initial bootstrap process. They are now getting ready to do a mass build for test and release. I’ve just delivered their rpm signing key, so we are pretty close to seeing content in

As an initial goal, they are working on and delivering rpms – but in parallel efforts are on to get container images into the registries as well, so folks using containers today are able to consume the software collections in either format.

The effort is being co-ordinated by Honza Horak ( ), and he’s the best person to get in touch with to join and help.


Our second stable Atomic Host release

Jason just announced our second stable CentOS Atomic Host release at

I’m very excited about this one, and its not only because I’ve helped make it happen – but this is also the first time a SIG in the CentOS Ecosystem has done a full release, from rpms, to images, to hosted vendor space ( AMI’s in 9 regions on Amazon’s EC2 ).

One of the other things that I’ve been really excited about is that this is the first time we’ve used the rpm-sign infra that I’ve been working on these past few days. It allows SIG built content ( rpms or images or ISOs or even text ) to be signed with pre-selected keys. And do this without having to compromise the key trust level. I will blog more around this process and how SIGs can consume these keys, and how this maps to the TAG model being used in

for now, go get started with the CentOS Atomic Host!


CentOS Dojo in Barcelona, 20th Oct 2016


We have a dojo coming up in Barcelona, co-located with the OpenNebula conference in late October. The event is going to run from 1:30pm to 6:30pm ( but I suspect it wont really end till well into the early hours of the morning as people keep talking about CentOS things over drinks, dinner, more drinks etc ! ).

You can get the details, including howto register at

Fabian is going to be there, and we are talking to a great set of potential speakers – the focus is going to be very much on hands on learning about technologies on and around CentOS Linux! And as in the past, we expect content to be sysadmin / operations folks specific rather than developers ( although, we highly encourage developers to come along as well, and talk to us and share their experiences with the sysadmin world! ).


timezone mangling

Because of what I do and how / where I do it, there are always online, realtime conversations going on ( irc or IM ); and its never really been a huge issue except for people in the US pacific coast. Its always a case of them starting work when I am finishing for the day, and even when i work late at night for the odd hours, its almost always whack in the middle of their lunch hours. And they finish work, even their late night sessions, just about when I am getting started for the day.

So to everyone on that TZ, just want to remind everyone that the best thing to do is stick with emails. I know its fashionable these days to complain about emails and all that, but by and large there is no other means of comms around these days that is easier to get to, mature and really very productive for async conversations. The other thing to keep in mind is that while there are other services and ideas floating around that help solve specific challenges that email isnt best suited for, none of them do a good enough job to remove the email process from the equation. So if we are still going to have email knocking about, lets just use it.

And I’m not ignoring people on irc :) but with 300+ panes in irssi, sometimes it can get hectic and I will often encourage you to ‘Lets Move to Mail’. Its not because I dont want to have the convo right now, its because I want to have the complete conversation!


CentOS Meetup in Pune, India – Sat Jul 11th 2015


Humble and the guys in the CentOS-India group are organising a meetup in Pune this Saturday, the 11th July 2015. is a link to the meetup page. The Agenda looks great :

  • Basics of Systemd: Praveen Kumar
  • UEFI and secure boot: Yogesh Babar
  • Docker container and Atomic Host: Ranjith Rajaram
  • Debugging Kernel Issues: Gopal Tiwari

The Event kicks off at 9am with a welcome and intro. So if you are in the area and able to make it, sign up for the event and come along. If you are based in India, it would be great to see you sign up to the CentOS-India group anyway, so you can keep an eye on events coming to your areas, and if possible- maybe even help us run an event in your area!


– KB

CentOS Shirt Fridays

Hi Everyone,

Over the last eight years we have handed out literally thousands of CentOS Project Tshirts. From Houston to Bangalore to Edinburgh to Sydney to Johannesburg to Porto Alegre to London and even the 4 that went to Antarctica. Lets get those shirts out every friday and see if we can get a CentOS Shirt Friday going. Get into them and lets get posting on social media! Lets use the #CentOSShirtFriday hashtag.

Thanks to Barton ( ) for the idea. And thanks to the folks at Cpanel ( ) for starting us off on this CentOS shirt journey many many years ago.

Dont have a CentOS shirt yet ? We typically have shirts at the events we attend as a project. You can see where we are going in the near future at the Events page ( ).

So, I’ll see you all tomorrow online, with my CentOS shirt on. Now, lets see if I can find the first one we ever did.

– KB

Regular office hours

Update: I’ve moved the Thurs evening slot to Wednesday so as to not clash with Atomic SIG meeting.


Starting this Thursday the 11th June, 2015 – I am going to make myself available to anyone who wants to come along and talk about CentOS Linux, the CentOS Project, the SIGs or anything else that is related to these.

To start things off, I am going to be available on all Wednesdays:

  • 04:00-04:30 pm UTC (5:00pm London, 12noon Eastern, 9am Pacific )

And on all Thursdays:

  • 08:30-09:00 am UTC (9:30am London, 2:00pm, India, 4:30pm Singapore)

During this time, you can find me on #centos-devel ( as ‘kbsingh’), and you can also call me on the phone at +44 207 009 4455.

Speak to you then!

– KB