Hey, you sass that hoopy Bill Napier? There's a frood who really knows where his towel is.

Borg, Borglets, and Borgmasters. Oh My!

I've been messing with EC2 and cloud computing for some time now. And it's always frustrated me how hard it was to deploy software, versus how easy it was for me to deploy software at work.

Prior to containers like Docker, bring up a new machine was a chore. If you took good notes that last time you did it, that helped (I never did). But chances are, even with your notes, some stuff had changed since the last time you did it.

Enter Docker and containers. I've talked (briefly) about Docker before, but as a refresher you build a container, and can simply deploy it on any machine you like (among many other important things). I say simply, and at first I truely believed it. But once I started running it, I found quite a few gnarly points to deal with. Like the fact that when your container crashes, it's up to you to restart it. Or if you want logs, feel free to configure it on the host. So much for easily moving containers from host to host, you still need to take notes on how your machine was originally setup.

At Google, this has been a solved problem for quite some time. Our production infrastructure handles almost all this for you. You build an image that contains the binaries you need, list the dependent images you need, fill out some constraints on how you want it to run, and hand it to the Borgmaster and it takes care of it for you. Now, I wouldn't call this system user friendly. The constraint language is SUPER complicated (and also SUPER flexible), but at least it's documented. Personally, I came up with a config file back in 2009 and have been using it as the basis for each new service I need to bring up.

I got really excited when I attended OScon last year and learned about Kubernetes (from Google). Being familiar with Borg, I knew immediately what they were talking about. They were re-building our internal Borg systems to run on EC2 (And of course, Google Compute Engine).

And now Google has gone ahead and made a research paper available describing it. (like other core Google tech like Bigtable and Spanner). If you run an internet stack somewhere, you should go read it. If you find research papers to be a little slow going and want a quick read, check out this blog post from the Kubernetes team describing how things learned when developing Borg influenced design decisions in Kubernetes.

One of the reasons I'm uncomfortably excited about Kubernetes is that I trust the people that brought me Borg, to get their second system right. I don't have that same trust from the Docker team, as I've seen them punt on issue that I felt they should also be solving, and in general not always go in the direction I felt was right.

So let's be honest. This isn't all for the greater good, Google expeects to turn a profit on this. Google offers Google Container Engine, which is a hosted kubernetes solution, just like Borg. I'm really looking forward to this system getting more mature, so I can simplify the amount of sysadmin work I need to do to run the small internet services I run.

comments powered by Disqus
Bill Napier

Bill Napier

Technologist, Gadget Lover, Father. Doing full time work at Google and part time work on everything else.