The Future of Vagrant – Toward 3.0

In order to support its growing ecosystem and community as we move toward the 3.0 release, we are making changes to Vagrant that will maintain its support for Ruby-based Vagrantfiles and Vagrant plugins. Learn about the details in this HashiConf session.

Speakers: Sophia Castellarin, Chris Roberts

»Transcript

Sophia Castellarin:

Hello, everyone. I hope you're having a wonderful HashiConf Europe so far. Thanks for checking out our talk. We're going to talk about the future of Vagrant, what's coming up in 2021 — and beyond. I'm Sophia. I'm an engineer on the Vagrant team. I'm here with my colleague, Chris, who is the engineering lead. I'll leave it to you, Chris.

Chris Roberts:

Great, thanks. Before we get started, I want to have a quick look at Vagrant's history. 

»The History of Vagrant 

Vagrant was the first HashiCorp product. It's so old in fact, it predates the company. The first HashiCorp product was Vagrant-based. The Vagrant VMware plugin was the first product offered by HashiCorp all the way back in 2013. It helped provide the initial revenue to start building new tools. 

Given the age of Vagrant, it's not surprising that Vagrant has a very large community base. In many cases, Vagrant is a user's first introduction to HashiCorp and HashiCorp tools. It's used by new and experienced developers alike. There's even tooling built on top of Vagrant itself.

»Challenges

However, Vagrant provides many difficult maintenance challenges. To start, let's take a look at how Vagrant is shipped. Vagrant is a Ruby-based application, and this alone makes Vagrant different than other HashiCorp products. Whereas the majority of other teams ship a binary, we ship an entire package. When we talk about shipping Vagrant, we're also talking about all of Vagrant's dependencies. Let's have a quick look at what they are — and what's included within the Vagrant package.

For Vagrant builds on POSIX-style systems, we build the Ruby interpreter and cURL along with installing Vagrant. This also means building all the required dependencies as well. On Windows, Ruby requires a POSIX compatibility layer. There we are able to lean on MSYS2 and their Pacman port for building our dependencies. We still build a custom Ruby and a custom cUR.L But their dependencies are provided by MSYS2. To further complicate things on Windows, our Windows builds ship with two different cURL executables: an MSYS2-based build and a custom build with WinSSL support.

»Why Packages? 

Why distribute these packages if they're difficult to maintain? Well, to start, it makes the installation process easy. One file to install, and Vagrant is ready to go. The installers also help to provide reproducibility. Given a version of Vagrant on a platform, we know exactly what is installed which helps us for debugging and finding issues with various things within Vagrant. Finally, the packages can be considered batteries-included. Once installed, Vagrant can be in a usable state without any requirement for installing third-party tools.

So while the packages are helpful in some situations, they can also cause us some issues. Installers are only available for a limited number of host platforms and distributions for Linux. AppImage has helped us to address this on the Linux side a bit, but it also incurs a significant runtime penalty. 

Packaging installers like this also leads to pinned Ruby versions, which can make it for difficult for users who are trying to run Vagrant outside of the installer packages that we provide. There are also other issues that we find with these heavy package installations: Tools that are provided with our package may conflict with other existing installations.

Terminal users also can have issues, especially on Windows. This generally crops up where a custom terminal may be being used by a user that's powered by Cygwin or MSYS2. And finally, plugin installation can become difficult due to dependencies and the system not being able to properly resolve them.

»Plugins

While packaging is a big challenge that we deal with, the internals of Vagrant also provide their own set of challenges. Vagrant currently includes around 70 built-in plugins. This means that the Vagrant team maintains a lot of plugins. These plugins include 38 different guest plugins, 12 host plugins, 10 provisioners, 4 providers, and 3 different communicators. 

Having all these built-in plugins is great. They helped us ensure the interface for the plugins is robust and functional. However, once included in the core, these plugins are assumed to be fully supported. That means we spend a lot of time maintaining plugins that may not see much actual usage. While having built-in plugins helps us maintain the plugin interface internally, it also introduces hardships on the maintenance of the plugins themselves. 

Since built-in plugins ship with Vagrant, any fix or update requires a full Vagrant release. And depending on the state of Vagrant's current development and what's in process, there may be a significant delay between updates getting merged and the availability of it in an actual release. These types of delays don't impact non-built-in plugins as they have no hard requirement on Vagrant's release cadence.

»Architectural Concerns 

There are some architectural constraints at play as well. Some features often requested are either not possible, or require difficult workarounds. Some of these include GUI interfaces for interacting with guests, remote system support for hosts for Vagrant guests, and sharing Vagrant projects themselves. 

Vagrant is not architecturally well-positioned to solve these new concerns in its current state. And that’s not surprising. Vagrant was designed ten years ago to be a single-user CLI tool and since then a lot of technology has changed. Vagrant is an important tool for managing VMs, but it should also be able to meet the evolving needs of its users. And given the uniqueness of Vagrant's implementation within HashiCorp, it leaves us unable to utilize common and useful HashiCorp libraries that have been developed for other products.

Sophia Castellarin:

Given all of these concerns we have and challenges we face on the Vagrant team, we asked ourselves this question: How do we maintain Vagrant for the next ten  years?“

We have a couple pieces to a solution to solve this issue. One piece of the solution is to externalize plugins. 

»Externalizing Plugins 

Chris mentioned that we have 70 plugins. This externalization of them means pulling them out of being a built-in plugin into their own project that is still installable using a Vagrant plugin install command.

Our goal here is to limit maintenance work that is not very impactful to what we do as developers. That way we free up our time, so that we can work on more impactful core features of Vagrant. Part of this process is also to enable the community by doing stuff like focusing on making docs that are more clear about how to develop a plugin and how to maintain it. The other ultimate goal of this project is to limit gatekeeping, so that the Vagrant team isn't stopping a pull request from getting merged or from plugins getting released and then used out in the wild.

»Switching to Golang

The next change that's coming is a switch over to Golang. Vagrant is a Ruby project. This represents a big change in how the project is going to go. What we plan to gain from this is being more tied into the HashiCorp ecosystem. That will make maintenance easier,  But this restructuring of the Vagrant architecture will also allow us to build more features that will help solve modern problems in virtual machine-based workflows.

»Maintaining Backward Compatibility

One of our big concerns, and one of the things that we plan to address in this move, is maintaining backward compatibility with both Vagrant files and Vagrant plugins. Ruby-based Vagrant plugins will continue to work for a long time. 

We're not going to get into too much of the engineering detail here, but we have a little diagram showing how we're going to do this backward compatibility. We have this Vagrant core written in Go, which we are going to call Vagrant-a-go-go here. We have this Vagrant shim, which translates information between Vagrant-a-go-go and our legacy Vagrant, which will have the Ruby runtime. Using this setup, we can continue to do stuff like understand Vagrant files and also run Ruby-based Vagrant plugins.

»Vagrant’s New Architecture

So a little bit about this new architecture. This goes hand-in-hand with the move over to Go. With this move, we also plan to address some of the plugin externalization issues. Doing stuff like maintaining a stricter component interface will be something that we'll be able to do in this new architecture.

Next, this new architecture will allow us to implement some changes that weren't possible — or hardly fought for — in Vagrant as it is today. Doing things like managing privilege escalations for security-minded folks or doing stuff like remote execution for solving issues around VMs in the cloud. It will also allow us to move to a Vagrant HCL file  — so writing your Vagrant files in HCL similar to what you can do in other HashiCorp products.

»Vagrant Demo 

Let's take a look at this a little bit in action. I'm in my Vagrant project here, and we can go build a Vagrant executable. Let's take a look. There it is. It's Vagrant. Let's ask Vagrant what commands it has for us. We'll give it a second, and we see it has all of the commands that exist in Vagrant today. It looks a little bit different, but all the functionality is still there.

Let's run a Vagrant status, and we'll get some verbose output, so we can see what's going on under the hood a little bit. We see a lot of Go logs. And at the end we have a familiar message about what is happening. If we scroll up a little bit, we can see that there are all of these Ruby plugins that are getting loaded. There are some Go plugins that are getting loaded. Then there are some more actions that are happening. But at the end, we have this Vagrant status that is running in the Ruby runtime.

Let's go do something useful. We have this Vagrant file here. Doesn't do anything very exciting. It can bring up a box. So let's bring up a box. In this command, we have Vagrant being run from Go. The up command is, again, being provided through a Ruby plugin. This is all getting executed in the Vagrant implementation of Ruby, and all of this output is coming from there too. If we wait a second, we'll have a machine up. (Unfortunately, Vagrant-a-go-go hasn't solved the issue of waiting for machines to start up.)

Let's do a Vagrant status to see if we have this machine — just to confirm Again, this is being run on the Ruby side. We see that we have a beautiful VirtualBox running, so let's move on to running some Go plugins. 

I have this plugin called myplugin. It's a very exciting name. It does some very exciting stuff. For example, doing things, being interactive, and showing info. Let's run the info command. This is — again — a Go plugin that is being run here. This will just output some information about the project. We see that we have the same machine that comes out in status — everything is consistent in here.

That was pretty much everything that is interesting here. Everything is working pretty much the same as it did in old Vagrant. We can clean up our environment. We'll do a destroy, again, Ruby plugin. All of this is coming from Ruby. You can still interact with a console in the same way. And we destroyed the machine. Let's do a Vagrant status one more time — for good measure — to make sure that our box — our virtual machine — has come down. Cool. And it's down.

»Demo Summary 

The main takeaways from that demo, we have a Vagrant CLI that is written in Golang. Core Vagrant is written in Golang,— or parts of it. We still have Ruby plugins that can be used, and we have new Vagrant plugins that can be written in Golang. But most importantly, everything still works the way it should. And as an end user you shouldn't be seeing any difference in how you interact with Vagrant.

»Vagrant Development Timeline

Now the exciting part. When do we get to play with these new fun toys? Right now we're on Vagrant 2.2, 2.2.x. And through the two series, we will continue to be shipping Ruby with our Vagrant installers. In this series, we're going to be porting the rest of the core Vagrant into Go. We'll be doing this plugin externalization piece — and that includes doing stuff like deprecating and importing plugins.

By the time we get to the three series, we're going to stop shipping Ruby with our packages, and we'll just be shipping Go binaries. For plugins that are still Ruby, we'll still be able to support those, but it will be more of a bring-your-own-Ruby situation. Similarly, Vagrant Share is also getting open-sourced, and it'll be at hashicorp/vagrant-share. Long term, we want to address modern issues with virtual machine-based workflows using Vagrant. This includes integration with HashiCorp tools.  I also want to emphasize we have a continued commitment to the community.

Chris Roberts:

We hope you enjoyed this glimpse into the future of Vagrant. Please see our blog post — Toward Vagrant 3.0 —for more detailed information than was covered here, including more about our timeline and the open-sourcing of the plugins. Thanks very much for coming, and enjoy the rest of HashiConf EU.