Working with and Customizing Virtualboxes with Vagrant

20 Oct 2017

I have been getting more into Vagrant and Virtualboxes recently. When I figured out how to script vagrant, I started to create custom environments for every project I work on. github repo

Having a custom vagrant file with each project has been beneficial:

I can instantly recreate each project’s environment instantly.
Other users of my projects can instantly recreate my project environment, without my involvement.
This setup can save valuable time, letting others focus on programming, not on the project’s configuration.

This article will go over Virtualbox and scripting Vagrant for a consistent developer experience.

We will cover:

overview of vagrant and virtualbox
interacting with the “computer” created by vagrant
scripting vagrant to setup virtual “computers”

Motivation

Configuration Hell

Do you remember the steps to install your favorite programming langauge? How about any accompanying frameworks?

How about the newest version of your tools, like editors, syntax checkers, source control?

For example, the installed version of Python in Ubuntu 14.04 is 2.7.6. To install 2.7.13, this is just one set of instructions:

Install the python-software-properties package
Install common Python build tools list here
Download Python 2.7.13 source
Configure and Install Python 2.7.13 source

To install the latest supported version of NodeJS, this is one set for Ubuntu:

Get & Setup NodeJS repository information for the system
Update software repository listing
Install NodeJS
Install NPM for the whole system

Each of those are at least four steps and can take a variable amount of time for each step. Getting it done once is a lot of work.

When you want to share your creation with another, or have an new team member join the project, do you remember all the steps taken to setup the software environment? Can you take time out to set up theirs?

Consistency

Instead of looking up or manually installing and configuring a system, write to script Vagrant to automatically create the system.

Setup a system once using a script, then share the script with anyone that needs to have the same environment.

This way, all

new team members download the script and run to get the same software environment as yours
there’s dependency issues with varying versions (i.e. Python 2 vs Python 3)
deliver software consistently

Terminology

Vagrant

This is an open source software tool that manages virtualized environments, like Virtualbox. It’s open-sourced and managed by Hashicorp.

Virtualbox

This is an open source software tool that creates an emulation layer inside another computer. The emulation layer allows for an operating system to be installed on it.

Virtualbox allows a single hardware computer system to operate multiple software computer systems at once. Just like having multiple applications open at once.

Host

The operating system that Virtualbox and Vagrant runs on. It is hosting other operating systems inside itself.

This is also known as Host Operating System, as Virtualbox builds its emulation layer on it.

Guest

The operating system that is run by Virtualbox and managed by Vagrant.

This is also known as Guest Operating System, as the operating system that is a guest to the host.

Think of both in terms of a friend visiting you. You are the host because you know more about the area. Your friend is the guest, as they depend on you for information.

Requirements

System

Required system resources depend on the guest OS requirements.

I will be using Ubuntu 14.04 (Trusty Tahr) that have requirements of:

1GB of free storage space
512MB of available RAM

I recommend more, and probably

free storage space >3GB recommended
system RAM >2GB (more is better)

Install Vagrant

To install Vagrant, download the appropriate installer from here and follow the instructions.

Vagrant is installed when in a terminal, running vagrant --version returns a value, ideally > 1.9

( host )$ vagrant --version
1.9.7

With vagrant, VirtualBox is install automatically. This is the homepage of the virtualbox project.

Setting up a new guest OS

These are the steps to setup Ubuntu as a guest OS on your computer.

The scripts I discuss are available at: https://github.com/a-leung/vagrant_shell_provisioning

Vagrantfile

To create a new guest OS with vagrant, a Vagrantfile is used to control the guest OS settings.

Create a file named: Vagrantfile in the new_development_machine directory with the following contents:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
end

"2" refers to use version 2 of the configuration format
config.vm.box refers to which version the guest operating system vagrant to install, in the above case, the guest operating system specified is Ubuntu 14.04, Trusty Tahr.

vagrant up

So far, with the directory and Vagrantfile, there is no guest OS to work with. To have Vagrant start building the guest operating system, run the following command in the same directory where the Vagrantfile is: vagrant up

The expected output is:

( host )$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
...
    default: Guest Additions Version: 4.3.36
    default: VirtualBox Version: 5.1
==> default: Mounting shared folders...
    default: /vagrant => /Users/user/Developer/new_development_machine

This does a lot of work:

vagrant downloads the guest operating system (Ubuntu 14.04) from cloud servers (~200MB)
setups Virtualbox with guest computer settings
installs the operating system in Virtualbox
sets up the operating system with credentials

The longest part of this process will be downloading the guest operating system. Once it is downloaded, vagrant saves it to the host computer to be reused in the future guest operating systems.

Accessing the Guest Computer

With a new guest computer inside the host computer, how do we access it?

The easiest way is to login to Ubuntu using SSH, secure shell. Vagrant has preconfigured SSH with keys in the vagrant up process.

To login to the guest computer from the same directory as the Vagrantfile:

( host )$ vagrant ssh
...
( guest )$ ls /
bin  boot  dev  etc  home  initrd.img  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  vagrant  var  vmlinuz

From here, commands can be run. If there is anything to be installed that requires superuser/root access, prepend: sudo to the command. For instance, to install Emacs:

( guest )$ sudo apt-get install emacs
Reading package lists... Done
...
Need to get 36.0 MB of archives.
After this operation, 143 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Accessing Files on Guest Computer

To share files between the guest system and host system, vagrant configures the current host computer’s folder (in this article, new_development_machine/ is used) to be shared on the guest computer as /vagrant.

( host )$ ls
Vagrantfile
( guest )$ ls /vagrant
Vagrantfile

Vagrant Provisioning

Now the guest computer is set up for basic operation, it can be used as is like a regular computer. Software can be installed, removed, files shared between the host and guest through the /vagrant directory.

Vagrant has a ‘provisioning’ system that sets up the guest OS according to a script.

I have found vagrant’s provisioning system drastically improves the development experience because it:

installs specified software
allows configurations to be shared through source control
allows computer to be destroyed and re-created with the same settings

I will go over how to use vagrant’s shell provisioner.

Scripting How-to

The shell provisioner is very simple: putting any commands that would be typed in the command line will be executed by the shell provisioner.

The shell provisioner is run as root, so sudo is not needed for any commands in the shell provisioner.

If a command requires an answer from the user, like above when installing Emacs, the shell provisioner will fail. So, ensure commands in the script do not require an answer by adding an option to say “yes”, like -y for apt-get.

Example: Date

Let’s try this by starting with a simple script that writes the current date and time to a file. Modify the Vagrantfile so it is like below:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision "shell", :inline => <<-SHELL
    date > date.txt
  SHELL
end

config.vm.provision tells vagrant to configure the system after initial installation.
"shell" tells vagrant to use the shell provisioner.
:inline => tells vagrant to use the included shell script starting on this line.
<<-SHELL and SHELL are delimiters to indicate the section between them are shell scripts.

To try it out, run $vagrant destroy then $vagrant up. Let’s start with a newly installed guest computer.

Connection to 127.0.0.1 closed.
( host )$ vagrant destroy
    default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
( host )$ vagrant up
...
( host )$ vagrant ssh
...
( guest )$ cat date.txt
Tue Oct 17 14:41:56 UTC 2017

Example: Installing Python

Let’s write a script to install a newer version of Python every time a new computer is created by vagrant:

First, let’s check what version of the Python programming langauge is currently installed:

( guest )$ which python
/usr/bin/python
( guest )$ python --version
Python 2.7.6

Change the Vagrantfile to have the following contents.

There are comments inline (those with a # in front) to explain the commands used.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision "shell", :inline => <<-SHELL
    # apt-add-repository depends on this
    # "-y" is used so apt-get will automatically say yes to any questions
    apt-get install -y python-software-properties
    # get common python build tools:
    # https://github.com/pyenv/pyenv/wiki/Common-build-problems
    apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
    libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
    xz-utils tk-dev
    # get python v2.7.13
    curl --silent -O https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz
    tar xzf Python-2.7.13.tgz
    # install python 2.7.13
    cd Python-2.7.13
    ./configure
    make install
  SHELL
end

source

Test installation:

( host )$ vagrant ssh
( guest )$ which python
/usr/local/bin/python
( guest )$ python  --version
Python 2.7.13
( guest )$ /usr/bin/python --version
Python 2.7.6

Example: Install node

On Ubuntu 14.04, Node.js is not installed by default:

( guest )$ node
The program 'node' can be found in the following packages:
 * node
 * nodejs-legacy
Ask your administrator to install one of them

( guest )$ sudo apt-get install nodejs
...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode
( guest )$ nodejs  --version
v0.10.25

Using the version of node included is a bit old (v0.10.25), which is pretty old by today’s standard, so let’s configure the shell provisioner to install the latest version.

Vagrant.configure("2") do |config|
  config.vm.box = 'ubuntu/trusty64'

  config.vm.provision "shell", :inline => <<-SHELL
  # install 6.x version of node for the system (instead of v0.6.x)
  # https://nodesource.com/blog/installing-node-js-tutorial-ubuntu/
  curl -sL https://deb.nodesource.com/setup_6.x | bash -
  apt-get update
  apt-get install -y nodejs

  # update npm with newer version of node
  npm install npm --global
  SHELL
end

( guest )$ which node
/usr/bin/node
( guest )$ node --version
v6.11.4

Destroy

To get rid of a guest computer, the easiest way is to run ( host )$ vagrant destroy from the host computer directory with the Vagrantfile.

This will remove the guest computer completely from the system. If there are any files on guest computer not in the /vagrant directory, they will be destroyed.

Where to go from here?

Documentation

The vagrant shell provisioner documentation is useful.

Such as:

Using external shell script instead of inlining the script in the Vagrantfile.
Using script arguments to customize the guest computer setup.

Other Guest OS

Other Guest OSes that are supported by Vagrant can be found here. Just setting the config.vm.box = 'vm box name' to use them. The current long term support version of Ubuntu, 16.04, has a name: ubuntu/xenial64 link

Other Programming Languages

A repository of Vagrantfiles for many programming language setup.

Other Provisioning

The shell provisioner is powerful enough to get a single machine up and running for a developer. More sophisticated provisioners exist and vagrant provides a great way to explore them. These include:

Conclusion

Vagrant and virtualbox form a great basis to setup another computer inside your current computer to isolate and consistently create your development environments on other computers.

With Vagrant’s scripting feature, it is easier to get a consistent setup every time so the newest version of a programming language, like python or node, is setup consistently each time.

This is why I love having vagrant, it’s an easy way to consistently create the application’s environment.

_Special thanks to abathologist and Gian Pablo for reading drafts and providing valuable suggestions.

Red Green Repeat Adventures of a Spec Driven Junkie