Red Green Repeat Adventures of a Spec Driven Junkie

Help - I can't ssh into My VirtualBox!

This is a post on what happened to me recently when suddenly I couldn’t ssh into my VirtualBox created by vagrant. I will share steps I did to get locked out and two solutions that might help you. By reading about my experience, you will learn why NOT to mess with the .ssh folder in VirtualBoxes set up by vagrant. All of this will only take less than five minutes of your time!

Sutra box with dragons amid clouds source and more details

Introduction

I got locked out of my VirtualBox box last week.

How is that even possible?!

  • Vagrant configured the VirtualBox
  • The VirtualBox was never accessible to the Internet
  • I only installed trusted open source software that did not have any recent exploits
  • No one else had access to my computer and VirtualBox

Steps I Took Before

The last thing I remember doing before I lost access to the VirtualBox, is that I was transferring SSH keys between a new computer’s VirtualBox and an old computer’s VirtualBox.

I got another computer and instead of copying the VirtualBox’s disk image, I recreated the VirtualBox with vagrant. I needed the SSH keys from the other computer’s VirtualBox to access external servers, like Github.

To do this, I copied over the ssh key-pairs by creating an archive including all the files in the .ssh folder of my home directory on the old computer’s VirtualBox using command:

$ tar -cvf ssh_files.tar ~/.ssh

and after transferring onto the new computer’s VirtualBox box, I ran the following command to expand the archive:

$ tar -zxvf ssh_files.tar

When I shutdown the new computer’s VirtualBox and started it up again, I saw the following messages from vagrant:

$ vagrant up
...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Authentication failure. Retrying...
    ...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
    default: Warning: Authentication failure. Retrying...
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

What the??? How did that happen?

Root Cause

Well, remember when I copied over those SSH key files using tar and included the whole directory?

This command adds everything in the folder to the archive, including the authorized_keys file in the directory.

authorized_keys file usage

The authorized_keys file is responsible for allowing system login using ssh key-pairs instead of requiring username and password . The file contains public keys and when a user wants to be able to login using ssh key-pair instead of username password, one would insert their public key into this file.

When logging in using ssh, the remote computer would use the equivalent private key for authorization. The ssh daemon would check this authorization matches one of the keys in the authorized_keys file.

As long as a key in that file is the corresponding private key, the ssh daemon grants access and the user can login. If not, the daemon rejects the requests.

vagrant and authorized_keys

When vagrant creates a new VirtualBox, it sets up a ssh key-pair, putting the public key into the authorized_keys file. This happens auto-magically and users don’t even know the key-pair generated.

If one loses this key-pair, the VirtualBox configured by vagrant will be basically inaccessible.

Eureka!

That’s exactly what happened when I copied over the ssh key-pairs from the old computer’s VirtualBox: it included the authorized_keys file, which has another key. On expansion, overwrote the new computer’s file, changing the key-pair combination.

So, that’s how I got locked out of my own VirtualBox, how can I get back in??

Solution: Copy the Private Key

The easiest solution if one is in a similar situation: copy over the private key!

This requires the original private key to be accessible to the user. In my case, it was still on the old computer I was transferring items from.

One can find the location of the private key by using the vagrant ssh-config command:

$ vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/<username>/<virtualbox directory>/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

The value of identityfile is the location on the host computer of the private key file. In the above case, the value is:

/Users/<username>/<virtualbox directory>/.vagrant/machines/default/virtualbox/private_key

Copy the private key over and replace it using command:

$ cp <location of new private key>/id_rsa /Users/<username>/<virtualbox directory>/.vagrant/machines/default/virtualbox/private_key

Whew - that saved some work in rebuilding a VirtualBox.

Solution: Replace Key-Pairs

Another solution: generate a new key-pair and update the private key and authorized_keys file with the new public key value.

This only works if there is still access into the VirtualBox through another connection (i.e. in a open terminal tab). If the VirtualBox has been shutdown or one has closed all terminal connections to the VirtualBox, the VirtualBox is basically inaccessible.

If there’s a terminal connection open:

  1. Create a new ssh key-pair: ssh-keygen
  2. Copy over the private key, like in the last step
  3. Copy over the public key, into the authorized_keys file

This happened to me but I wasn’t lucky enough to have a terminal window open, hence this article.

Lessons

I learned the following from a simple “copy over the ssh directory”:

  • Do not blindly copy/paste the .ssh folder from one computer to another. Only take what’s necessary and nothing more.
  • Vagrant does a lot of work behind the scenes to set up a VirtualBox for you, even setting up SSH key-pairs for the host and guest VirtualBoxes. It’s impressive how this happens seamlessly with different operating systems.
  • Always keep a terminal open to your VirtualBox. It might save you from rebuilding the VirtualBox again.
  • Shutting down a VirtualBox is not worth it, unless you are absolutely OK with throwing it away.

I was lucky that this happened on a VirtualBox I controlled. If this happened on a production server that I have no physical access or know another user with administrative privileges, it would not be a fun time.

I will be careful the next time I am working with ssh keys, messing up can have dire consequences!