Red Green Repeat Adventures of a Spec Driven Junkie

Networking Lesson .1

TL;DR

Do not assign IP address ending in .1, even on a private network.

I learned a valuable networking lesson when setting up multiple virtual boxes within a private network.

This article documents the process of using virtual boxes in private networking mode, methods to testing network connectivity, finding HTTP connection issue and the resolution.

Requirements

If you would like to replicate my work, install these items:

Set up

I want to set up four virtual machines using vagrant on a private network. I set up four Vagrantfiles in their own folders as so:

.
├── host1
│   └── Vagrantfile
├── host2
│   └── Vagrantfile
├── host3
│   └── Vagrantfile
└── host4
    └── Vagrantfile

The Vagrantfile contents:

HOST_IP="192.168.100.1"

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/xenial64"

  config.vm.network "private_network", ip: HOST_IP

  config.vm.provision "shell", inline: <<-SHELL
     apt-get install -y \
             apt-transport-https \
             ca-certificates \
             curl \
             nginx \
             software-properties-common

     /etc/init.d/nginx start
  SHELL
end

With four hosts on a private network, I would like each host to correspond to a logical IP address, such that: host1 is 192.168.100.1, host2 is 192.168.100.2, etc.

To configure this, the main difference between each hosts’ Vagrantfile is:

host IP Address configuration
host1 192.168.100.1 HOST_IP="192.168.100.1"
host2 192.168.100.2 HOST_IP="192.168.100.2"
host3 192.168.100.3 HOST_IP="192.168.100.3"
host4 192.168.100.4 HOST_IP="192.168.100.4"

This looks straight-forward, let’s bring each up by running command:

$ cd host1 && vagrant up
$ cd host2 && vagrant up
$ cd host3 && vagrant up
$ cd host4 && vagrant up

Ping Test

To confirm each host can reach the other, let’s have them ping ech other with the following commands:

$ cd host1 && vagrant ssh -c 'ping 192.168.100.2'
$ cd host1 && vagrant ssh -c 'ping 192.168.100.3'
$ cd host1 && vagrant ssh -c 'ping 192.168.100.4'

$ cd host2 && vagrant ssh -c 'ping 192.168.100.1'
$ cd host2 && vagrant ssh -c 'ping 192.168.100.3'
$ cd host2 && vagrant ssh -c 'ping 192.168.100.4'

$ cd host3 && vagrant ssh -c 'ping 192.168.100.1'
$ cd host3 && vagrant ssh -c 'ping 192.168.100.2'
$ cd host3 && vagrant ssh -c 'ping 192.168.100.4'

$ cd host4 && vagrant ssh -c 'ping 192.168.100.1'
$ cd host4 && vagrant ssh -c 'ping 192.168.100.2'
$ cd host4 && vagrant ssh -c 'ping 192.168.100.3'

The outputs of each command have the general shape:

host3$ vagrant ssh -c 'ping 192.168.100.4'
PING 192.168.100.4 (192.168.100.4) 56(84) bytes of data.
64 bytes from 192.168.100.4: icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from 192.168.100.4: icmp_seq=2 ttl=64 time=0.036 ms
^C
--- 192.168.100.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.031/0.033/0.036/0.006 ms
Connection to 127.0.0.1 closed.

Each host can reach the other using ping. Great!

HTTP Test

With ping confirming each host can reach the other, I expect each host is connectable to the other over HTTP.

Let’s run a quick test since each host has nginx and the server is running.

$ cd host1 && vagrant ssh -c 'curl 192.168.100.2'
$ cd host2 && vagrant ssh -c 'curl 192.168.100.1'
$ cd host3 && vagrant ssh -c 'curl 192.168.100.1'
$ cd host4 && vagrant ssh -c 'curl 192.168.100.1'

A successful output of the curl command in this situation would be curl retrieving the default nginx page::

vagrant@ubuntu-xenial:~$ curl localhost:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

HTTP Test Results

host1:

command output
$ cd host1 && vagrant ssh -c ‘curl 192.168.100.2’ successful nginx output
$ cd host1 && vagrant ssh -c ‘curl 192.168.100.3’ successful nginx output
$ cd host1 && vagrant ssh -c ‘curl 192.168.100.4’ successful nginx output
command output
$ cd host2 && vagrant ssh -c ‘curl 192.168.100.1’ successful nginx output
$ cd host2 && vagrant ssh -c ‘curl 192.168.100.3’ successful nginx output
$ cd host2 && vagrant ssh -c ‘curl 192.168.100.4’ successful nginx output
command output
$ cd host3 && vagrant ssh -c ‘curl 192.168.100.1’ successful nginx output
$ cd host3 && vagrant ssh -c ‘curl 192.168.100.2’ successful nginx output
$ cd host3 && vagrant ssh -c ‘curl 192.168.100.4’ successful nginx output
command output
$ cd host4 && vagrant ssh -c ‘curl 192.168.100.1’ connection error
$ cd host4 && vagrant ssh -c ‘curl 192.168.100.2’ successful nginx output
$ cd host4 && vagrant ssh -c ‘curl 192.168.100.3’ successful nginx output

In the siutations where there is a connection error, the output is:

host4$ vagrant ssh -c 'curl 192.168.100.1:80'
curl: (7) Failed to connect to 192.168.100.1 port 80: No route to host
Connection to 127.0.0.1 closed.

Ping Test Again

This HTTP connection failure makes me wonder and I directly confirm using ping again:

Pinging from host4 => host1:

host4$ vagrant ssh -c 'ping 192.168.100.1'
PING 192.168.100.1 (192.168.100.1) 56(84) bytes of data.
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=0.373 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=0.227 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=0.217 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=0.217 ms
64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=0.223 ms
64 bytes from 192.168.100.1: icmp_seq=6 ttl=64 time=0.221 ms
^C
--- 192.168.100.1 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 0.217/0.246/0.373/0.058 ms
Connection to 127.0.0.1 closed.

Hmm… this is strange, why can host1, host2, and host3 connect to each other over HTTP because the curl request succeeded but host4 and host1 cannot? All the other hosts can connect to each other, except these two?

Curl Details

When I run curl -v, to get more verbose output, this is the response:

host4$ vagrant ssh -c 'curl 192.168.100.1:80 -v'
* Rebuilt URL to: 192.168.100.1:80/
*   Trying 192.168.100.1...
* connect to 192.168.100.1 port 80 failed: Connection refused
* Failed to connect to 192.168.100.1 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 192.168.100.1 port 80: Connection refused
Connection to 127.0.0.1 closed.

This does not add up… hosts1 refuses the HTTP connection from host3, but the ping connection works??

Other Possible Solutions??

I tried solutions such as flushing the firewall entries (which would deny connections from other computers). That did not work, this makes sense since I did not do anything to trigger an entry to the firewall.

I destroyed everything and rebuilt each virtual machine, when I saw this message during the build for host1:

==> default: You assigned a static IP ending in ".1" to this machine.
==> default: This is very often used by the router and can cause the
==> default: network to not work properly. If the network doesn't work
==> default: properly, try changing this IP.

face palm

Oh, now I get it… any network address ending in .1 is a router address.

In my quest of have matching hostname numbers with the actual address, I inadvertently assigned a host’s address to the router address.

The really funky part about debugging this was that ping between all hosts worked, but curl worked between host1 and host2, which made the issue weirder.

Conclusion

The ultimate goal of the lesson: do not set the address of anything to .1. It’s just not worth the debugging headache. :-)