Thursday, April 24, 2014

Accessing docker container private network easily from your boot2docker host

I wrote a blog post a while back about how I was running Acme Air in docker along with Cassandra.  This setup has become far more complex and I wanted to stop doing port mapping specifically for every container.  Here are the simple steps to get all the ports and ip's to route cleanly from a hosting system (Mac OS X, but windows works the same). to all of your docker containers.

So my setup is:

- Macbook Pro laptop running Mac OS X 10.9.2
- VirtualBox 4.3.10
- Boot2docker 0.8.0
- Docker 0.10.0

To help understand the concept I'll communicate with a "server" on a container that is listening on a TCP port.  To demonstrate, I'll use the netcat tool listening on port 3333 on a base ubuntu image.  The goal is to be able to telnet directly to that port from my base laptop.  Using netcat is just an example.  Once this works any server listening on any port should be just as easy to access.

To help understand the below terminal sessions, my laptop's hostname is "ispyker", my docker vm running on VirtualBox's hostname is "boot2docker" and containers usually have hostnames like "e79e432696f7".

First, let's go ahead and run the netcat/unbuntu container:

ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ docker run -t -i ubuntu /bin/bash
root@e79e432696f7:/# /sbin/ifconfig eth0 |grep addr
eth0      Link encap:Ethernet  HWaddr 7e:20:1b:29:bb:b6  
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::7c20:1bff:fe29:bbb6/64 Scope:Link
root@e79e432696f7:/# nc -l 3333

Now, on another Mac OS terminal:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

Ok, so let's fix this ...
ispyker:~ aspyker$ ~/bin/boot2docker stop
[2014-04-24 13:19:16] Shutting down boot2docker-vm...

First, we need to open up the VirtualBox application from finder. From the menu, select:

VirtualBox->Preferences->Network->Host-only Networks

Either edit an existing or create a network called "vboxnet0" with the following settings:

Under adapter:

IPv4 Address: 172.16.0.1
IPv4 Network Mask: 255.255.0.0
IPv6 Address: (blank)
IPv6 Network Mask: 0

Under DHCP server:

Uncheck "Enable Server"

Next, right click the "boot2docker-vm" and select:

Settings->Network

Create an Adapter 2 with the following settings:

Check Enable Network Adapter
Attached to: Host-only Adapter
Name: vboxnet0
Advanced:
Adapter Type: Intel Pro/1000 MT Desktop
Promiscuous Mode: Deny Mac
Address: (use the default)
Enable Cable Connected

Save all your settings and let's start back up that netcat/ubuntu container:
ispyker:~ aspyker$ ~/bin/boot2docker up
[2014-04-24 13:27:12] Starting boot2docker-vm...
[2014-04-24 13:27:32] Started.
ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ docker run -i -t ubuntu /bin/bash
root@1560f377bf4a:/# netcat -l 3333

We still at this point won't be able to "see" this port from MacOS, as we haven't yet assigned an IP address to the boot2docker VM nor have we created a route from MacOS to the docker host-only network.

Let's test that to be sure:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

First, let's add an IP address to the host-only network for this new interface on the boot2docker VM:
ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:DC:5A:BA  
          inet6 addr: fe80::a00:27ff:fedc:5aba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:11703 (11.4 KiB)

docker@boot2docker:~$ sudo ifconfig eth1 172.16.0.11
docker@boot2docker:~$ sudo ifconfig eth1 netmask 255.255.0.0
docker@boot2docker:~$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:DC:5A:BA  
          inet addr:172.16.0.11  Bcast:172.16.255.255  Mask:255.255.0.0
          inet6 addr: fe80::a00:27ff:fedc:5aba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:15723 (15.3 KiB)


At this point, you should be able to ping your boot2docker VM on it's new ip address from your Mac:
ispyker:~ aspyker$ ping -c 1 172.16.0.11
PING 172.16.0.11 (172.16.0.11): 56 data bytes
64 bytes from 172.16.0.11: icmp_seq=0 ttl=64 time=0.349 ms

--- 172.16.0.11 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.349/0.349/0.349/0.000 ms

However, you still can't get to the netcat container port:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

Now, we'll add the route to the hosting Mac OS:
ispyker:~ aspyker$ netstat -nr |grep 172\.17
ispyker:~ aspyker$ sudo route -n add 172.17.0.0/16 172.16.0.11
Password:
add net 172.17.0.0: gateway 172.16.0.11
ispyker:~ aspyker$ netstat -nr |grep 172\.17
172.17             172.16.0.11        UGSc            0        0 vboxnet
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
Connected to 172.17.0.2.
Escape character is '^]'.
hello container world

If you followed along correctly, and typed "hello container world" once telnet connects, "hello container world" should have been printed out in your ubuntu/netcat container. At this point you should be able to access any container's ip address and ports.  You can get the IP address of any container by running docker inspect [containername] looking for it's 172.17.0.x address.

Welcome to your easier local host-only fully TCP accessible cloud.
Thanks to Takahiro Inaba for helping put this together.