Thursday, April 24, 2014

Accessing docker container private network easily from your boot2docker host

I wrote a blog post a while back about how I was running Acme Air in docker along with Cassandra.  This setup has become far more complex and I wanted to stop doing port mapping specifically for every container.  Here are the simple steps to get all the ports and ip's to route cleanly from a hosting system (Mac OS X, but windows works the same). to all of your docker containers.

So my setup is:

- Macbook Pro laptop running Mac OS X 10.9.2
- VirtualBox 4.3.10
- Boot2docker 0.8.0
- Docker 0.10.0

To help understand the concept I'll communicate with a "server" on a container that is listening on a TCP port.  To demonstrate, I'll use the netcat tool listening on port 3333 on a base ubuntu image.  The goal is to be able to telnet directly to that port from my base laptop.  Using netcat is just an example.  Once this works any server listening on any port should be just as easy to access.

To help understand the below terminal sessions, my laptop's hostname is "ispyker", my docker vm running on VirtualBox's hostname is "boot2docker" and containers usually have hostnames like "e79e432696f7".

First, let's go ahead and run the netcat/unbuntu container:

ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ docker run -t -i ubuntu /bin/bash
root@e79e432696f7:/# /sbin/ifconfig eth0 |grep addr
eth0      Link encap:Ethernet  HWaddr 7e:20:1b:29:bb:b6  
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::7c20:1bff:fe29:bbb6/64 Scope:Link
root@e79e432696f7:/# nc -l 3333

Now, on another Mac OS terminal:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

Ok, so let's fix this ...
ispyker:~ aspyker$ ~/bin/boot2docker stop
[2014-04-24 13:19:16] Shutting down boot2docker-vm...

First, we need to open up the VirtualBox application from finder. From the menu, select:

VirtualBox->Preferences->Network->Host-only Networks

Either edit an existing or create a network called "vboxnet0" with the following settings:

Under adapter:

IPv4 Address: 172.16.0.1
IPv4 Network Mask: 255.255.0.0
IPv6 Address: (blank)
IPv6 Network Mask: 0

Under DHCP server:

Uncheck "Enable Server"

Next, right click the "boot2docker-vm" and select:

Settings->Network

Create an Adapter 2 with the following settings:

Check Enable Network Adapter
Attached to: Host-only Adapter
Name: vboxnet0
Advanced:
Adapter Type: Intel Pro/1000 MT Desktop
Promiscuous Mode: Deny Mac
Address: (use the default)
Enable Cable Connected

Save all your settings and let's start back up that netcat/ubuntu container:
ispyker:~ aspyker$ ~/bin/boot2docker up
[2014-04-24 13:27:12] Starting boot2docker-vm...
[2014-04-24 13:27:32] Started.
ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ docker run -i -t ubuntu /bin/bash
root@1560f377bf4a:/# netcat -l 3333

We still at this point won't be able to "see" this port from MacOS, as we haven't yet assigned an IP address to the boot2docker VM nor have we created a route from MacOS to the docker host-only network.

Let's test that to be sure:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

First, let's add an IP address to the host-only network for this new interface on the boot2docker VM:
ispyker:~ aspyker$ ~/bin/boot2docker ssh
docker@localhost's password: 
                        ##        .
                  ## ## ##       ==
               ## ## ## ##      ===
           /""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
           \______ o          __/
             \    \        __/
              \____\______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
boot2docker: 0.8.0
docker@boot2docker:~$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:DC:5A:BA  
          inet6 addr: fe80::a00:27ff:fedc:5aba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:11703 (11.4 KiB)

docker@boot2docker:~$ sudo ifconfig eth1 172.16.0.11
docker@boot2docker:~$ sudo ifconfig eth1 netmask 255.255.0.0
docker@boot2docker:~$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 08:00:27:DC:5A:BA  
          inet addr:172.16.0.11  Bcast:172.16.255.255  Mask:255.255.0.0
          inet6 addr: fe80::a00:27ff:fedc:5aba/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:15723 (15.3 KiB)


At this point, you should be able to ping your boot2docker VM on it's new ip address from your Mac:
ispyker:~ aspyker$ ping -c 1 172.16.0.11
PING 172.16.0.11 (172.16.0.11): 56 data bytes
64 bytes from 172.16.0.11: icmp_seq=0 ttl=64 time=0.349 ms

--- 172.16.0.11 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.349/0.349/0.349/0.000 ms

However, you still can't get to the netcat container port:
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
telnet: connect to address 172.17.0.2: Operation timed out
telnet: Unable to connect to remote host

Now, we'll add the route to the hosting Mac OS:
ispyker:~ aspyker$ netstat -nr |grep 172\.17
ispyker:~ aspyker$ sudo route -n add 172.17.0.0/16 172.16.0.11
Password:
add net 172.17.0.0: gateway 172.16.0.11
ispyker:~ aspyker$ netstat -nr |grep 172\.17
172.17             172.16.0.11        UGSc            0        0 vboxnet
ispyker:~ aspyker$ telnet 172.17.0.2 3333
Trying 172.17.0.2...
Connected to 172.17.0.2.
Escape character is '^]'.
hello container world

If you followed along correctly, and typed "hello container world" once telnet connects, "hello container world" should have been printed out in your ubuntu/netcat container. At this point you should be able to access any container's ip address and ports.  You can get the IP address of any container by running docker inspect [containername] looking for it's 172.17.0.x address.

Welcome to your easier local host-only fully TCP accessible cloud.
Thanks to Takahiro Inaba for helping put this together.

14 comments:

  1. Nice approach, thanks for sharing. I'd been using a pipeworks-esque method of moving the H-O network address from eth1 over to docker0 and then bridging eth1 to docker0. Had mixed results, likely due to my lack of linux networking skills. This method looks easy enough to put into an inline Vagrant shell provisioner to automate the process of adding the routes.

    ReplyDelete
  2. Works well, is straight forward and provides a flexible working environment. Any other reasons not to simply expose ports in the Dockerfiles, and then access using port-forwarding?

    ReplyDelete
  3. Robbie, port forwarding to the host works, but get crazy as you start to access alot of ports. This approach gives you pretty much full network access.

    ReplyDelete
  4. There's an even easier way that works in the latest version of boot2docker, no extra interfaces required:

    sudo route -n add 172.17.0.0/16 `boot2docker ip`

    ReplyDelete
  5. @Dustin, thanks for the info. Personally, I'm not a huge fan of boot2docker as I like a more fully functioned host for the containers. But for b2d folks, sounds good.

    ReplyDelete
  6. For ubuntu, you can do:

    # The secondary network interface
    auto eth1
    iface eth1 inet static
    address 172.16.0.254
    netmask 255.255.0.0

    ReplyDelete
  7. For me it does not work:

    END SSH
    Error requesting socket: No IP address found 4: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:db:d2:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:fedb:d271/64 scope link
    valid_lft forever preferred_lft forever

    ReplyDelete
  8. Everything works for me, but how can I set ip address to container?
    Because when I try to do that "docker run -p 172.17.0.10:80:80 -t image" it throws Error starting userland proxy: listen tcp 172.17.0.10:80: bind: cannot assign requested address

    ReplyDelete
  9. There are alternatives to it, I've developed an http(s) proxy to do that, check out this blog post for more information and instructions of how to do it http://blog.nunes.io/2015/05/02/how-to-access-docker-containers-from-external-devices.html

    ReplyDelete
  10. Thanks a lot. After configure host-only network, add route to container ip range with host only ip of guest os.

    ReplyDelete
  11. Thank you so much for this post. Its taken me about a week of messing around trying everything to get this working.

    ReplyDelete
  12. One question, now that Docker support the ability to specify an IP address providing its on a user defined network. e.g.


    # create a new bridge network with your subnet and gateway for your ip block
    $ docker network create --subnet 203.0.113.0/24 --gateway 203.0.113.254 iptastic

    # run a nginx container with a specific ip in that block
    $ docker run --rm -it --net iptastic --ip 203.0.113.2 nginx

    # curl the ip from any other place (assuming this is a public ip block duh)
    $ curl 203.0.113.2

    What would I need to do to get this to work ?

    ReplyDelete
  13. TL;DR:
    subnetcalc 172.17.0.1 255.255.0.0
    sudo route -n add 172.17.0.0/16 192.168.99.100 # latter is VM docker host ip accessible from localhost

    ReplyDelete
  14. Awesome! Can now see fully dockerized Cassandra cluster. Cheers

    ReplyDelete