Services launched from Marathon in bridged networking mode cannot be discovered

Symptoms

When an application is launched in marathon with "network": "BRIDGE", Marathon passes the internal docker IP address to Mesos-DNS. The Symptoms you see are that the docker IP is not resolvable from any other application.

 

If you launch the application with "network": "HOST", the application works as expected.

Explanation

Using BRIDGE networking, Marathon assigns an external port of the slave's port range if the container is configured with portMappings.

Using host kibana.marathon.mesos (as example) would be insufficient, even if it would actually return the correct IP address.

$ host kibana.marathon.mesos
kibana.marathon.mesos has address 172.17.16.156

Workaround

The best way to use this is to query for the SRV records of the service, like below

$ dig _kibana._tcp.marathon.mesos SRV

; <<>> DiG 9.10.2-P2 <<>> _kibana._tcp.marathon.mesos SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23087
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;_kibana._tcp.marathon.mesos.   IN      SRV

;; ANSWER SECTION:
_kibana._tcp.marathon.mesos. 60 IN      SRV     0 0 31537 kibana-8711-s0.marathon.slave.mesos.

;; ADDITIONAL SECTION:
kibana-8711-s0.marathon.slave.mesos. 60 IN A    192.168.200.162

;; Query time: 0 msec
;; SERVER: 192.168.200.162#53(192.168.200.162)
;; WHEN: Wed Nov 11 13:51:23 UTC 2015
;; MSG SIZE  rcvd: 146

In the ANSWER SECTION, one would find the port (here 31537), and in the ADDITIONAL SECTION the IP address (here 192.168.200.162).

By mapping the "hostname" kibana-8711-s0.marathon.slave.mesos. between both sections, one can derive the actual host:port endpoint where the service lives (here 192.168.200.162:31537).

Resolution

Mesos 0.27 introduces a new field in state.json to determine whether Docker containers are bridged or not. The NetworkInfo IP provider can reasonably use this information to make better decisions about which IP to provide. This information will be exposed as part of Mesos 0.27/DCOS1.5, which is available now.

Have more questions? Submit a request

Comments

Powered by Zendesk