This page describes how to troubleshoot and debug the daemon if you run intoissues.

You can turn on debugging on the daemon to learn about the runtime activity ofthe daemon and to aid in troubleshooting. If the daemon is unresponsive, you canalsoforce a full stack trace of allthreads to be added to the daemon log by sending theSIGUSR signal to theDocker daemon.

Daemon

Unable to connect to the Docker daemon

Cannot connect to the Docker daemon. Is 'docker daemon' running on this host?

This error may indicate:

• The Docker daemon isn't running on your system. Start the daemon and tryrunning the command again.
• Your Docker client is attempting to connect to a Docker daemon on a differenthost, and that host is unreachable.

Check whether Docker is running

The operating-system independent way to check whether Docker is running is toask Docker, using thedocker info command.

You can also use operating system utilities, such assudo systemctl is-active docker orsudo status docker orsudo service docker status, or checking the service status using Windowsutilities.

Finally, you can check in the process list for thedockerd process, usingcommands likeps ortop.

Check which host your client is connecting to

To see which host your client is connecting to, check the value of theDOCKER_HOST variable in your environment.

$ env| grep DOCKER_HOST

If this command returns a value, the Docker client is set to connect to a Dockerdaemon running on that host. If it's unset, the Docker client is set to connectto the Docker daemon running on the local host. If it's set in error, use thefollowing command to unset it:

$unset DOCKER_HOST

You may need to edit your environment in files such as~/.bashrc or~/.profile to prevent theDOCKER_HOST variable from being set erroneously.

IfDOCKER_HOST is set as intended, verify that the Docker daemon is running onthe remote host and that a firewall or network outage isn't preventing you fromconnecting.

Troubleshoot conflicts between thedaemon.json and startup scripts

If you use adaemon.json file and also pass options to thedockerd commandmanually or using start-up scripts, and these options conflict, Docker fails tostart with an error such as:

unable to configure the Docker daemon with file /etc/docker/daemon.json:the following directives are specified both as a flag and in the configurationfile: hosts: (from flag: [unix:///var/run/docker.sock], from file: [tcp://127.0.0.1:2376])

If you see an error similar to this one and you are starting the daemon manuallywith flags, you may need to adjust your flags or thedaemon.json to remove theconflict.

Note

If you see this specific error message abouthosts, continue to thenext sectionfor a workaround.

If you are starting Docker using your operating system's init scripts, you mayneed to override the defaults in these scripts in ways that are specific to theoperating system.

Configure the daemon host with systemd

One notable example of a configuration conflict that's difficult totroubleshoot is when you want to specify a different daemon address from thedefault. Docker listens on a socket by default. On Debian and Ubuntu systemsusingsystemd, this means that a host flag-H is always used when startingdockerd. If you specify ahosts entry in thedaemon.json, this causes aconfiguration conflict and results in the Docker daemon failing to start.

To work around this problem, create a new file/etc/systemd/system/docker.service.d/docker.conf with the following contents,to remove the-H argument that's used when starting the daemon by default.

[Service]ExecStart=ExecStart=/usr/bin/dockerd

There are other times when you might need to configuresystemd with Docker,such asconfiguring a HTTP or HTTPS proxy.

Note

If you override this option without specifying ahosts entry in thedaemon.json or a-H flag when starting Docker manually, Docker fails tostart.

Runsudo systemctl daemon-reload before attempting to start Docker. If Dockerstarts successfully, it's now listening on the IP address specified in thehosts key of thedaemon.json instead of a socket.

Important

Settinghosts in thedaemon.json isn't supported on DockerDesktop for Windows or Docker Desktop for Mac.

Out of memory issues

If your containers attempt to use more memory than the system has available, youmay experience an Out of Memory (OOM) exception, and a container, or the Dockerdaemon, might be stopped by the kernel OOM killer. To prevent this fromhappening, ensure that your application runs on hosts with adequate memory andseeUnderstand the risks of running out of memory.

Kernel compatibility

Docker can't run correctly if your kernel is older than version 3.10, or if it'smissing kernel modules. To check kernel compatibility, you can download and runthecheck-config.shscript.

$ curl https://raw.githubusercontent.com/docker/docker/master/contrib/check-config.sh > check-config.sh$ bash ./check-config.sh

The script only works on Linux.

Kernel cgroup swap limit capabilities

On Ubuntu or Debian hosts, you may see messages similar to the following whenworking with an image.

WARNING: Your kernel does not support swap limit capabilities. Limitation discarded.

If you don't need these capabilities, you can ignore the warning.

You can turn on these capabilities on Ubuntu or Debian by following theseinstructions. Memory and swap accounting incur an overhead of about 1% of thetotal available memory and a 10% overall performance degradation, even whenDocker isn't running.

1. Log into the Ubuntu or Debian host as a user withsudo privileges.

2. Edit the/etc/default/grub file. Add or edit theGRUB_CMDLINE_LINUX lineto add the following two key-value pairs:

   GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

   Save and close the file.

3. Update the GRUB boot loader.

   $ sudo update-grub

   An error occurs if your GRUB configuration file has incorrect syntax. In thiscase, repeat steps 2 and 3.

   The changes take effect when you reboot the system.

Networking

IP forwarding problems

If you manually configure your network usingsystemd-network with systemdversion 219 or later, Docker containers may not be able to access your network.Beginning with systemd version 220, the forwarding setting for a given network(net.ipv4.conf.<interface>.forwarding) defaults to off. This setting preventsIP forwarding. It also conflicts with Docker's behavior of enabling thenet.ipv4.conf.all.forwarding setting within containers.

To work around this on RHEL, CentOS, or Fedora, edit the<interface>.networkfile in/usr/lib/systemd/network/ on your Docker host, for example,/usr/lib/systemd/network/80-container-host0.network.

Add the following block within the[Network] section.

[Network]...IPForward=kernel# ORIPForward=true

This configuration allows IP forwarding from the container as expected.

DNS resolver issues

DNS resolver found in resolv.conf and containers can't use it

Linux desktop environments often have a network manager program running, thatusesdnsmasq to cache DNS requests by adding them to/etc/resolv.conf. Thednsmasq instance runs on a loopback address such as127.0.0.1 or127.0.1.1. It speeds up DNS look-ups and provides DHCP services. Such aconfiguration doesn't work within a Docker container. The Docker container usesits own network namespace, and resolves loopback addresses such as127.0.0.1to itself, and it's unlikely to be running a DNS server on its own loopbackaddress.

If Docker detects that no DNS server referenced in/etc/resolv.conf is a fullyfunctional DNS server, the following warning occurs:

WARNING: Local (127.0.0.1) DNS resolver found in resolv.conf and containerscan't use it. Using default external servers : [8.8.8.8 8.8.4.4]

If you see this warning, first check to see if you usednsmasq:

$ ps aux| grep dnsmasq

If your container needs to resolve hosts which are internal to your network, thepublic nameservers aren't adequate. You have two choices:

• Specify DNS servers for Docker to use.

• Turn offdnsmasq.

  Turning offdnsmasq adds the IP addresses of actual DNS nameservers to/etc/resolv.conf, and you lose the benefits ofdnsmasq.

You only need to use one of these methods.

Specify DNS servers for Docker

The default location of the configuration file is/etc/docker/daemon.json. Youcan change the location of the configuration file using the--config-filedaemon flag. The following instruction assumes that the location of theconfiguration file is/etc/docker/daemon.json.

1. Create or edit the Docker daemon configuration file, which defaults to/etc/docker/daemon.json file, which controls the Docker daemonconfiguration.

   $ sudo nano /etc/docker/daemon.json

2. Add adns key with one or more DNS server IP addresses as values.

   {"dns":["8.8.8.8","8.8.4.4"]}

   If the file has existing contents, you only need to add or edit thednsline. If your internal DNS server can't resolve public IP addresses, includeat least one DNS server that can. Doing so allows you to connect to DockerHub, and your containers to resolve internet domain names.

   Save and close the file.

3. Restart the Docker daemon.

   $ sudo service docker restart

4. Verify that Docker can resolve external IP addresses by trying to pull animage:

   $ docker pull hello-world

5. If necessary, verify that Docker containers can resolve an internal hostnameby pinging it.

   $ docker run --rm -it alpine ping -c4 <my_internal_host>PING google.com (192.168.1.2): 56 data bytes64 bytes from 192.168.1.2: seq=0 ttl=41 time=7.597 ms64 bytes from 192.168.1.2: seq=1 ttl=41 time=7.635 ms64 bytes from 192.168.1.2: seq=2 ttl=41 time=7.660 ms64 bytes from 192.168.1.2: seq=3 ttl=41 time=7.677 ms

Turn offdnsmasq

Docker networks disappearing

If a Docker network, such as thedocker0 bridge or a custom network, randomlydisappears or otherwise appears to be working incorrectly, it could be becauseanother service is interfering with or modifying Docker interfaces. Tools thatmanage networking interfaces on the host are known to sometimes alsoinappropriately modify Docker interfaces.

Refer to the following sections for instructions on how to configure yournetwork manager to set Docker interfaces as un-managed, depending on thenetwork management tools that exist on the host:

• Ifnetscript is installed, consideruninstalling it
• Configure the network manager totreat Docker interfaces as un-managed
• If you're using Netplan, you may need toapply a custom Netplan configuration

Uninstallnetscript

Ifnetscript is installed on your system, you can likely fix this issue byuninstalling it. For example, on a Debian-based system:

$ sudo apt-get remove netscript-2.4

Un-manage Docker interfaces

In some cases, the network manager will attempt to manage Docker interfaces bydefault. You can try to explicitly flag Docker networks as un-managed byediting your system's network configuration settings.

Prevent Netplan from overriding network configuration

On systems that useNetplan throughcloud-init, you mayneed to apply a custom configuration to preventnetplan from overriding thenetwork manager configuration:

1. Follow the steps inUn-manage Docker interfacesfor creating the network manager configuration.

2. Create anetplan configuration file under/etc/netplan/50-cloud-init.yml.

   The following example configuration file is a starting point.Adjust it to match the interfaces you want to un-manage.Incorrect configuration can lead to network connectivity issues.

   /etc/netplan/50-cloud-init.yml

   network:ethernets:all:dhcp4:truedhcp6:truematch:# edit this filter to match whatever makes sense for your systemname:en*renderer:networkdversion:2

3. Apply the new Netplan configuration.

   $ sudo netplan apply

4. Restart the Docker daemon:

   $ sudo systemctl restart docker

5. Verify that the Docker interfaces have theunmanaged state.

   $ networkctl

Volumes

Unable to remove filesystem

Error: Unable to remove filesystem

Some container-based utilities, suchasGoogle cAdvisor, mount Docker systemdirectories, such as/var/lib/docker/, into a container. For instance, thedocumentation forcadvisor instructs you to run thecadvisor container asfollows:

$ sudo docker run\  --volume=/:/rootfs:ro \  --volume=/var/run:/var/run:rw \  --volume=/sys:/sys:ro \  --volume=/var/lib/docker/:/var/lib/docker:ro \  --publish=8080:8080 \  --detach=true \  --name=cadvisor \  google/cadvisor:latest

When you bind-mount/var/lib/docker/, this effectively mounts all resources ofall other running containers as filesystems within the container which mounts/var/lib/docker/. When you attempt to remove any of these containers, theremoval attempt may fail with an error like the following:

Error: Unable to remove filesystem for74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515:remove /var/lib/docker/containers/74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515/shm:Device or resource busy

The problem occurs if the container which bind-mounts/var/lib/docker/usesstatfs orfstatfs on filesystem handles within/var/lib/docker/and does not close them.

Typically, we would advise against bind-mounting/var/lib/docker in this way.However,cAdvisor requires this bind-mount for core functionality.

If you are unsure which process is causing the path mentioned in the error tobe busy and preventing it from being removed, you can use thelsof commandto find its process. For instance, for the error above:

$ sudo lsof /var/lib/docker/containers/74bef250361c7817bee19349c93139621b272bc8f654ae112dd4eb9652af9515/shm

To work around this problem, stop the container which bind-mounts/var/lib/docker and try again to remove the other container.