Scratch debugging

The main difficulty I found to convince people to consider using scratch containers is the fear to not have the good tools in case of. In fact, it is possible to deploy what you want and this is how.

Namespaces

As a container build out of cgroups, namespaces, and capabilities, it is possible to reuse namespaces of containers when creating others:

Let’s start a webserver in a scratch container:

$ docker run --rm -ti --name whoami emilevauge/whoami

We can use the network namespace of our container whoami in an other container. Thus the two containers have the same network stack and the same loopback address:

$ docker run --rm -ti --name curl --net container:whoami --name curl appropriate/curl curl 127.0.0.1:80

There are many namespaces: net, pid, user, …

Sidekicks: namespaces and capabilities

There’s a tone of resources on Internet to explain this. To make it short, our container named curl is a sidekick: it shares some namespace(s) of a previous container.

What is important is that if you share some namespaces, you still have an other container with different possibilities. For example, I can strace the webserver with this:

$ docker run --rm -ti --pid container:whoami --cap-add SYS_PTRACE debian
# apt update && apt install -qy strace
# strace -p 1

Filesystem

One of the limits of the docker containers is that you can’t share the full filesystem of the container. You can only share a directory which has already been precut as a volume.

But there is a workaround: /proc. This pseudo-filesystem can provide you all the information you need about a process:

• /proc/PID/environment for the environment variables (not updated BTW)
• /proc/PID/status for some stats
• /proc/PID/cmdline for the command line used to start the process
• /proc/PID/ns for the list of the namespace
• … see man proc …
• /proc/PID/root for the filesystem as viewed by the process.

To view the filesystem in the container, first find the pid of the process:

$ docker top whoami
UID                 PID    ...
root                22001  ...

So the filesystem of our scratch container can be visited:

# cd /proc/22001/root
# ls
dev  etc  proc  sys  whoamI

Injecting an OS

Once we have access to a filesystem we can deploy whatever we want. Like a complete OS. I chose alpine because there’s a package management system but it could be busybox.

To deploy alpine:

1. Download the alpine packet manager name apk
2. Use apk to install the basics
3. Configure the repo

All the steps are in this script so you just have to run it.

$ wget --quiet https://gist.githubusercontent.com/cell/c2771582f28bf9413b5bd81426338a1d/raw/7c910b4f20203f80888e8fd511f4a7d5824336a0/inject-alpine.sh
$ chmod a+x inject-alpine.sh
$ sudo -s
# PID=22001 ./inject-alpine.sh
...
You can now enter your formerly scratch container with: docker exec -ti container-name sh

Now we can:

$ docker exec -ti whoami sh
/ #

If you need any tool, just install it:

/ # apk --update --clean add curl

Sidekicks vs OS-injection

Both solutions have pros and cons.

Deploying an OS help to feel in a more traditional environment. It’s easier to just docker exec and you are in a normal container even if you inherit the limitation of standard containers. On the other hand, sidekicks fill stranger and stretch be you can do nearly everything . You could even deploy the OS from a sidekick :)