Repository files navigation

Run your favourite virtual routers in docker for convenient labbing,development and testing.

vrnetlab is being developed for the TeraStream project at Deutsche Telekom aspart of an automated CI environment for testing our network provisioningsystem.

It supports:

• Arista vEOS
• Cisco CSR1000v
• Cisco Nexus NX-OS (using Titanium emulator)
• Cisco XRv
• Cisco XRv 9000
• Juniper vMX
• Juniper vQFX
• Nokia VSR

I talk a little about it during my presentation about TeraStream testing atthe NetNod autumn meeting 2016 -https://youtu.be/R_vCdGkGeSk?t=9m25s

Brian Linkletter has written a good introduction toohttps://www.brianlinkletter.com/vrnetlab-emulate-networks-using-kvm-and-docker/

You have to build the virtual router docker images yourself since the licenseagreements of commercial virtual routers do not allow me to distribute theimages. See the README files of the respective virtual router types for moredetails.

You need KVM enabled in your kernel for hardware assisted virtualization. Whileit may be possible to run without it, it has not been tested. Make sure youload the kvm kernel module:modprobe kvm.

Let's assume you've built thexrv router.

Start two virtual routers:

docker run -d --name vr1 --privileged vr-xrv:5.3.3.51Udocker run -d --name vr2 --privileged vr-xrv:5.3.3.51U

I'm calling them vr1 and vr2. Note that I'm using XRv 5.3.3.51U - you shouldfill in your XRv version in the image tag as the "latest" tag is not added toany images.

It takes a few minutes for XRv to start but once up you should be able to SSHinto each virtual router. You can get the IP address using docker inspect:

root@host# docker inspect --format '{{.NetworkSettings.IPAddress}}' vr1172.17.0.98

Now SSH to that address and login with the default credentials ofvrnetlab/VR-netlab9:

root@host# ssh -l vrnetlab $(docker inspect --format '{{.NetworkSettings.IPAddress}}' vr1)The authenticity of host '172.17.0.98 (172.17.0.98)' can't be established.RSA key fingerprint is e0:61:28:ba:12:77:59:5e:96:cc:58:e2:36:55:00:fa.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '172.17.0.98' (RSA) to the list of known hosts.IMPORTANT:  READ CAREFULLYWelcome to the Demo Version of Cisco IOS XRv (the "Software").The Software is subject to and governed by the terms and conditionsof the End User License Agreement and the Supplemental End UserLicense Agreement accompanying the product, made available at thetime of your order, or posted on the Cisco website atwww.cisco.com/go/terms (collectively, the "Agreement").As set forth more fully in the Agreement, use of the Software isstrictly limited to internal use in a non-production environmentsolely for demonstration and evaluation purposes.  Downloading,installing, or using the Software constitutes acceptance of theAgreement, and you are binding yourself and the business entitythat you represent to the Agreement.  If you do not agree to allof the terms of the Agreement, then Cisco is unwilling to licensethe Software to you and (a) you may not download, install or use theSoftware, and (b) you may return the Software as more fully set forthin the Agreement.Please login with any configured user/password, or cisco/ciscovrnetlab@172.17.0.98's password:RP/0/0/CPU0:ios#show versionMon Jul 18 09:04:45.261 UTCCisco IOS XR Software, Version 5.3.3.51U[Default]...

You can also login via NETCONF:

root@host# ssh -l vrnetlab $(docker inspect --format '{{.NetworkSettings.IPAddress}}' vr1) -p 830 -s netconfvrnetlab@172.17.0.98's password:<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> <capabilities>  <capability>urn:ietf:params:netconf:base:1.1</capability>  <capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>  <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>  <capability>urn:ietf:params:netconf:capability:rollback-on-error:1.0</capability>  <capability>urn:ietf:params:netconf:capability:validate:1.1</capability>  <capability>urn:ietf:params:netconf:capability:confirmed-commit:1.1</capability>  <capability>http://cisco.com/ns/yang/Cisco-IOS-XR-aaa-lib-cfg?module=Cisco-IOS-XR-aaa-lib-cfg&amp;revision=2015-08-27</capability>  <capability>http://cisco.com/ns/yang/Cisco-IOS-XR-aaa-locald-admin-cfg?module=Cisco-IOS-XR-aaa-locald-admin-cfg&amp;revision=2015-08-27</capability>  <capability>http://cisco.com/ns/yang/Cisco-IOS-XR-aaa-locald-cfg?module=Cisco-IOS-XR-aaa-locald-cfg&amp;revision=2015-08-27</capability>  <capability>http://cisco.com/ns/yang/Cisco-IOS-XR-aaa-locald-oper?module=Cisco-IOS-XR-aaa-locald-oper&amp;revision=2015-08-27</capability>  <capability>http://cisco.com/ns/yang/Cisco-IOS-XR-bundlemgr-cfg?module=Cisco-IOS-XR-bundlemgr-cfg&amp;revision=2015-08-27</capability>...

The serial console of the devices are mapped to port 5000. Use telnet to connect:

root@host# telnet $(docker inspect --format '{{.NetworkSettings.IPAddress}}' vr1) 5000

Just like with any serial port, you can only have one connection at a time andwhile the router is booting the launch script will connect to the serial portto do the initialization of the router. As soon as it is done the port will bereleased and made available to the next connection.

To connect two virtual routers with each other we can use thevr-xconcontainer. Let's say we want to connect Gi0/0/0/0 of vr1 and vr2 with eachother, we would do:

docker run -d --name vr-xcon --link vr1 --link vr2 vr-xcon --p2p vr1/1--vr2/1

Configure a link network on vr1 and vr2 and you should be able to ping!

P/0/0/CPU0:ios(config)#inte GigabitEthernet 0/0/0/0RP/0/0/CPU0:ios(config-if)#no shutdownRP/0/0/CPU0:ios(config-if)#ipv4 address 192.168.1.2/24RP/0/0/CPU0:ios(config-if)#commitMon Jul 18 09:13:24.196 UTCRP/0/0/CPU0:Jul 18 09:13:24.216 : ifmgr[227]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to DownRP/0/0/CPU0:ios(config-if)#dRP/0/0/CPU0:Jul 18 09:13:24.256 : ifmgr[227]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/0/0/0, changed state to Upo ping 192.168.1.1Mon Jul 18 09:13:26.896 UTCType escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

(obviously I configured the other end too!)

All of the NICs of the virtual routers are exposed via TCP ports by KVM. TCPport 10001 maps to the first NIC of the virtual router, which in the case of anXR router is GigabitEthernet 0/0/0/0. By simply connecting two of these TCPsockets together we can bridge the traffic between those two NICs and this isexactly what vr-xcon is for. Use the--p2p argument to specify the links.The format is X/Y--Z/N where X is the name of the first router and Y is theport on that router. Z is the second router and N is the port on the secondrouter.

To set up more than one p2p link, simply add more mappings separated by spaceand don't forget to link the virtual routers:

docker run -d --name vr-xcon --link vr1 --link vr2 --link vr3 vr-xcon --p2p vr1/1--vr2/1 vr1/2--vr3/1

See topology-machine/README.md for details on topology machine which can helpyou with managing more complex topologies.

The containers expose port 22 for SSH, port 161 for SNMP, port 830 for NETCONFand port 5000 is mapped to the virtual serial device (use telnet). All the NICsof the virtual routers are exposed via TCP ports in the range 10001-10099.

Usedocker rm -f vr1 to stop and remote a virtual router.

There are some handy shell functions in vrnetlab.sh that provides shorthandsfor connecting to ssh and console.

1. Load the functions into your shell

. vrnetlab.sh

2. Login via ssh to router vr1, you can optionally specify a username. If nousername is provided, the default of vrnetlab will be used. If sshpass isinstalled, you will not be promted for password when you login with the defaultusername.

vrssh vr1 myuser

3. Connect console to router vr1

vrcons vr1

4. Create a bridge between two router interfaces, the below command bridgesinterface 1 of router vr1 with interface 1 of router 2.

vrbridge vr1 1 vr2 1

To load these aliases on login, copy it to ~/.vrnetlab_bashrc and add thefollowing to your .bashrc

test -f ~/.vrnetlab_bashrc && . ~/.vrnetlab_bashrc

There are a number of virtual routers available on the market:

• Cisco XRv
• Juniper VRR
• Juniper vMX
• Nokia VSR

All of the above are released as a qcow2 or vmdk file (which can easily beconverted into qcow2) making them easy to spin up on a Linux machine. Once spunup there are a few tasks one normally wants to perform:

• set an IP address on a management interface
• start SSH / NETCONF daemon (and generate crypto keys)
• create initial user so we can login

There might be more things to the list but this is the bare minimum which makesthe router remotely reachable and thus we can configure the rest from thenormal provisioning system.

vrnetlab aims to make this process as simple and convenient as possible so thatit may be used both by humans and automated systems to spin up virtual routers.In addition, there are scripts to help you generate topologies.

The virtual machines are packaged up in docker container. Since we need tostart KVM the docker containers have to be run with--privileged whicheffectively defeats the security features of docker. Our use of docker isessentially reduced to being a packaging format but a rather good one at that.Also note that since we still rely on KVM the same amount of resources, if notsightly more, will be consumed by vrnetlab. A container is no thinner than a VMif the container contains a VM!

The assignment of a management IP address is handed over to docker, so you canuse whatever docker IPAM plugin you want. Overall the network setup of thevirtual routers are kind of shoe-horned into the world of docker networking.I'm not sure this is a good idea but it seems to work for now and it was funputting it together ;)

It's possible to remotely control a docker engine and tell it to start/stopcontainers. It's not entirely uncommon to run the CI system in a VM and lettingit remotely control another docker engine can give us some flexibility in wherethe CI runner is executed vs where the virtual routers are running.

libvirt can also be remotely controlled so it could potentially be used to thesame effect. However, unlike libvirt, docker also has a registry concept whichgreatly simplifies the distribution of the virtual routers. It's already neatlypackaged up into a container image and now we can pull that image through asingle command. With libvirt we would need to distribute the VM image andlaunch scripts as individual files.

The launch script differ from router to router. For example, it's possible tofeed a Cisco XR router a bootup config via a virtual CD-ROM drive so we can usethat to enable SSH/NETCONF and create a user. Nokia VSR however does not, so weneed to tell KVM to emulate a serial device and then have the launch scriptaccess that virtual serial port via telnet to do the initial config.

The intention is to keep the arguments to each virtual router type as similaras possible so that a test orchestrator or similar need minimal knowledge aboutthe different router types.

You need to run these docker images on a machine that has a docker engine andthat supports KVM, i.e. you need a Linux kernel.

Docker is available for OS X and it works by spinning up a Linux VM on top ofthe xhyve hypervisor. While this means that we do have a docker engine and aLinux kernel, we are unable to use this for vrnetlab as xhyve does not offernested virtualization and thus we cannot run KVM in the VM running in xhyve.

VirtualBox does not offer nested virtualization either. Parallels and VMWaresupposedely do but I don't have access to those and can't test with.

See the README file of each virtual router type for CPU, RAM and diskrequirements.

If you are having problems with performance, like routers not starting or beingvery slow, there are a few knobs to tweak in order to improve the situation.

The basic problem is an unfortunate combination of CPU throttling and processscheduling causing cache thrasing which in turn leads to terrible performance.No detailed measurements have been done to confirm this exact behaviour but therecommended remedy has been confirmed working in multiple cases.

vrnetlab runs virtual machines using qemu/KVM, which appear just as normalprocesses in Linux and are thus subject to the Linux process scheduler. If aprocess wants to do work it will be scheduled to run on a core. Now, if not allcores are used, APM will throttle down some of the cores such that the workloadcan run on the remaining, say 3 out of 12 cores. The Linux scheduler will tryto schedule processes on the cores with the higher clock speed but if you havemore VMs than cores with high clock speed than it will start moving VMs around.L1/L2 caches are not shared by CPU cores, only L3. Moving a process from onecore to another inevitably means that the cache is evicted. When processes aremoved around continuously we get cache thrasing and this appears to lowerperformance for the VMs significantly. For some virtual routers it is to thepoint where we hit various watchdog timeouts and the VMs will restart.

The very first step is to make sure you aren't trying to run too many virtualrouters on the same physical host. Some virtual routers, like Nokia SROS, has arather low idle CPU usage of a few percent typically. Others, like Cisco XRV9kand Juniper vMX have a forwarding plane that is busy-looping over multiple CPUcores, thus consuming the entire CPU core. Trying to schedule multiple suchvirtual machines over the same CPU cores can lead to failure.

To improve performance, we can start by changing the CPU governor in Linux toperformance, for example usingcpupower frequency-set -g performance. Itlikely won't help much but try it first since it's considerably easier than thefollowing steps.

Disable Advanced Power Management (APM) or similar in BIOS. This willcompletely prevent the CPU cores from throttling down and they will run attheir designed maximum clock frequency. This probably means turbo boost(increasing clock frequency on a smaller subset of cores while decreasing thefrequency on remaining cores to remain at the same power and temperatureenvelope) will be disabled too. Performance across all cores will however bemuch more deterministic. This alone usually means that the Linux processscheduler will now keep processes on the same cores instead of moving themaround. Before only some of the cores would run at a higher frequency and sowould be more attractive to schedule work on. With all cores at the samefrequency, there is no reason for the process scheduler to move processesaround. This removes the main cause of cache thrashing. At least that's thesimplified view of it but it appears to be working rather well in reality.

If performance is still not adequate the next step would be to disablehyperthreading. Hyperthreading is a technology to expose two logical cores thatare executed by the same physical core. It's a strategy to avoid pipelinestalls, essentially where the CPU waits for memory. By having two logicalthreads, the CPU core can switch to the other thread whenever it needs to waitfor memory lookups. It increases total concurrent throughput, however, eachlogical thread will run slower than if it had run directly on a physical CPUcore.

You can avoid the effects of hyperthreading by only scheduling your qemuprocesses on half of the cores. You would need to inspect /proc/cpuinfo todetermine the exact logical core layout and make sure you only scheduleprocesses on one logical thread of each physical core. However, since you wouldthen only use half of the threads, it is easier to just disable hyperthreadingin BIOS altogether.

Applying the mentioned mitigations has so far resolved performance issues inall cases. Report if it doesn't for you.

vrnetlab containers use the Docker healthcheck mechanism to report whetherthey've started up properly or not.

A: I don't think Cisco, Juniper or Nokia would allow me to distribute their virtualrouter images and since one of the main points of vrnetlab is to have a selfcontained docker image I don't see any other way than for you to build yourown image based on vrnetlab but where you get to download the router imageyourself.

A: I don't like the concept as it means you have to ship around an extra file.If it's a self-contained image then all you have to do is push it to yourdocker registry and then ask a box in your swarm cluster to spin it up!

A: It is not persisted. The state of the virtual routers is lost once they arestopped/removed. It's not possible to restart vrnetlab or at least it's not atall tested and I don't see how it would work really. Since the primary use caseis lab / CI you should embrace the statelessness :)

A: No. vrnetlab still runs KVM (in docker) to start the virtual router whichmeans that we will consume just as much CPU and memory, if not slightly more,than running the router in KVM.

A: It's used primarily as a packaging format. All vrnetlab containers can berun with similar arguments. The differences between different platforms areeffectively hidden to present a clean uniform interface. That's certainly nottrue for trying to run XRv or vMX directly with qemu / virsh.

A: IOS XE is available through the CSR1000v image which should satisfy allyour oldskool needs.

A: I'm not entirely sure. For now you have to live with only communicatingbetween vrnetlab routers. There'shttps://github.com/TOGoS/TUN2UDP and Isuppose the same idea could be used to bridge the TCP-socket NICs used byvrnetlab to a tun device, but if all this should happen inside a dockercontainer or if we should rely on setting this up on the docker host (usingsomething similar to pipework) is not entirely clear to me. I'll probably workon it.

A: It was a long time since I used GNS3 and I have only briefly looked atUNetLab and VIRL but from what I know or can see, these are all more targetedtowards interactive labbing. You get a pretty UI and similar whereas vrnetlabis controlled in a completely programmatic fashion which makes them good atdifferent things. vrnetlab is superb for CI and programmatic testing where theothers probably target labs run by humans.

vrnetlab ships with a .gitlab-ci.yml config file so if you happen to be usingGitLab CI you can use this file to let your CI infrastructure build the dockerimages and push them to your registry.

GitLab features a built-in Docker registry which will be used per default - allyou need to do is enable the registry for your vrnetlab project. The necessaryinformation will be exposed as env vars in GitLab CI which is picked up by thebuild config.

The CI runner executing the jobs must have the tag 'vrnetlab'. Make sure thisrunner supports running VMs (has KVM) and allows the execution of siblingdocker containers.

If you want, you can use an external docker registry by explicitly configuringthe following environment variables:

• DOCKER_USER - the username to authenticate to the docker registry with
• DOCKER_PASSWORD - the password to authenticate to the docker registry with
• DOCKER_REGISTRY - the URL to the docker registry, like reg.example.com:5000

Next you need to add the actual virtual router images to the git repository.You can create a separate branch where you add the images as to avoid potentialgit merge issues. I recommend using LFS:

git checkout -b imagesgit lfs track "*.vmdk"git add xrv/iosxrv-k9-demo-6.0.0.vmdk .gitattributesgit commit -a -m "Added Cisco XRv 6.0.0 image"git push your-git-repo images

Now CI should build the images and push to wherever $DOCKER_REGISTRY points. Ifyou don't want to use LFS then just skip that command.

When new changes are commited to the upstream repo/master you can just rebaseyour branch on top of that:

git checkout mastergit pull origin mastergit checkout imagesgit rebase mastergit push --force your-git-repo images

Note that you have to force push since you've rewritten git history.

LFS is a way to store large files with git but keeping them out of git. It'sgreat for the virtual router images as they never change (version is in thefilename) and so we don't really need git's version tracking for them. LFS isconsiderably faster than plain git. For very large files it is possible to runinto LFS timeouts, try setting:

git config lfs.dialtimeout 60

vrnetlab ships with a .github/workflows/test.yml config file which is primarilyused in the public upstream repository:https://github.com/vrnetlab/vrnetlab.

The GitHub Actions runner must support running VMs (has KVM) and allows theexecution of sibling docker containers. The first requirement is not met by thefree GitHub Actions runners, so we use a self-hosted runner for the VM test(thanksGleSYS for sponsoring the runner ❤️).

This also solves the issue where to the public nature of the repository wecannot include the actual virtual router images in public repository via GitLFS. Instead, the images are provided to the build container via a local bindvolume mount. This means the router images must be available on the host machinewhere the build is running. The images are mounted into the build container at/vrnetlab-images.

The self-hosted GitHub Actions runner was set up on a machine using theinstructions here:https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runnersWe then create a script that reads in theVRNETLAB_IMAGES environment variable(set up as a repository variable) and creates a named docker volumevrnetlab-images that bind mounts the host path provided in the variable:

$ cat create-vrnetlab-images-volume.shdocker volume create --opt o=bind --opt type=none --opt device=${VRNETLAB_IMAGES} vrnetlab-imagesdocker volume inspect vrnetlab-images

Then in the runners.env file, we instruct the runner to run a script on jobstartup to create the volume:

ACTIONS_RUNNER_HOOK_JOB_STARTED=/path/to/runner/create-vrnetlab-images-volume.sh

The job container can then mount this volume and access the images in/vrnetlab-images regardless of the host path.:

test-vr:runs-on:["self-hosted"]container:image:vrnetlab/ci-buildervolumes:        -vrnetlab-images:/vrnetlab-images