- Notifications
You must be signed in to change notification settings - Fork6
jcpowermac/homelab-ovn
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
The current switch is a Cisco Catalyst 4006 w/Supervisor V. The chassisis 17 years old and the supervisor almost eight years old. The age plusthe amount of power required to run I decided it was time to go. I havea significantly newer switch that is unfortunately only layer-2. Ofcourse a sane person would say that is good enough.
Just a fair warning I would never claim to be an expert on networking,Open vSwitch or OVN. To get this to work was mostly working fromother individual’s blog posts, documentation, etc. If something isincorrect please either submit an issue or pull request. With that saidlet’s get started.
All the nodes directly participating geneve overlay will be Fedora 27.All the following commands will be based on Fedora. The configuration ofOVN should generally be distribution independent.
First let’s install some packages.
dnf install openvswitch \ openvswitch-ovn-central \ openvswitch-ovn-common \ openvswitch-ovn-host \ python2-openvswitch \ python3-openvswitch -y
If you want to use the container version of Skydive install docker as well. (recommended)
dnf install docker \ docker-compose -y
Then start and enable openvswitch and OVN services.
systemctl start openvswitchsystemctl enable openvswitchsystemctl start ovn-northdsystemctl enable ovn-northd
Hypervisor in this case is any node that is to be native on OVN. In mycase that is any Fedora physical node including my laptop - at leastwhen its docked.
sudo dnf install openvswitch \ openvswitch-ovn-common \ openvswitch-ovn-host
If you want Skydive install docker.
sudo dnf install docker \ docker-compose -y
Enable and start services, wait to startovn-controller until the nextsection.
systemctl start openvswitchsystemctlenable openvswitchsystemctlenable ovn-controller
On the node runningovn-northd execute the following commandsto allow remote connections.
ovn-nbctl set-connection ptcp:6641ovn-sbctl set-connection ptcp:6642ovs-appctl -t ovsdb-server ovsdb-server/add-remote ptcp:6640
Thedocker how toprovides the commands required to configure the controller. I alsolisted them below.
CENTRAL_IP=172.30.1.10 (1)ENCAP_TYPE=geneveLOCAL_IP=172.30.1.52 (2)ovs-vsctlset open. external_ids:ovn-remote="tcp:${CENTRAL_IP}:6642"ovs-vsctlset open. external_ids:ovn-nb="tcp:${CENTRAL_IP}:6641"ovs-vsctlset open. external_ids:ovn-encap-ip=${LOCAL_IP}ovs-vsctlset open. external_ids:ovn-encap-type="${ENCAP_TYPE}"
IP address of the node running
ovn-northdIP address of current node
After configuration start theovn-controller.
systemctl start ovn-controller
If you runovn-vsctl show you should see at least a new bridgebr-int.
$ sudo ovs-vsctl show90a55931-3706-4d55-9913-4a6e1f4b09e5 Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal
While I am in the process of migrating to OVN I need access to existingdevices on thelegacy network. It will also be required for devicesthat do not support OVN. Not sure if this is technically the bestway to do this but it works. Additional OVN configuration is required whichwill be listed below.
I made the assumption that IP forwarding would be required so I enabled it.
net.ipv4.ip_forward = 1
Configure thebr-dmz Open vSwitch bridge. This will be in file/etc/sysconfig/network-scripts/ifcfg-br-dmz
DEVICE=br-dmzONBOOT=yesBOOTPROTO=noneTYPE=OVSBridgeDEVICETYPE=ovsOVS_EXTRA="set Open_vSwitch . external-ids:ovn-bridge-mappings=dmz_localnet:br-dmz" (1)Map the
dmz_localnetswitch port to thebr-dmzbridge.
Configure the physical interface attached to thelegacy networkand add to thebr-dmz bridge. The IP address will be defined in OVN.This configuration will be in file/etc/sysconfig/network-scripts/ifcfg-enp2s5.
NAME="enp2s5"DEVICE="enp2s5"ONBOOT="yes"NETBOOT="yes"IPV6INIT="no"BOOTPROTO="none"DEFROUTE="no"IPV4_FAILURE_FATAL="no"IPV6_AUTOCONF="no"IPV6_DEFROUTE="no"IPV6_FAILURE_FATAL="no"TYPE=OVSPortDEVICETYPE=ovsOVS_BRIDGE=br-dmz
Finally bring up the bridge and interface.
ifup br-dmzifup enp2s5
The end goal is to have all network traffic on the overlay so each physicalmachine will have a OVS interface port configure. This configurationfor this example will be in file/etc/sysconfig/network-scripts/ifcfg-bromine.
DEVICE=bromineNAME=bromineTYPE=OVSIntPortDEVICETYPE=ovsOVS_BRIDGE=br-intOVS_EXTRA="set Interface bromine external_ids:iface-id=sw51_bromine" (1)IPADDR=172.31.51.52NETMASK=255.255.255.0GATEWAY=172.31.51.1DEFROUTE=yesMTU=1500DNS1=10.53.252.123DNS2=10.53.252.246The
iface-idwill be name provided in thelsp-addcommand.
Bring up the interface but it won’t pass traffic until the logical switch port iscreated.
ifup bromine
The topology and most of the OVN configuration below were modifications fromthisguide so I recommend reading it for additional information.
Normal switches:sw50,sw51, andsw52.
Transit switch is between routerr0 and the gateway routergr0. Assumingthis is to allowr0 to be distributed while maintaining a connection to thelocalizedgr0.
desk=sw51transit=tsw0outsw=osw0prod=sw52ose=sw50ovn-nbctl --may-exist ls-add${desk}ovn-nbctl --may-exist ls-add${transit}ovn-nbctl --may-exist ls-add${outsw}ovn-nbctl --may-exist ls-add${prod}ovn-nbctl --may-exist ls-add${ose}
Only need two routersr0 andgr0.
router=r0gr=gr0ovn-nbctl --may-exist lr-add${router}chassis_uuid=$(ovn-sbctl --bare --columns name find Chassis hostname=ovn-gateway0.virtomation.com)ovn-nbctl create Logical_Router name=${gr} options:chassis=${chassis_uuid} (1)
The gateway router must be configured on a specific node or chassis.
Create logical router ports with mac and ip addresses for each network.
ovn-nbctl --may-exist lrp-add${router}${router}_${desk} 02:ac:10:1f:33:01 172.31.51.1/24ovn-nbctl --may-exist lrp-add${router}${router}_${prod} 02:ac:10:1f:34:01 172.31.52.1/24ovn-nbctl --may-exist lrp-add${router}${router}_${ose} 02:ac:10:1f:32:01 172.31.50.1/24ovn-nbctl --may-exist lrp-add${router}${router}_${transit} 02:ac:10:1f:ff:02 172.31.255.2/30ovn-nbctl --may-exist lrp-add${gr}${gr}_${transit} 02:ac:10:1f:ff:01 172.31.255.1/30ovn-nbctl --may-exist lrp-add${gr}${gr}_${outsw} 02:ac:10:1f:0c:f6 10.53.12.246/24
Create static routes to enable traffic between networks.
ovn-nbctl lr-route-add${gr} 0.0.0.0/0 10.53.12.1 (1)ovn-nbctl lr-route-add${gr} 10.53.0.0/16 10.53.12.254 (2)ovn-nbctl lr-route-add${gr} 172.31.0.0/16 172.31.255.2 (3)ovn-nbctl lr-route-add${router} 0.0.0.0/0 172.31.255.1 (4)
Static route for internet traffic.
Static route for
legacynetworks.Static route for overlay networks.
Static route for all external networks.
Create logical switch ports for each router, physical device and the gateway.
# Routerovn-nbctl --may-exist lsp-add${desk}${desk}_${router}ovn-nbctl --may-exist lsp-add${prod}${prod}_${router}ovn-nbctl --may-exist lsp-add${ose}${ose}_${router}ovn-nbctl --may-exist lsp-add${transit}${transit}_${router}ovn-nbctl --may-exist lsp-add${outsw}${outsw}_${gr}ovn-nbctl --may-exist lsp-add${transit}${transit}_${gr}# Physicalovn-nbctl --may-exist lsp-add${desk}${desk}_bromineovn-nbctl --may-exist lsp-add${prod}${prod}_uranium# Gatewayovn-nbctl --may-exist lsp-add${outsw}${outsw}_localnet
For each port configure the type, allowed address, and appropriate options.
# Routerovn-nbctl lsp-set-type${desk}_${router} routerovn-nbctl lsp-set-addresses${desk}_${router} 02:ac:10:1f:33:01ovn-nbctl lsp-set-options${desk}_${router} router-port=${router}_${desk}ovn-nbctl lsp-set-type${prod}_${router} routerovn-nbctl lsp-set-addresses${prod}_${router} 02:ac:10:1f:34:01ovn-nbctl lsp-set-options${prod}_${router} router-port=${router}_${prod}ovn-nbctl lsp-set-type${ose}_${router} routerovn-nbctl lsp-set-addresses${ose}_${router} 02:ac:10:1f:32:01ovn-nbctl lsp-set-options${ose}_${router} router-port=${router}_${ose}ovn-nbctl lsp-set-type${outsw}_${gr} routerovn-nbctl lsp-set-addresses${outsw}_${gr} 02:ac:10:1f:0c:f6ovn-nbctl lsp-set-options${outsw}_${gr} router-port=${gr}_${outsw}ovn-nbctl lsp-set-type${transit}_${gr} routerovn-nbctl lsp-set-addresses${transit}_${gr} 02:ac:10:1f:ff:01ovn-nbctl lsp-set-options${transit}_${gr} router-port=${gr}_${transit}ovn-nbctl lsp-set-type${transit}_${router} routerovn-nbctl lsp-set-addresses${transit}_${router} 02:ac:10:1f:ff:02ovn-nbctl lsp-set-options${transit}_${router} router-port=${router}_${transit}# Gatewayovn-nbctl lsp-set-type${outsw}_localnet localnetovn-nbctl lsp-set-addresses${outsw}_localnet unknownovn-nbctl lsp-set-options${outsw}_localnet network_name=dmz_localnet# Physicalovn-nbctl lsp-set-addresses${desk}_bromine unknownovn-nbctl lsp-set-addresses${prod}_uranium unknown
After getting a few physical machines up and running on OVN the next stepwas my real hypervisor nodes. This was more of a challenge than Ioriginally thought it was going to be. I started by reviewing thisblog post: Using OVN with KVM and Libvirtwhich certainly provided valuable insight. Though being a lazy programmer there had tobe a better way -libvirt hooks.
Provided in this repository is aqemuhook for OVN. It adds and removes the switch port when the machine is started or stopped.The configuration for theovn-northd node and the switch name is stored inthe virtual machines metadata.
First some prerequisites.
dnf install git -ypip install ovsdbappgit clone https://github.com/jcpowermac/homelab-ovn
If the directory doesn’t exist (which it didn’t on my hypervisor) create it.
mkdir -p /etc/libvirt/hooks/cp homelab-ovn/libvirt-hook/qemu /etc/libvirt/hooks/chmod 744 /etc/libvirt/hooks/qemu
After the hook is available libvirtd needs to be restarted.
systemctl restart libvirtd
Thevirt-install command is an example the--network option that must be used to connect avirtual machine to a specific logical switch. Thevirsh metadata command below adds metadatato a defined virtual machine. This commandmust be written exactly as below for theqemu hook tofunction properly.
virt-install --import --name$vm --memory 8192 --vcpus 2 \ --graphics none --console pty,target_type=serial \ --os-type linux --os-variant rhel7.0 --noautoconsole \ --disk path=/instances/$vm.qcow2,format=qcow2,bus=virtio \ --network bridge=br-int,virtualport_type=openvswitch (1)virsh metadata$vm --uri ovs \ --key ovn \ --set'<parameters northd="172.30.1.10" switch="sw50"/>' (2)
The virtual machine must attach to the
br-intbridge and have avirtualport_typeofopenvswitch.The parameters are farily simple,
northdis the ip address ofovn-northdnode andswitchis where the virtual machine should be connected.
Sometimes its better to have a picture or two.
As I suggested above runningSkydive is a good idea.At least then you have a visual representation of the interfaces and bridges that are used.
Included in this repo is aJupyter notebook. It currently displays logical routers, switches, ports, and networks on the edge.
Missing links for ipywidgets, matplotlib