Nutanix AHV – Networking Commands How-To Super Post

This post is meant to be a "quick reference" guide for performing configuration with AHV networking, including Open vSwitch (OVS) configuration, setting VLAN tags and other miscellaneous networking related commands.

For further details and a helpful video, check out this post by Jason Burns which goes a bit deeper into which configurations you should use and the "why" of the commands: https://next.nutanix.com/t5/Nutanix-Connect-Blog/Network-Load-Balancing-with-Acropolis-Hypervisor/ba-p/6463

CHANGE AHV TO ONLY USE THE 10 GB INTERFACES

# Enable 10 GB links only (ran from CVM):

manage_ovs –interfaces 10g update_uplinks

or use the “allssh” command to run on all hosts in cluster:

allssh manage_ovs –interfaces 10g update_uplinks

SHOW VSWITCH CONFIGURATION AND UPLINKS WITH SPEEDS

# Show the OVS uplinks

manage_ovs show_uplinks

# Show the OVS interfaces # Shows link speed and connected state

manage_ovs show_interfaces

SETTING LOAD BALANCING MODE – ACTIVE-BACKUP

# Active-backup mode is enabled by default, but can be configured with the following AHV command (Ran from CVM):

ssh root@192.168.5.1 "ovs-vsctl set port bond0 bond_mode=active-backup"

# Or run from CVM and change for all hosts in cluster:

hostssh "ovs-vsctl set port bond0 bond_mode=active-backup"

Active-backup bond mode is the simplest, easily allowing connections to multiple upstream switches without any additional switch configuration. The downside is that traffic from all VMs use only the single active link within the bond. All backup links remain unused. In a system with dual 10 gigabit Ethernet adapters, the maximum throughput of all VMs running on a Nutanix node is limited to 10 Gbps.

SETTING LOAD BALANCING MODE – BALANCE-SLB (*PREFERRED*)

# Balance-slb algorithm is configured for each bond on all AHV nodes in the Nutanix cluster with the following commands (Ran from CVM):

ssh root@192.168.5.1 "ovs-vsctl set port bond0 bond_mode=balance-slb"

ssh root@192.168.5.1 "ovs-vsctl set port bond0 other_config:bond-rebalance-interval=60000"

# Or run from CVM and change for all hosts in cluster:

hostssh "ovs-vsctl set port bond0 bond_mode=balance-slb"

hostssh "ovs-vsctl set port bond0 other_config:bond-rebalance-interval=60000"

# Verify/View the bond mode with the following commands:

ssh root@192.168.5.1 "ovs-appctl bond/show bond0"

To take advantage of the bandwidth provided by multiple upstream switch links, we recommend configuring the bond mode as balance-slb. The balance-slb bond mode in OVS takes advantage of all links in a bond and uses measured traffic load to rebalance VM traffic from highly used to less used interfaces. When the configurable bond-rebalance-interval expires, OVS uses the measured load for each interface and the load for each source MAC hash to spread traffic evenly among links in the bond.

SETTING LOAD BALANCING MODE – LACP WITH BALANCE-TCP

# LACP with balance-tcp is configured with the following commands. Upstream switch configuration of LACP is required. Run commands in the order listed below.

ssh root@192.168.5.1 "ovs-vsctl set port bond0 other_config:lacp-fallback-ab=true"

ssh root@192.168.5.1 "ovs-vsctl set port bond0 bond_mode=balance-tcp"

ssh root@192.168.5.1 "ovs-vsctl set port bond0 lacp=active"

If upstream LACP negotiation fails, the default configuration is to disable the bond, which would block all traffic. The following command allows fallback to active-backup bond mode in the event of LACP negotiation failure:

ssh root@192.168.5.1 "ovs-vsctl set port bond0 other_config:lacp-fallback-ab=true"

# Verify/View the bond mode with the following commands:

ssh root@192.168.5.1 "ovs-appctl bond/show bond0”

LACP and Link Aggregation

Because LACP and balance-tcp require upstream switch configuration, and because network connectivity may be disabled if cables from AHV nodes are moved to incorrectly configured switches, Nutanix does not recommend using link aggregation or LACP.

However, to take full advantage of the bandwidth provided by multiple links to upstream switches from a single VM, link aggregation in OVS using Link Aggregation Control Protocol (LACP) and balance-tcp is required. Note that appropriate configuration of the upstream switches is also required. With LACP, multiple links to separate physical switches appear as a single Layer-2 link. Traffic can be split between multiple links in an active-active fashion based on a traffic-hashing algorithm.

VLAN TAGGING

# CVM VLAN – Change CVM VLAN (ran from CVM)

change_cvm_vlan <tag>

# AHV HOST VLAN – Change VLAN tagging On AHV host (ran from AHV host):

ovs-vsctl set port br0 tag=<tag>

# IPMI VLAN – Change IPMI VLAN tagging (ran from AHV host):

ipmitool lan set 1 vlan id <tag>

 

Examples:

# Run from CVM, sets CVM VLAN ID to 94, on all hosts in cluster

for g in `svmips`; do ssh $g "nohup sleep 90 && change_cvm_vlan 94” & ;done ;

# Run from CVM, sets Hypervisor Host VLAN ID to 94, on all hosts in cluster

hostssh ovs-vsctl set port br0 tag=94

# Run from CVM, sets IPMI VLAN ID to 92 on all hosts in cluster

hostssh ipmitool lan set 1 vlan id 92

MISC OTHER HELPFUL NETWORKING RELATED COMMANDS

# Show the switch configuration

ovs-vsctl show

# List the bridges

ovs-vsctl list br

# Show the interface configuration

ovs-vsctl list interface br0

# Show ports on a bridge

ovs-vsctl list-ports br0

# Show interfaces on a bridge

ovs-vsctl list-ifaces br0

# Create an OVS bridge

ovs-vsctl add-br <bridge>

# Add ports to a bridge

ovs-vsctl add-port <bridge> <port>

# Remove ports from a bridge (example removes eth2 from br0):

ovs-vsctl del-port br0 eth2

# Add bond port to bridge

ovs-vsctl add-bond <bridge> <port> <iface>

# Show bond details

ovs-appctl bond/show <bond>

# Set ports to fall back to active-backup mode

ovs-vsctl set port bond0 other_config:lacp-fallback-ab=true

# Remove a configuration on a port (useful if you entered a typo)

ovs-vsctl remove port bond0 other_config <config>

# exampleovs-vsctl remove port bond0 other_config lacp-fallback-ab