Monday, January 18, 2016

MultiPath TCP: how to test MPTCP with OpenWRT and QEMU

The goal of this tutorial is to create a simple scenario with OpenWRT, QEMU and VDE (Virtual Distributed Ethernet - http://vde.sourceforge.net/) to analyze the behavior of MPTCP.

This figure sums up the overall scenario



The test bed is based on two emulated devices (NodeA and NodeB) connected using a VDE Switch.
Each device has three Ethernet interfaces: eth2 is used only for management and MPTCP will be disabled on this interface. eth0 and eth1 will be used for MPTCP communications.

Network failure will be simulated changing the configuration of the VDE Switch (e.g. assigning a port to another VLAN).

Environment creation

Switch configuration

The VDE Switch is launched with this options:

sudo vde_switch -s /tmp/switch1 -tap tap0 

sudo ip a a 10.168.255.254/24 dev tap0 brd +

The tap interface is used to talk with eth2 ports, for the nodes management.
The configuration of the switch is:

vlan/create 1
vlan/create 2
vlan/create 3
vlan/create 4
port/setvlan 2 3
port/setvlan 3 3
port/setvlan 10 1
port/setvlan 20 1
port/setvlan 11 2
port/setvlan 21 2 


As you can see, ports 10 and 20 are assigned to VLAN 1, ports 11 and 21 to VLAN 2.

QEMU OpenWRT configuration

Now you can launch the two nodes with these commands:

sudo vdeq qemu-system-i386 test01.img \
  -net nic,vlan=0,macaddr=52:54:00:01:01:01 \

  -net vde,vlan=0,sock=/tmp/switch1,port=10 \
  -net nic,vlan=1,macaddr=52:54:00:01:02:01 \

  -net vde,vlan=1,sock=/tmp/switch1,port=11 \
  -net nic,vlan=2,macaddr=52:54:00:01:00:02 \

  -net vde,vlan=2,sock=/tmp/switch1,port=2 \
  -name NodeA



sudo vdeq qemu-system-i386 test02.img \
  -net nic,vlan=0,macaddr=52:54:00:02:01:01 \

  -net vde,vlan=0,sock=/tmp/switch1,port=20 \
  -net nic,vlan=1,macaddr=52:54:00:02:02:01 \

  -net vde,vlan=1,sock=/tmp/switch1,port=21 \
  -net nic,vlan=2,macaddr=52:54:00:02:00:03 \
  -net vde,vlan=2,sock=/tmp/switch1,port=3 \
  -name NodeB 


test01.img and test02.img are two OpenWRT images built for x86 (see MultiPath TCP (MPTCP) for OpenWRT page).

OpenWRT Configuration

Network configuration files -  /etc/config/network 

Node A configuration

config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config interface 'lan'
option ifname 'eth0'
option proto 'static'
option ipaddr '10.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'

config interface 'lan1'
option ifname 'eth1'
option proto 'static'
option ipaddr '10.168.2.1'
option netmask '255.255.255.0'

config interface 'lan2'
option ifname 'eth2'
option proto 'static'
option ipaddr '10.168.255.1'
option netmask '255.255.255.0'

Node B configuration

config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config interface 'lan'
option ifname 'eth0'
option proto 'static'
option ipaddr '10.168.1.2'
option netmask '255.255.255.0'
option ip6assign '60'

config interface 'lan1'
option ifname 'eth1'
option proto 'static'
option ipaddr '10.168.2.2'
option netmask '255.255.255.0'

config interface 'lan2'
option ifname 'eth2'
option proto 'static'
option ipaddr '10.168.255.2'
option netmask '255.255.255.0'

MPTCP Configuration

Node A

multipath eth2 off

# This creates two different routing tables, 
# that we use based on the source-address
ip rule add from 10.168.1.1 table 1
ip rule add from 10.168.2.1 table 2

# Configure the two different routing tables
ip route add 10.168.1.0/24 dev eth0 scope link table 1
ip route add default via 10.168.1.2 dev eth0 table 1

ip route add 10.168.2.0/24 dev eth1 scope link table 2
ip route add default via 10.168.2.2 dev eth1 table 2

ip route add default via 10.168.1.2

Node B

multipath eth2 off

# This creates two different routing tables
# that we use based on the source-address.
ip rule add from 10.168.1.2 table 1
ip rule add from 10.168.2.2 table 2

# Configure the two different routing tables
ip route add 10.168.1.0/24 dev eth0 scope link table 1
ip route add default via 10.168.1.1 dev eth0 table 1

ip route add 10.168.2.0/24 dev eth1 scope link table 2
ip route add default via 10.168.2.1 dev eth1 table 2

ip route add default via 10.168.1.1

MPTCP at work

Now you can check the configuration with multipath command. The output is (for both nodes):

eth0 is in default mode
eth1 is in default mode
eth2 is deactivated

You can check also the status of MPTCP with:
# cat /proc/sys/net/mptcp/mptcp_enabled 
1

The output "1" means it is enabled.

My suggestion is to enable fullmesh option with this command on the two nodes:

sysctl -w net.mptcp.mptcp_path_manager=fullmesh

Now you can generate a TCP stream with iperf:
On node B:
iperf -s -i 10

And on node A:
iperf -c 10.168.1.2 -i 1 -t 1000

And you can check the established flows with:
multipath -f
Index, Address-ID, Backup, IP-address
IPv4, next v4-index: 4
1, 2, 0, 10.168.1.1
2, 3, 0, 10.168.2.1
IPv6, next v6-index: 0

Now you can simulate a network disconnection changing a VLAN port with the command:
port/setvlan 10 4

With tcpdump command you can check on Node A and Node B interfaces how TCP flows are routed.