This figure sums up the overall scenario
The test bed is based on two emulated devices (NodeA and NodeB) connected using a VDE Switch.
Each device has three Ethernet interfaces: eth2 is used only for management and MPTCP will be disabled on this interface. eth0 and eth1 will be used for MPTCP communications.
Network failure will be simulated changing the configuration of the VDE Switch (e.g. assigning a port to another VLAN).
Environment creation
Switch configuration
The VDE Switch is launched with this options:sudo vde_switch -s /tmp/switch1 -tap tap0
sudo ip a a 10.168.255.254/24 dev tap0 brd +
The tap interface is used to talk with eth2 ports, for the nodes management.
The configuration of the switch is:
vlan/create 1
vlan/create 2
vlan/create 3
vlan/create 4
port/setvlan 2 3
port/setvlan 3 3
port/setvlan 10 1
port/setvlan 20 1
port/setvlan 11 2
port/setvlan 21 2
As you can see, ports 10 and 20 are assigned to VLAN 1, ports 11 and 21 to VLAN 2.
QEMU OpenWRT configuration
Now you can launch the two nodes with these commands:sudo vdeq qemu-system-i386 test01.img \
-net nic,vlan=0,macaddr=52:54:00:01:01:01 \
-net vde,vlan=0,sock=/tmp/switch1,port=10 \
-net nic,vlan=1,macaddr=52:54:00:01:02:01 \
-net vde,vlan=1,sock=/tmp/switch1,port=11 \
-net nic,vlan=2,macaddr=52:54:00:01:00:02 \
-net vde,vlan=2,sock=/tmp/switch1,port=2 \
-name NodeA
sudo vdeq qemu-system-i386 test02.img \
-net nic,vlan=0,macaddr=52:54:00:02:01:01 \
-net vde,vlan=0,sock=/tmp/switch1,port=20 \
-net nic,vlan=1,macaddr=52:54:00:02:02:01 \
-net vde,vlan=1,sock=/tmp/switch1,port=21 \
-net nic,vlan=2,macaddr=52:54:00:02:00:03 \
-net vde,vlan=2,sock=/tmp/switch1,port=3 \
-name NodeB
test01.img and test02.img are two OpenWRT images built for x86 (see MultiPath TCP (MPTCP) for OpenWRT page).
OpenWRT Configuration
Network configuration files - /etc/config/network
Node A configuration
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config interface 'lan'
option ifname 'eth0'
option proto 'static'
option ipaddr '10.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
config interface 'lan1'
option ifname 'eth1'
option proto 'static'
option ipaddr '10.168.2.1'
option netmask '255.255.255.0'
config interface 'lan2'
option ifname 'eth2'
option proto 'static'
option ipaddr '10.168.255.1'
option netmask '255.255.255.0'
Node B configuration
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config interface 'lan'
option ifname 'eth0'
option proto 'static'
option ipaddr '10.168.1.2'
option netmask '255.255.255.0'
option ip6assign '60'
config interface 'lan1'
option ifname 'eth1'
option proto 'static'
option ipaddr '10.168.2.2'
option netmask '255.255.255.0'
config interface 'lan2'
option ifname 'eth2'
option proto 'static'
option ipaddr '10.168.255.2'
option netmask '255.255.255.0'
MPTCP Configuration
Node A
multipath eth2 off
# This creates two different routing tables,
# that we use based on the source-address
ip rule add from 10.168.1.1 table 1
ip rule add from 10.168.2.1 table 2
# Configure the two different routing tables
ip route add 10.168.1.0/24 dev eth0 scope link table 1
ip route add default via 10.168.1.2 dev eth0 table 1
ip route add 10.168.2.0/24 dev eth1 scope link table 2
ip route add default via 10.168.2.2 dev eth1 table 2
ip route add default via 10.168.1.2
Node B
multipath eth2 off
# This creates two different routing tables
# that we use based on the source-address.
ip rule add from 10.168.1.2 table 1
ip rule add from 10.168.2.2 table 2
# Configure the two different routing tables
ip route add 10.168.1.0/24 dev eth0 scope link table 1
ip route add default via 10.168.1.1 dev eth0 table 1
ip route add 10.168.2.0/24 dev eth1 scope link table 2
ip route add default via 10.168.2.1 dev eth1 table 2
ip route add default via 10.168.1.1
MPTCP at work
Now you can check the configuration with multipath command. The output is (for both nodes):
eth0 is in default mode
eth1 is in default mode
eth2 is deactivated
You can check also the status of MPTCP with:
# cat /proc/sys/net/mptcp/mptcp_enabled
1
The output "1" means it is enabled.
My suggestion is to enable fullmesh option with this command on the two nodes:
sysctl -w net.mptcp.mptcp_path_manager=fullmesh
Now you can generate a TCP stream with iperf:
On node B:
iperf -s -i 10
And on node A:
iperf -c 10.168.1.2 -i 1 -t 1000
And you can check the established flows with:
multipath -f
Index, Address-ID, Backup, IP-address
IPv4, next v4-index: 4
1, 2, 0, 10.168.1.1
2, 3, 0, 10.168.2.1
IPv6, next v6-index: 0
Now you can simulate a network disconnection changing a VLAN port with the command:
port/setvlan 10 4
With tcpdump command you can check on Node A and Node B interfaces how TCP flows are routed.
