# lvs-dpdk

**Repository Path**: mirrors/lvs-dpdk

## Basic Information

- **Project Name**: lvs-dpdk
- **Description**: LVS on dpdk
- **Primary Language**: C/C++
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: https://www.oschina.net/p/lvs-dpdk
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 0
- **Created**: 2017-04-03
- **Last Updated**: 2026-01-03

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# lvs-dpdk

This project has ported LVS FULLNAT/DR/NAT and SNAT-GATEWAY to OpenFastPath(base on odp-dpdk).

NAT is only available on single core while FULLNAT,DR and SNAT-GATEWAY support multi-cores, because in lvs-dpdk each core has one local session table and depend on flow director.

LVS-FULLNAT origin source code is at https://github.com/alibaba/LVS

LVS-SNAT gateway origin source code is at https://github.com/jlijian3/lvs-snat

I had forked OpenFastPath project and added support for flow director and ofp_vs example, see https://github.com/lvsgate/ofp/tree/master/example/ofp_vs

I had forked odp-dpdk project and added support for flow director, see https://github.com/lvsgate/odp-dpdk.git

Please note that this project is experimental, it only had limited testing and never runned in production environment.

# Prerequisites

- Intel x86 CPU
- NIC which support flow director, if you want to run on multi-cores
- lvs-dpdk has been compiled and tested on Centos 7.2 with 3.10 kernel
- 5G memory at least

# Performance
- CPU E5-2682 v4 @ 2.50GHz
- NIC intel 82599 10G
- fullnat 800w pps if use 5 cores (in 400w pps and out 400w pps)

# Build all in one step
	git clone https://github.com/lvsgate/lvs-dpdk.git
	cd lvs-dpdk
	./build_all.sh
	cd ofp_vs
If build all successfully [Goto step 5](#5-prepare-dpdk-for-running-lvs-dpdk), otherwise build one by one.

# Build steps one by one

## 1. Fetch and compile DPDK

	git clone git://dpdk.org/dpdk ./<dpdk-dir>
	cd <dpdk-dir>
	git checkout -b 17.02 tags/v17.02
	make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
	cd <dpdk-dir>/x86_64-native-linuxapp-gcc
	sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_PCAP=).*,\1y,' .config
	cd ..
	make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS="-fPIC"
	
	#Add env value
	export RTE_SDK=<dpdk-dir>
    	export RTE_TARGET=x86_64-native-linuxapp-gcc
	    
## 2. Fetch and compile odp-dpdk

	git clone  https://github.com/lvsgate/odp-dpdk.git <odp-dir>
	cd <odp-dir>
	./bootstrap
	#SHARED library
	./configure --enable-shared --with-platform=linux-dpdk --enable-helper-linux --with-sdk-install-path=<dpdk-dir>/x86_64-native-linuxapp-gcc --prefix=<INSTALL ODP-DPDK TO THIS DIR>
	#If build only static library, dpdk driver register function defined with __attribute__(construction) won't be called.
	
	make
	make install
	
## 3. Fetch and compile ofp

	yum install libnl3 libnl3-cli libnl3-devel #or download rpm from https://www.infradead.org/~tgr/libnl/
	git clone https://github.com/lvsgate/ofp.git <ofp-dir>
	cd <ofp-dir>
	./bootstrap
	./configure --disable-libconfig --with-odp-lib=odp-dpdk --with-odp=<ODP-DPDK INSTALLATION DIR> --enable-shared=no --enable-sp=yes --disable-mtrie CPPFLAGS=-I<ODP-DPDK INSTALLATION DIR>/include/odp/arch/x86_64-linux/
	make

## 4. Fetch and compiled lvs-dpdk tools

	git clone https://github.com/lvsgate/lvs-dpdk.git
	cd lvs-dpdk/tools/keepalived
	sh configure --prefix=/usr --sysconfdir=/etc/ CPPFLAGS=-I/usr/include/libnl3/ LDFLAGS=-L/usr/lib64/
	make
	make install
	cd lvs-dpdk/tools/ipvsadm
	make
	make install
	
## 5. Prepare DPDK for running lvs-dpdk
	echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
	echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
	mkdir /mnt/huge
	mount -t hugetlbfs nodev /mnt/huge
	modprobe uio
	insmod <dpdk-dir>/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
	cd <dpdk-dir>
	./usertools/dpdk-devbind.py --status
	./usertools/dpdk-devbind.py --bind=igb_uio <pci-id-1>
	./usertools/dpdk-devbind.py --bind=igb_uio <pci-id-2>

		
## 6. Run ofp_vs
    modprobe ip_vs  #add this line to /etc/rc.local, because ipvsadm and keepalived depend on it.
    cd <ofp-dir>/examples/ofp_vs
    ./ofp_vs -i 0,1 -c 0x00f0 -p 1 -f ofp.conf # -i <port1>,<port2>  
                                     # -c <worker core mask> 
                                     # -o <required for snat-gw, outer port to wan, snat-gw fdir rule will be add to this port>
                                     # -p <required for fullnat, inner port to lan, fullnat fdir rule  will be add to this port>
                                     # -f <config file include default command which you can change in ofp cli>
    #If worker core count > 0, -o is required for snat-gw, -p is required for fullnat.


## 7. Connect to ofp cli or edit ofp.conf to configure network
    telnet localhost 2345
    type in "?" or "help"
    >>> ?
    # fp0 equal to port number 0 in dpdk
    >>> ifconfig fp0 <ip_addr>/<net_mask> 
    >>> ifconfig fp1 <ip_addr>/<net_mask> 
    #default gw don't work if enable mtries routing, may be ofp's bug.
    >>> route add 0.0.0.0/0 gw <next hop> dev fp0
    >>> route add <ip_addr>/<net_mask> gw <next hop> dev fp1
    #Add vs ip to lo if you want to ping it
    >>> ifconfig lo0 <vip1/32>
    >>> ifconfig lo1 <vip2/32>
    
## 8. Configure network on your switch/router
    #The ip on the port is used as nexthop of vs/snat ip or local address.
    #H3C switch example.
    #add route for virutal server address or SNAT source address
    >>> ip route-static <vs.ip.x.0> 255.255.255.0 <nexthop ip on the interface fp0> 
    #add route for local address of FULLNAT
    >>> ip route-static <local.addr.x.0> 255.255.255.0 <nexthop ip on the interface fp1> 
    
    
## 9. Start and configure SNAT-GATEWAY
    ./ofp_vs -i 0,1 -c 0x00f0 -o 0 -f ofp.conf  #-o specify the port to wan
    telnet localhost 2345
    >>> snat enable
    >>> snat add from 10.1.0.0/16 to 0.0.0.0/0 out_dev fp0 source 192.168.50.253 - 192.168.50.253 algo sd
    >>> snat add from 10.1.0.10/32 to 0.0.0.0/0 out_dev fp0 source 192.168.50.100 - 192.168.50.103 algo sdfn
    >>> snat del from 10.1.0.10/32 to 0.0.0.0/0 out_dev fp0
    >>> snat show
    

## 10. Start and configure FULLNAT virtual server
	./ofp_vs -i 0,1 -c 0x00f0 -p 1 -f ofp.conf  #-p specify the port to lan
	
	#ipvsadm and keepalived will comunicate with ofp_vs process but not the kernel module.
	#Create FULLNAT virtual server, local address count must be greater than worker count
	ipvsadm  -A  -t <vip:vport> -s rr
	ipvsadm  -a  -t <vip:vport> -r <rsip1:rsport> -b
	ipvsadm  -a  -t <vip:vport> -r <rsip2:rsport> -b
	ipvsadm  -P  -t <vip:vport> -z <local_addr1>
	ipvsadm  -P  -t <vip:vport> -z <local_addr2>
	ipvsadm  -P  -t <vip:vport> -z <local_addr3>
	ipvsadm  -P  -t <vip:vport> -z <local_addr4>
	...
	...
	ipvsadm  -P  -t <vip:vport> -z <local_addr254>
	ipvsadm -ln
	ipvadm -G
    
## 11. Try to visit vs now
	curl <vip:vport>

## 12. More details about ofp and odp-dpdk
    http://www.openfastpath.org/
    http://opendataplane.org/
    https://github.com/OpenFastPath/ofp
    https://github.com/lvsgate/ofp
    https://github.com/lvsgate/odp-dpdk