代码拉取完成，页面将自动刷新
QStack is an efficient user-space TCP/IP network stack based on DPDK. QStack could serve workloads with high concurrency, up to tens of millions of concurrent TCP connections. QStack provides flexible thread allocation and arrangement to fit application requirements. QStack supports user-defined packet-level priority scheduling to guarantee response tail latency of high-priority requests. QStack can provide low tail latency for latency-sensitive requests with high priority under high concurrent loads. In our IoT high-concurrency benchmark, QStack maintained a tail latency of 3.5 milliseconds for high-priority requests while serving 1 million concurrent TCP connections.
QStack source code is distributed under the Mulan PSL v2 License.
QStack uses part of mTCP source code for TCP/IP processing.
QStack uses part of libco source code for coroutine management.
Architecture
Supported features
We require the following libraries to run QStack.
libnuma (libnuma.so.1.0.0)libpthread (libpthread-2.17)openssl-devel(1.0.2k-24)QStack runs on Linux-based operating systems with generic x86_64 CPUs, but to help evaluation, we provide our tested environments as follows.
Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz,
128 GB of RAM,
82599ES 10-Gigabit SFI/SFP+,
CentOS Linux release 7.6.1810 (Linux 3.10.0)
iotepserver is a benchmark of IoT server serving bursty workloads with high concurrency. The requests received by iotepserver are classified to request messages and heartbeat messages. Both of them will query the Redis database, but only send back responses to request messages.
The iotepserver benchmark requires 1 server, 1 client and 1 monitor.
The server machine needs to have at least 64 logic cores (with hyper-thread enabled) with at least 64GB RAM (reserve 20GB hugepage for each NUMA node), 10Gbps NIC.
Note: the NIC should be supported by DPDK drivers, see http://dpdk.org/doc/nics for the support list.
Note: the user could decrease the max concurrency support by modify MAX_FLOW_NUM in <qstack_home>/qstack/src/include/typical_macro_iotepserver.h to run QStack with less memory.
Note: our current thread layout is optimized for the architecture in our tested environment. Adjust PHYSOCK_NUM and CORE_PER_SOCK in <qstack_home>/qstack/src/include/typical_macro_iotepserver.h based on the actual architecture in use for the optimal thread layout, and then run make.sh again.
Install and set up Intel's DPDK driver. prepare DPDK source code
# cd <qstack_home>
# tar xvf deps/dpdk.tar.gz -C ./
# cd dpdk
# git apply ../deps/dpdk-patch
use the following steps in the /dpdk/usertools/dpdk-setup.sh script for setup:
- Press [14] to compile the package
- Press [17] to install the driver
- Press [21] to setup hugepages, input 10240 for every NUMA node with 2M hugepages, and input 20 for every NUMA node with 1G hugepages.
- Press [23] to register the Ethernet ports
- Press [34] to quit the tool
Check that DPDK package creates a new directory of compiled
libraries. For x86_64 machines, the new subdirectory should be
dpdk/x86_64-native-linuxapp-gcc
Create soft links for include/ and lib/ directories inside
empty dpdk-lnk/ directory:
# cd <qstack_home>
# mkdir dpdk-lnk
# cd dpdk-lnk
# ln -s ../dpdk/x86_64-native-linuxapp-gcc/lib lib
# ln -s ../dpdk/x86_64-native-linuxapp-gcc/include include
Prepare redis as iotepserver backend server install redis
# tar xvf <qstack_home>/deps/redis-4.0.2.tar.gz -C /home/
# cd /home/redis-4.0.2; make; make install
# cd deps/hiredis; make; make install
Note: we strongly recommend to install redis at /home/redis-4.0.2, as it's the default path for redis in the following redis-setup scripts
setup redis
# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
# mkdir /var/run/redis
# cd <qstack_home>/tools/redis-scripts
# ./dir-redis.sh 16 sock
# cd <qstack_home>/apps/redis; make
Note: We Must apply non-root user privileges to compile parts as below, otherwise the comiling will be stopped.
# cd <qstack_home>/qstack/src;./make.sh
start redis
# cd <qstack_home>/tools/redis-scripts
# ./setup-redis.sh 16 up sock
Note: the param 16 refer to start 16 redis-server;
start iotepserver
# cd <qstack_home>/apps/
# vim iotepserver.conf
# No. of cores setting (enabling this option will override
# the `cpu' config for those applications that accept
# To start the echo_server, please run command
# ./build/echo_server -f echo_server.conf
# To stop the echo_server, just run Ctrl+C
# num_cores as command line arguments)
#
num_cores = 32
# The ip address of this host
host_ip = 192.168.78.100/16
# Maximum concurrency in total (default = 10000)
max_concurrency = 14000000
#The number of stack threads
stack_thread = 4
#The number of application threads
app_thread = 4
#The core id dedicated for stack info printing
stat_print = 31
#Enable the multi-priority identification and processing
pri_enable = 1
# cd build
# ./iotepserver -f iotepserver.conf -w 4 -i 32 2>err
-f : configuration file
-i : refers to the TCP header length of requests, used for priority classification of every packet(32 for default).
-w : refers to the worker threads distributed by per app server thread.
In iotepserver benchmark, we use MCC as the workload generator running on the client. MCC is an open-source network workload generator which could generate bursty workload with extremely high concurrency.
max_concurrency and max_num_buffers in mtcp.conf should be no less than 1500000# cd <mcc>/build/release/apps/mcc
# ./mcc -c 10368000 -b 172800 -e 60 -s 200 -w 20 -l 200 -r 0.05 --ips 230 --log-duration 60 --network-stack mtcp --dest 192.168.3.10 --smp 9 2>err
-c : the number of concurrent TCP connections.
-b : the requets to be send every second, which should be 1/60 of the concurrency.
-s : the time to establish the connections.
-e : the interval between two requests in every TCP connection.
-l : the request length
-r : the proportion of high-priority requests
--ips : the number of host ip address
--dest : the dest (server) ip address
--smp : the number of application thread used by MCC.
In iotepserver benchmark, we use HCMonitor to estimate the response latency. HCMonitor is an open-source high-performance monitor system which could accurately estimate the response latency with full traffic under high-concurrent workload.
README from HCMonitor)../build/monitor -c fff -n3 -- -p1
cdf.txt.We test iotepserver with 10,368,000 concurrent TCP connections on machines of Tested environments, and the 99th tail latency estimated by HCMonitor is about 2.4ms.
Add the configuration module to make application and qstack parameters configurable in .conf files.
Add following files in qstack/src:
config.cinclude/config.hMerge and optimize application oriented APIs, making them more user-friendly for porting.
Fix request drop bug caused by bitfield competition.
Add the callback check in the receiving path to indicate abnormal conditions.
Updated global macro.
The following macro in qstack/src/include/mbuf.h can cause zero window,
and it's disabled now:
MBUF_STREAMIDCheck MAC address for IP packets, accept broadcast ARP requests.
Ignore local path changes in tools.
Updated global macros.
The following two macros in qstack/src/include/global_macro.h can cause fatal errors, and they are disabled now:
SCAN_INACTIVE_SOCKETACTIVE_DROP_EMULATEUpdated dpdk-patch and dpdk_module.
The dpdk-patch was simplified, so that QStack doesn't modify DPDK too much now.
Added MAC and IP check when receiving a packet.
Removed uncompleted in-stack SSL.
QStack is a high-concurrency-and-low-latency user-level TCP stack for multicore systems. The version 1.0 is able to support 10 million concurrent TCP long connections, using a cloud-supported IoT application scenario. The work was supported by the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No. XDA06010401, the National Key Research and Development Program of China under Grant No. 2016YFB1000203 & 2017YFB1001602. We appreciated project leaders, Prof. Xu Zhiwei, Prof. Chen Mingyu and Prof. Bao Yungang, for their selfless support and guidance. We also appreciated the QStack team for their efforts. Among them, Shen Yifan is responsible for the system design and development of the TCP protocol stack, Song Hui is responsible for the event framework, Zhang Zhao is responsible for the user-level driver. Wang Zhuang, Feng Xiao and Wu Wenqing are responsible for the development of the high concurrency generator MCC, Song Hui is responsible for the development of full traffic monitor HCMonitor, which are important tools for QStack test. And Li Zixuan and Wang Haokun also help to developed some tools and do a large number of tests.
Issue board is the preferred way to report bugs and ask questions about QStack or contact Zhang Wenli (zhangwl at ict.ac.cn).
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。