- Notifications
You must be signed in to change notification settings - Fork101
Fork from official nginx-1.12.2, and run on the dpdk user space TCP/IP stack(ANS"accelerated network stack").
License
NotificationsYou must be signed in to change notification settings
ansyun/dpdk-nginx
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
dpdk-nginx fork from official nginx-1.9.5, and run on the dpdk user space TCP/IP stack(ANS). For detail function, please refer to nginx official website(http://nginx.org/).
- Download latest dpdk version fromdpdk website
$ make config T=x86_64-native-linuxapp-gcc$ make install T=x86_64-native-linuxapp-gcc$ export RTE_SDK=/home/mytest/dpdk$ export RTE_TARGET=x86_64-native-linuxapp-gcc- Build dpdk and ANS following theANS wiki
$ git clone https://github.com/ansyun/dpdk-ans.git$ export RTE_ANS=/home/mytest/dpdk-ans$ ./install_deps.sh$ cd ans$ make$ sudo ./build/ans -c 0x2 -n 1 -- -p 0x1 --config="(0,0,1)"EAL: Detected lcore 0 as core 0 on socket 0EAL: Detected lcore 1 as core 1 on socket 0EAL: Support maximum 128 logical core(s) by configuration.EAL: Detected 2 lcore(s)EAL: VFIO modules not all loaded, skip VFIO support...EAL: Setting up physically contiguous memory...EAL: Ask a virtual area of 0x400000 bytesEAL: Virtual area found at 0x7fdf90c00000 (size = 0x400000)EAL: Ask a virtual area of 0x15400000 bytes- Download dpdk-nginx, build dpdk-nginx
$ git clone https://github.com/ansyun/dpdk-nginx.git$ ./configure --with-http_dav_module$ make$ make install # default install dir is /usr/local/nginx- Setup DPDK Environment
Refer toGetting Started Guide for Linux
- Startup ANS TCP/IP stack
$ sudo ./build/ans -c 0x2 -n 1 -- -p 0x1 --config="(0,0,1)"EAL: Detected lcore 0 as core 0 on socket 0EAL: Detected lcore 1 as core 1 on socket 0EAL: Support maximum 128 logical core(s) by configuration.EAL: Detected 2 lcore(s)EAL: VFIO modules not all loaded, skip VFIO support...EAL: Setting up physically contiguous memory......- startup nginx
$ sudo ./objs/nginxEAL: Detected lcore 0 as core 0 on socket 0EAL: Detected lcore 1 as core 1 on socket 0....2015/11/09 19:04:37 [notice] 1812#0: OS: Linux 3.16.0-30-generic2015/11/09 19:04:37 [notice] 1812#0: getrlimit(RLIMIT_NOFILE): 1024:4096- Test http connection by ab tool
CPU:Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz.NIC:Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ANS run on a lcore.# ab -n 30000 -c 500 10.0.0.2:80/This is ApacheBench, Version 2.3 <$Revision: 1796539 $>Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking 10.0.0.2 (be patient)Completed 3000 requestsCompleted 6000 requestsCompleted 9000 requestsCompleted 12000 requestsCompleted 15000 requestsCompleted 18000 requestsCompleted 21000 requestsCompleted 24000 requestsCompleted 27000 requestsCompleted 30000 requestsFinished 30000 requestsServer Software: nginx/1.12.2Server Hostname: 10.0.0.2Server Port: 80Document Path: /Document Length: 612 bytesConcurrency Level: 500Time taken for tests: 0.851 secondsComplete requests: 30000Failed requests: 0Total transferred: 25350000 bytesHTML transferred: 18360000 bytesRequests per second: 35262.84 [#/sec] (mean)Time per request: 14.179 [ms] (mean)Time per request: 0.028 [ms] (mean, across all concurrent requests)Transfer rate: 29098.73 [Kbytes/sec] receivedConnection Times (ms) min mean[+/-sd] median maxConnect: 3 6 0.9 6 10Processing: 2 8 1.5 8 17Waiting: 2 6 1.5 7 15Total: 8 14 1.8 14 24Percentage of the requests served within a certain time (ms) 50% 14 66% 14 75% 15 80% 15 90% 16 95% 17 98% 18 99% 19 100% 24 (longest request)- Test file download by wget tool
CPU:Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz.NIC:Intel Corporation 82576 Gigabit Network Connection (rev 01) ANS run on a lcore.root@h163:~# wget http://2.2.2.2/nginx_big_data--2016-01-02 20:58:24-- http://2.2.2.2/nginx_big_dataConnecting to 2.2.2.2:80... connected.HTTP request sent, awaiting response... 200 OKLength: 44046102 (42M) [application/octet-stream]Saving to: ‘nginx_big_data.1’100%[=====================================>] 44,046,102 111MB/s in 0.4s2016-01-02 20:58:24 (111 MB/s) - ‘nginx_big_data.1’ saved [44046102/44046102]root@h163:~#- dpdk-nginx CPS performance
CPU: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHzNIC:82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ANS run on a lcore.4 dpdk-nginx run on ANS.# ./wrk --timeout=1 --latency -H "Connection: close" -t20 -c100 -d30s http://10.0.0.2Running 30s test @ http://10.0.0.2 20 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 480.80us 73.12us 4.66ms 88.86% Req/Sec 5.32k 125.94 6.98k 84.30% Latency Distribution 50% 478.00us 75% 505.00us 90% 535.00us 99% 648.00us 3186335 requests in 30.10s, 2.51GB readRequests/sec: 105860.26Transfer/sec: 85.31MBANS run on two lcore.8 dpdk-nginx run on ANS.# ./wrk --timeout=1 --latency -H "Connection: close" -t20 -c100 -d30s http://10.0.0.2Running 30s test @ http://10.0.0.2 20 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 251.70us 89.45us 4.39ms 68.59% Req/Sec 10.28k 340.17 12.40k 67.67% Latency Distribution 50% 246.00us 75% 310.00us 90% 363.00us 99% 480.00us 6155775 requests in 30.10s, 4.84GB readRequests/sec: 204512.88Transfer/sec: 164.81MB- dpdk-nginx QPS performance
CPU: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHzNIC:82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ANS run on a lcore.8 dpdk-nginx run on ANS.# ./wrk --timeout=1 --latency -t20 -c100 -d30s http://10.0.0.2Running 30s test @ http://10.0.0.2 20 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 257.91us 407.36us 24.37ms 98.77% Req/Sec 20.85k 1.84k 26.88k 70.23% Latency Distribution 50% 214.00us 75% 289.00us 90% 338.00us 99% 825.00us 12488349 requests in 30.10s, 9.89GB readRequests/sec: 414900.42Transfer/sec: 336.31MBANS run on two lcore.10 dpdk-nginx run on ANS.# ./wrk --timeout=1 --latency -t20 -c100 -d30s http://10.0.0.2Running 30s test @ http://10.0.0.2 20 threads and 100 connections Thread Stats Avg Stdev Max +/- Stdev Latency 184.60us 165.39us 12.97ms 98.92% Req/Sec 26.69k 0.93k 29.92k 79.17% Latency Distribution 50% 186.00us 75% 200.00us 90% 216.00us 99% 370.00us 15985217 requests in 30.10s, 12.65GB readRequests/sec: 531077.97Transfer/sec: 430.48MB- Shall use the same gcc version to compile your application.
- ANS tcp stack support reuseport, so can enable nginx reuseport feature, multi nginx can listen on same port.
- proxy_pass is supported.
- In order to improve ANS performance, you shall isolate ANS'lcore from kernel by isolcpus and isolcate interrupt from ANS's lcore by update /proc/irq/default_smp_affinity file.
- You shall include dpdk libs as below way because mempool lib hasattribute((constructor, used)) in dpdk-16.07 version, otherwise your application would coredump.
$(RTE_ANS)/librte_anssock/librte_anssock.a \ -L$(RTE_SDK)/$(RTE_TARGET)/lib \ -Wl,--whole-archive -Wl,-lrte_mbuf -Wl,-lrte_mempool -Wl,-lrte_ring -Wl,-lrte_eal -Wl,--no-whole-archive -Wl,-export-dynamic \- In order to establish mass TCP connection in linux kernel, shall change these parameters as below.
vi /etc/sysctl.conf#/sbin/sysctl -pnet.ipv4.ip_local_port_range = 1024 65535net.ipv4.tcp_syncookies = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_fin_timeout = 30For free support, please use ANS team mail list atanssupport@163.com, or QQ Group:86883521, orhttps://dpdk-ans.slack.com.
About
Fork from official nginx-1.12.2, and run on the dpdk user space TCP/IP stack(ANS"accelerated network stack").
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
No packages published
Contributors3
Uh oh!
There was an error while loading.Please reload this page.