In my previousarticle, we discussed endianness, its importance, and how to work with it. Understanding endianness is crucial for dealing with data at the byte level, especially in network programming. We examined several examples that highlighted how endianness affects data interpretation.

One key area we focused on was the relationship between endianness andTCP stream sockets. We explained why thebind syscall expects certain information for the INET family to be in a specific endianness, known as network byte order. This ensures that data is correctly interpreted across different systems, which might use different native endianness.

One of the next interesting topics after binding the socket is putting it into listening mode with thelisten syscall. In this article, we are going to dive a little deeper into this next phase.

Listen: Handshake and SYN/Accept Queues

Recall that TCP is connection-oriented protocol. This means the sender and receiver need to establish a connection based on agreed parameters through athree-way handshake. The server must be listening for connection requests from clients before a connection can be established, and this is done using thelisten syscall.

We won't delve into the details of the three-way handshake here, as many excellent articles cover this topic in depth. Additionally, there are related topics likeTFO (TCP Fast Open)andsyncookies that modify its behavior. Instead, we'll provide a brief overview. If you're interested, we can explore these topics in greater detail in future articles.

The three-way handshake involves the following steps:

1. The client sends a SYN (synchronize) packet to the server, indicating a desire to establish a connection.
2. The server responds with a SYN-ACK (synchronize-acknowledge) packet, indicating a willingness to establish a connection.
3. The client replies with an ACK (acknowledge) packet, confirming it has received the SYN-ACK message, and the connection is established.

Do you remember in the first article when we mentioned that thelisten operation involves the kernel creating two queues for the socket. The kernel uses these queues to store information related to the state of the connection.

The kernel creates these twoqueues:

• SYN Queue: Its size is determined by a system-wide setting. Although referred to as a queue, it is actually a hash table.
• Accept Queue: Its size is specified by the application. We will discuss this further later. It functions as a FIFO (First In, First Out) queue for established connections.

As shown in the previous diagram, these queues play a crucial role in the process. When the server receives aSYN request from the client, the kernel stores the connection in theSYN queue. After the server receives anACK from the client, the kernel moves the connection from theSYN queue to theaccept queue. Finally, when the server makes theaccept system call, the connection is taken out of the accept queue.

Accept Queue Size

As mentioned, these queues have size limits, which act as thresholds for managing incoming connections. If the size limit of thequeues are exceeded, the kernel may discard or drop the incoming connection request or return an RST (Reset) packet to the client, indicating that the connection cannot be established.

If you recall our previous examples where we created a socket server, you'll notice that we specified a backlog value and passed it to thelisten function. This backlog value determines the length of the queue for completely established sockets waiting to be accepted, known as the accept queue.

// Listen for incoming connectionsletbacklog=Backlog::new(1).expect("Failed to create backlog");listen(&socket_fd,backlog).expect("Failed to listen for connections");

If we inspect theBacklog::new function, we will uncover some interesting details. Here's the function definition:

...pubconstMAXCONN:Self=Self(libc::SOMAXCONN);.../// Create a `Backlog`, an `EINVAL` will be returned if `val` is invalid.pubfnnew<I:Into<i32>>(val:I)->Result<Self>{cfg_if!{if#[cfg(any(target_os="linux",target_os="freebsd"))]{constMIN:i32=-1;}else{constMIN:i32=0;}}letval=val.into();if!(MIN..Self::MAXCONN.0).contains(&val){returnErr(Errno::EINVAL);}Ok(Self(val))}

We can notice a few things:

• On Linux and FreeBSD systems, the minimum acceptable value is-1, while on other systems, it is0. We will explore the behavior associated with these values later.
• The value should not exceedMAXCONN,which is the default maximum number of established connections that can be queued. Unlike inC, where a value greater thanMAXCONN is silently capped to that limit, in Rust'slibc crate, exceedingMAXCONN results in an error.
• Interestingly, in thelibc crate,MAXCONN is hardcoded to4096. Therefore, even if we modifymaxconn, this won't affect the listening socket, and we won't be able to use a greater value.

TheSOMAXCONN Parameter

Thenet.core.somaxconn kernel parameter sets the maximum number of established connections that can be queued for a socket. This parameter helps prevent a flood of connection requests from overwhelming the system. The default value fornet.core.somaxconn is defined by theSOMAXCONN constanthere .

You can view and modify this value using thesysctl command:

$sudosysctl net.core.somaxconnnet.core.somaxconn= 4096$sudosysctl-w net.core.somaxconn=8000net.core.somaxconn= 8000

Analyzing the Accept Queue with example

If we run the examplelisten-not-accept, which sets the backlog to1 without callingaccept, and then usetelnet to connect to it, we can use thess command to inspect the queue for that socket.

# server$cargo run--bin listen-not-acceptSocket file descriptor: 3# telnet$telnet 127.0.0.1 6797Trying 127.0.0.1...Connected to 127.0.0.1.Escape character is'^]'.

Usingss:

State                      Recv-Q                     Send-Q                                          Local Address:Port                                            Peer Address:Port                     Process                     LISTEN                     1                          1                                                   127.0.0.1:6797                                                 0.0.0.0:*

• Recv-Q: The size of the current accept queue. This indicates the number of completed connections waiting for the application to callaccept(). Since we have one connection fromtelnet and the application is not callingaccept, we see a value of 1.
• Send-Q: The maximum length of the accept queue, which corresponds to the backlog size we set. In this case, it is 1.

Testing Backlog Behavior on Linux

We are going to execute the following code to dynamically test the backlog configuration:

fnmain(){println!("Testing backlog...");forbacklog_sizein-1..5{letserver_socket=socket(AddressFamily::Inet,SockType::Stream,SockFlag::empty(),SockProtocol::Tcp,).expect("Failed to create server socket");letserver_address=SockaddrIn::from_str("127.0.0.1:8080").expect("...");bind(server_socket.as_raw_fd(),&server_address).expect("...");listen(&server_socket,Backlog::new(backlog_size).unwrap()).expect("...");letmutsuccessful_connections=vec![];letmutattempts=0;loop{letclient_socket=socket(AddressFamily::Inet,SockType::Stream,SockFlag::empty(),SockProtocol::Tcp,).expect("...");attempts+=1;matchconnect(client_socket.as_raw_fd(),&server_address){Ok(_)=>successful_connections.push(client_socket),Err(_)=>break,}}println!("Backlog {} successful connections: {} - attempts: {}",backlog_size,successful_connections.len(),attempts);}}

The key functions of the code are:

• It iterates over a range of backlog values, creating a server socket and putting it in listen mode with each specific backlog value.
• It then enters a loop, attempting to connect client sockets to the server socket. If a connection is successful, it adds the client socket to thesuccessful_connections vector. If it fails, it breaks the loop.

This allows us to determine the exact number of successful connections for each backlog configuration.

When we run the program, we will see output similar to the following. Note that this process may take a few minutes:

$cargo run--bin test-backlogBacklog-1 successful connections: 4097 - attempts: 4098Backlog 0 successful connections: 1 - attempts: 2Backlog 1 successful connections: 2 - attempts: 3Backlog 2 successful connections: 3 - attempts: 4Backlog 3 successful connections: 4 - attempts: 5Backlog 4 successful connections: 5 - attempts: 6

With the above output you can notice two things:

• When listen receive the the min acceptable value-1 it will use the default value ofSOMAXCONN, we already know if we pass a value greather thanSOMAXCONN it will return an error.
• And the other "weird" that can notice is that always the connections are 1 more than the limit set by the backlog, for example, backlog is 3 but there are 4 connections,

The last result is due to the behavior of thisfunction in Linux. Thesk_ack_backlog represents the current number of acknowledged connections, while thesk_max_ack_backlog represents our backlog value. For example, when the backlog is set to 3, the function checks ifsk_ack_backlog > sk_max_ack_backlog. When the third connection arrives, this check (3 > 3) evaluates to false, meaning the queue is not considered full, and it allows one more connection.

staticinlineboolsk_acceptq_is_full(conststructsock*sk){returnREAD_ONCE(sk->sk_ack_backlog)>READ_ONCE(sk->sk_max_ack_backlog);}

Handling a Full Accept Queue

As we can see from our program's logs, the client will attempt to connect to the server one more time than the specified backlog value. For example, if the backlog is 4, it will attempt to connect 5 times. In the last attempt, the client and server go through the handshake process: the client sends theSYN, the server responds withSYN-ACK, and the client sends anACK back, thinking the connection is established.

Unfortunately, if the server'saccept queue is full, it cannot move the connection from theSYN queue to theaccept queue. The server's behavior when the accept queue overflows is primarily determined by thenet.ipv4.tcp_abort_on_overflow option.

Normally, this value is set to0, meaning that when there is an overflow, the server will discard the client'sACK packet from the third handshake, acting as if it never received it. The server will then retransmit theSYN-ACK packet. The number of retransmissions the server will attempt is determined by thenet.ipv4.tcp_synack_retries option.

We can reduce or increase thetcp_synack_retries value. By default, this value is set to 5 on Linux.

In our final attempt, when the retransmission limit is reached, the connection is removed from theSYN queue, and the client receives an error. Again, this is an oversimplified overview of the process. The actual process is much more complex and involves additional steps and scenarios. However, for our purposes, we can keep things simpler.

By running the commandnetstat -s | grep "overflowed", we can check for occurrences of socket listen queue overflows. For example, in my case, the output indicates that"433 times the listen queue of a socket overflowed," highlighting instances where the system's accept queue was unable to handle incoming connection requests. This can lead to potential connection issues.

$netstat-s |grep"overflowed"433timesthe listen queue of a socket overflowed

Testing Backlog Behavior on Mac

If we execute the same program on a Mac, we will observe some differences:

• As mentioned, for other systems, the minimum acceptable value is0. Using this value gives us the default setting.
• The default value forSOMAXCONN is128, which can be retrieved using the commandsysctl kern.ipc.somaxconn.
• Lastly, on a Mac, we don't encounter the issue of allowingBacklog + 1 connections. Instead, we accept the exact number of connections specified in the backlog.

$cargo run--bin test-backlogBacklog 0 successful connections: 128 - attempts: 129Backlog 1 successful connections: 1 - attempts: 2Backlog 2 successful connections: 2 - attempts: 3Backlog 3 successful connections: 3 - attempts: 4Backlog 4 successful connections: 4 - attempts: 5

SYN Queue Size Considerations

As thelisten man page states, the maximum length of the queue for incomplete sockets(SYN queue) can be set using thetcp_max_syn_backlog parameter. However, whensyncookies are enabled, there is no logical maximum length, and this setting is ignored. Additionally, other factors influence the queue's size. As mentioned earlier:its size is determined by a system-wide setting.

You can find the code for this example and future ones in thisrepo.

To Conclude

In this article, we explored the role of TCP queues in connection management, focusing on theSYN queue and theaccept queue. We tested various backlog configurations and examined how different settings impact connection handling. Additionally, we discussed system parameters liketcp_max_syn_backlog andnet.ipv4.tcp_abort_on_overflow, and their effects on queue behavior.

Understanding these concepts is crucial for optimizing server performance and managing high traffic scenarios. By running practical tests and using system commands, we gained insights into how to handle connection limits effectively.

As we move forward, keeping these fundamentals in mind will help ensure robust and efficient network communication.

Thank you for reading along. This blog is a part of my learning journey and your feedback is highly valued. There's more to explore and share regarding socket and network, so stay tuned for upcoming posts. Your insights and experiences are welcome as we learn and grow together in this domain. Happy coding!