Tips for performance

Save communication volume

It is best to avoid sending and receiving unnecessary messages. For example, for quests that are visible to the gamer, it is better to show the quest string as a pre-stored resource in the game client instead of sending the entire quest string. However, for games with multi-language support, this may be a necessity.

Dead reckoning helps reduce the number of messages sent and received. Data quantization helps reduce the size of a single message.

Multicast Optimization

When multicasting with RMI, it is much better to send to multiple hosts simultaneously with a single RMI function call. A big part of the cost of maintaining an MMO game server is the line speed from where the server is running (e.g. IDC). ProudNet has the following devices to optimize this.

- Multicast NPC Simulation Results

Normally, NPC (Non player character) is simulated on a server. The idea is to multicast the simulated results of the NPC to the clients, utilizing the P2P connections that each client is connected to.

- Multicast PC (Player Character) Simulation Results

Typically, a PC is simulated on each client. The client sends information about the PC's location, speed, etc. to the server, which receives it and multicasts it to clients in the PC's neighborhood.

However, ProudNet's P2P creation, destruction, recruitment, and subtraction processes are very fast. This allows each PC's location multicast to be sent directly in P2P communication to other clients that have each client's PC in their line of sight, bypassing the server.

Here are the main methods.

• At regular intervals, clients in each PC's line-of-sight form a P2P group.

• Each client sends the location, speed, etc. of the simulated PC to the server and its P2P group of clients.

• Based on the information the server receives, the server continuously creates, destroys, grows, and shrinks P2P groups.

• The client updates the location of the PC based on the information it receives.

Reference

Routed multicast from server to multiple clients Multicast

Game server operating system (OS)

If your CCUs are 100 or less, it does not matter what operating system you use, but if you go above that, you should use a server-specific operating system. The server-specific operating systems supported by ProudNet are Windows 2003 Server and later. Linux is also supported and has been tested on CentOS 6, Ubuntu 12 and above.

How the server uses UDP ports

When a client connects to the server, it should be assigned 1 TCP port, but ProudNet uses 1 UDP port to connect to the client. (However, in an environment where the client cannot use the UDP port, only the TCP port is used.)

The server separates the UDP port assignment policy for each client connection as follows.

• per-client assign mode: Assign a different UDP port to each client.

• static assign mode: Allocate a set number of reserved UDP ports for all clients.

- Static Assign Mode

Other than the list of UDP port numbers to be used by the server, no other UDP ports are used, but one from the list of UDP port numbers is used. This means that two or more clients will share the same UDP port. (There will be no problem with message flow for these clients.)

With a large number of clients, a small number of UDP sockets will have to cope with communicating with many clients, which can lead to packet loss if the buffer inside the UDP socket is overwhelmed. That said, you should be careful not to prepare too many UDP sockets from scratch, as the server's processing load for these many UDP sockets can increase memory and CPU usage.

Because tolerance to ICMP host unreachable packet is weak, an ICMP firewall setup must be carried out.

- Per-client Assign Mode

Each client connecting to the server is assigned a different UDP port, and a random port number is used in addition to the list of UDP port numbers the server uses. In general, we recommend using Per-client assign mode because it has many advantages over static assign mode and performs better, but there is a precaution. If more clients connect than the list of UDP port numbers available to the server, a random UDP port is assigned, and if the assigned port is not allowed by the server-side firewall, UDP communication may not be smooth.

Some server firewalls allow port usage only when outbound packet detection is present. This enables secure and seamless communication when UDP port authorization is turned off while using Per-client assign mode.

To set which assign mode the server will use, set Proud.CStartServerParameter.m_udpAssignMode when calling Proud.CNetServer.Start. And the list of UDP ports that the server will use can be set in Proud.CStartServerParameter.m_udpPorts.

When using ProudNet, we recommend the following types of assign mode, length of UDP port list, and firewall settings.

Recommended level	Assign Mode	Length of UDP port list	Firewall settings	Applicable conditions
Highly recommended	Per-client	(Ignore per-client) -	Temporarily allow outbound packet detection	Some physical servers, not cloud servers, that can turn on/off the 'Temporarily allow outbound packet detection' function
Recommended	Per-client	-	Always allow numbers listed in the UDP port list
Recommended	Static	1/10 of the number of concurrent connections	Always allow numbers in the UDP port list / Block ICMP host unreachable
Danger	Per-client	-	Always allow numbers listed in the UDP port list
Danger	Static	None	No UDP port allowed / Does not block ICMP host unreachable

Reference

ICMP firewall settings

Setting up a defender against DDOS attacks

Number of threads in the server thread pool

When you run the Proud.CNetServer.Start server, you can specify the number of threads in the thread pool. Depending on the role and functionality of the server, the number of server processes that can be launched on the server computer or the number of threads that it creates varies.

Some representative guides are as follows.

First, assume that the number of threads is equal to the number of CPU cores on the server.

Increase the number of threads if the server where the ProudNet server is used takes up device burst time rather than purely CPU burst time. A typical case of device burst time on a game server is when the user DB is being accessed.

If your game server's logic is structured so that it can be accessed by multiple threads at the same time, you can run only one game server process on the server machine. However, if your game server's logic is protected by critical sections or mutexes that allow only one thread to run at any given time, you may want to consider having multiple game server processes running.

Speed Hack Detection and Server Performance

ProudNet has an internal speed hack detection feature. It works by checking the frequency of ping messages sent back and forth between the client and server, sending pings from the client to the server at intervals of several seconds. If the frequency of ping messages arriving on the server varies by more than 30% of the interval, with more than 20 pings arriving, we consider that a Speed Hack is being used.

When a Speed Hack is detected in this way, the server event Proud.INetServerEvent.OnClientHackSuspected for that client is called so that a list of bad users can be collected or kicked (Proud.CNetServer.CloseConnection).

Detection can be difficult in the following situations.

• If you hack a computer at less than 20% speed, it will not be detected.

• A client with poor communication can be mistaken for using a speed Hack.

On the other hand, a gamer might intentionally use a speed hack very briefly. To detect this, the server compares the "client internal time value" and the "time value at the server" that the client exchanges since first connecting to the server.

If the difference between the "client internal time value" and the "server measured time value" that the client consistently sends to the server since the time of first connection is too large, it is considered a speed hack. However, this check method is implemented tens of seconds after the client connects to the server.

C++ function	C# function	Description
Proud.INetServerEvent.OnClientHackSuspected	Nettention.Proud.NetServer.ClientHackSuspectedHandler	The callback server event for a client with a detected speed hack
Proud.CNetServer.CloseConnection	Nettention.Proud.NetServer.CloseConnection	Kicking a user
Proud.CNetServer.SetSpeedHackDetectorReckRatioPercent	Nettention.Proud.NativeNetServer.SetSpeedHackDetectorReckRatioPercent	Adjusting the speed and accuracy of speed hack detection
Proud.CNetServer.EnableSpeedHackDetector	Nettention.Proud.NativeNetServer.EnableSpeedHackDetector	Speed Hack detector on/off

Speed hack detection causes a certain amount of traffic because clients send one to two UDP packets per second to the server, and the traffic increases proportionally to the number of clients connected to the server. Therefore, it is efficient to refrain from using them unless necessary.

With 10,000 clients connected to a server PC with Xeon E312XX (Sandy Bridge) memory 4G, we measured 0-3% CPU usage without Speed Hack Detector and 25-35% CPU usage with it.

Optimize incoming processing routines

Taking up a lot of time during an incoming processing routine can have a significant impact on server performance. First of all, it is important to find and resolve underperforming reception processing routines by accessing the receiving side (stub) call point.

Next, we need to find out if it is device burst time or CPU burst time that is accounting for the slow receive processing routine.

• If the device burst time is long, critical section lock must be designed to be minimized, and performance of device burst causes (databases, etc.) must be improved.

• If the CPU burst time is long, calculation routines need to be optimized, parallelized, or distributed to other servers. Example: NPC's AI

Reference

Burst Time

Using the Stay Connected feature

In a mobile environment, switching between cellular and Wi-Fi is frequent. Even in these environments, ProudNet handles network handovers automatically at the engine end.

C++

Before the user connects to the server, you only need to add the following code.

Proud::CNetConnectionParam cp;
cp.m_enableAutoConnectionRecovery = true;
 
client.Connect(cp);

For event functions during the handover process on the server, you can override the following functions in INetServerEvent.

virtual void OnClientOffline(CRemoteOfflineEventArgs &args) {}
virtual void OnClientOnline(CRemoteOnlineEventArgs &args) {}

Similarly, event functions on the client override the following functions in INetClientEvent.

virtual void OnServerOffline(CRemoteOfflineEventArgs &args) {}
virtual void OnServerOnline(CRemoteOnlineEventArgs &args) {}
virtual void OnP2PMemberOffline(CRemoteOfflineEventArgs &args) {}
virtual void OnP2PMemberOnline(CRemoteOnlineEventArgs &args) {}

C#

Enabling the Stay Connected feature can be set through the Nettention.Proud.NetConnectionParam.enableAutoConnectionRecovery variable in DotNet NetClient, and no separate setting is required in NetServer.

Nettention.Proud.NetConnectionParam cp = new Nettention.Proud.NetConnectionParam();
cp.protocolVersion.Set(SimpleCSharp.Vars.m_Version);

cp.serverIP = "127.0.0.1";
cp.serverPort = (ushort)SimpleCSharp.Vars.m_serverPort;
cp.enableAutoConnectionRecovery = true;

Nettention.Proud.ErrorInfo errorInfo = new Nettention.Proud.ErrorInfo();

if (Client.Connect(cp, errorInfo) == false)
{
    Console.WriteLine("Failed to connect client ~!!\n");
}
...

The following events are callbacks related to the Stay Connected feature.

NetClient	NetServer
Nettention.Proud.NetClient.ServerOfflineHandler	Nettention.Proud.NetServer.ClientOfflineHandler
Nettention.Proud.NetClient.ServerOnlineHandler	Nettention.Proud.NetServer.ClientOnlineHandler
Nettention.Proud.NetClient.P2PMemberOfflineHandler	-
Nettention.Proud.NetClient.P2PMemberOnlineHandler	-

...
Client.ServerOfflineHandler = OnServerOffline;
Client.ServerOnlineHandler = OnServerOnline;

Client.P2PMemberOfflineHandler = OnP2PMemberOffline;
Client.P2PMemberOnlineHandler = OnP2PMemberOnline;

public void OnServerOffline(Nettention.Proud.RemoteOfflineEventArgs args){}
public void OnServerOnline(Nettention.Proud.RemoteOnlineEventArgs args){}
public void OnP2PMemberOffline(Nettention.Proud.RemoteOfflineEventArgs args){}
public void OnP2PMemberOnline(Nettention.Proud.RemoteOnlineEventArgs args){}
...

...
server.ExceptionHandler = OnException;
server.ClientOnlineHandler = OnClientOnline;
server.ClientOfflineHandler = OnClientOffline;
server.ReceiveUserMessageHandler = OnReceiveUserMessage;

...

public void OnClientOnline(Nettention.Proud.RemoteOnlineEventArgs args){}
public void OnClientOffline(Nettention.Proud.RemoteOfflineEventArgs args){}

After the client's network is disconnected and Offline is called, INetClientEvent.OnLeaveServer, INetServerEvent.OnClientLeave will be called if the client does not reconnect for a period of time.

In versions 1.7.36365 and later, servers can set a connection hold time for each client.

NetServer.SetAutoConnectionRecoveryTimeoutTimeMs(HostID, int)
NetServer.SetDefaultAutoConnectionRecoveryTimeoutTimeMs(int)

Networker Thread

ProudNet's clients have a worker thread that handles network I/O internally. This serves to facilitate network connectivity and ping latency measurements without requiring the user to call Proud.CNetClient.FrameMove() at regular intervals.

However, on some low-end hardware, such as the iPhone 3GS, it is not recommended to have more than one thread working at the same time because having multiple threads working can adversely affect performance. Instead, run the work of the networker thread in the main thread of your app.

• Call Proud.CNetClient.UseNetworkerThread_EveryInstance(false), then networker thread will no longer work.

• Call Proud.CNetClient.NetworkerThreadHeartbeat_EveryInstance() frequently. If you need to call this function for an unavoidably long period of time, such as when loading game data, call UseNetworkerThread_EveryInstance(true) beforehand and UseNetworkerThread_EveryInstance(false) after you are done.

When the server is equipped with multiple LAN cards (NICs)

Sometimes game servers are equipped with multiple LAN cards (NICs). In this case, you need to specify which NIC to use at server startup, while allowing only some of them to communicate with gamers for security purposes. If this is not specified, clients may be able to connect to the server, but UDP or P2P communication may not work properly.

You can specify which NICs to use through the input parameter Proud.CStartServerParameter.m_localNicAddr of Proud.CNetServer.Start. To get a list of the local addresses of the NICs to use, it is helpful to use Proud.CNetUtil.GetLocalIPAddresses.

Setting up a server behind a NAT router or L4 switch

When running a server on a development device behind an Internet router, you want to place the server behind an Internet router (NAT router), or the server needs to operate on a server device behind an L4 switch. Or, if the server operates on a cloud server that requires port forwarding, you will set up port forwarding on the router.

This requires additional configuration on CNetServer, without which Unreliable messaging will always use TCP. P2P messaging will also always use TCP relay.

Consider the illustration below, assuming Server1 and Server2 are behind the router. Each server has a private address of 100.10.0.1 and 100.10.0.2, and the router performs port forwarding to 11.22.33.44:5555 and 11.22.33.44:5556 for each server.

Port forward

Include the parameter Proud.CStartServerParameter.m_serverAddrAtClient that is passed in when calling CNetServer.Start. For the illustration above, we need to put in the public address of the NAT router, 11.22.33.44, and we will also be forwarding UDP ports to the NAT router.

UDP port forward

If the port forwarding is in the range of 6,000 - 6,500 as shown in the picture above, CNetServer must be set to use UDP port 6,000 - 6,500. To do this, set UDP port assign mode to static. And set the port to be used by CNetServer between 6,000 and 6,500.

Example

C++

Proud::CNetServer* s;

...

Proud::CStartServerParameter p;

...

p.m_serverAddrAtClient = "11.22.33.44";
p.m_udpAssignMode = Proud::ServerUdpAssignMode::ServerUdpAssignMode_Static;
for(int i=6000;i<=6500;i++)
    p.m_udpPorts.Add(i);
 
s->Start(p);

C#

Nettention.Proud.NetServer s = new Nettention.Proud.NetServer();

...

Nettention.Proud.StartServerParameter p = new Nettention.Proud.StartServerParameter();

...

p.serverAddrAtClient = "11.22.33.44";
p.udpAssignMode = Nettention.Proud.ServerUdpAssignMode.ServerUdpAssignMode_Static;
for(int i = 6000; i <= 6500; i++)
    p.udpPorts.Add(i);
 
s.Start(p);

For operational verification, two clients form a P2P group and check if they go to and from P2P when sending and receiving messages. Due to the nature of ProudNet, messages to and from P2P change to direct P2P only after a few seconds.

When receiving a message over P2P, check that the value of m_relayed in the parameter RmiContext changes to false, or that OnChangeP2PRelayState is called and the parameter contains OK.

If you are not getting the results you want, you can use CNetServer.EnableLog to look through the logs to find the source of the problem.

Unlike what is behind a router or switch, if you have a server behind a load balancer, the ability to stay connected might not work.