The seamless load balancer.
With PikkoServer, the development of a seamless MMO will be easy, faster, and able to handle many more players.
Overview.
PikkoServer is made for enabling the development of large-scale MMOs using existing engine technologies. It works as a load-balancer that transforms a simple single server based architecture into a distributed set of servers working together to provide scalability to large numbers of players. PikkoServer is inherently engine-agnostic and could be integrated into virtually any game engine. It is currently integrated with the Unity engine, but could be used to transform any multiplayer engine into a seamless MMO engine.
Workflow.
The development of a game using PikkoServer takes place in the same environment as the engine normally uses. The only requirement for the game server programmer is to implement a rather simple API for talking to PikkoServer, which then takes the burden of load balancing, object distribution over servers, handovers between servers and more. This greatly simplifies the work that has to be done by the game server programmer. The work to be done on the client is even smaller than the server, since the engine’s existing network infrastructure is reused.
PikkoServer offers horizontal scalability on the server side, and allows for game developers to basically program single threaded game server code and still get a many-core architecture, which greatly simplifies and increases the speed of development. The server side becomes a set of distributed game servers and the network traffic to these servers is routed by PikkoServer as described in more detail here: tech summary, and here: scalability.
Game tick & bandwidth.
With PikkoServer, the number of players that can interact with each other in one shared space depends on the bandwidth limitation, and how well the gameplay can handle network . As a reference, in our world record game we send updates for the closest objects up to 15 times per second whereas objects further away aren’t updated more than 3 times per second. The total client downstream bandwidth is restricted to 20 KB/s for each client. This way the client can easily handle the updates from the 999 other players.
Different game designs and game scenarios require different network capabilities. The world record game has a simple game design but requires a lot of traffic management. Other game designs have different interactivity patterns and thus require different traffic patterns. PikkoServer can be configured to handle different types of scenarios.
Game design limitations.
Depending on your gameplay design the bottleneck on the server side is either limited by bandwidth or CPU. We have covered many different scenarios with different limitations in the document Horizontal Scalability. PikkoServer can often help a game scale both when the game is bandwidth bound and when it is CPU bound.
To simplify studies of communication patterns in network games we classify traffic by message type into two classes. Messages of a given message type are classified as either direct or indirect and either discrete or continuous:
- Direct communication - well-defined receivers, like personal delivery.
- Indirect communication - no well-defined receiver, like a billboard, available for anyone who might be interested.
- Discrete communication - discrete events, like buying things, it’s either yours or it’s not yours.
- Continuous communication - non-discrete events, like gossip and rumors, new information replaces old information.
To receive indirect information an object typically needs to subscribe to certain types of events within a certain area. It can also be possible to query a server on indirect information of a certain type. The classes are combined into four patterns where we assign different message types to.
Typical message types for different patterns:
- Direct-discrete - multicast: chat message, trading, object creation/destruction.
- Indirect-discrete - broadcast: visual and audible effects, gunfire, missile launches.
- (Direct-continuous - multicast: information about a party - often low frequency.)
- Indirect-continuous - broadcast: position and character state - often high, or very high, frequency, such as up to every single game tick for every single object, most of which will be replaced well before it’s actually sent.
These patterns can then be handled by PikkoServer with different filtering capabilities. Typical filtering configurations for different patterns:
- Direct-discrete - not filtered at all, hence use these sparsely. To scale somewhat nicely the number of direct-discrete broadcasts per minute need to be less than a low fixed number.
- Indirect-discrete - might not always be delivered to everyone if they are too many. Different message types may have different priority rules.
- Direct-continuous - will be delivered, but not right away.
- Indirect-continuous - will be delivered, but not right away.
Note that a direct-discrete broadcast is the only broadcast type that can guarantee that everyone actually receives that specific message. Continuous type broadcasts will be delivered, but not every single one of them.
Persistent communication - Some amount of communication will often go to/from a database as well. This type of communication can also be categorized using the same patterns as above but are handled by defining such communications as specific message types as different message types are filtered individually. Thus a persistent message type sent to/from a database is just handled as any other message with a given pattern.
If the game design allows for the main bulk of the traffic to be communicated as continuous or indirect-discrete communication then PikkoServer can help a lot to filter which packages to deliver first.
Zones.
A zone is a collection of cell servers that together simulate a seamless world. Multiple zones can be instantiated to cover regions without seamless boundaries or to cover the same region in different parallel instances.
Zone sizes.
The area size of each zone is limited by the client’s physics engine. PikkoServer puts no inherent restriction on the size of zones and can handle working with coordinates spanning over large regions.
Recovery of cell servers.
If you reach a situation where all the cell servers handling one game zone are overloaded, the most important thing is fast recovery. In such a situation, prioritization of incoming messages is secondary to handling outgoing traffic. Slowing down the game tick on both client and server for one zone is a very graceful safety net to have when handling extreme situations, but the idea behind PikkoServer is that you can add more cell servers dynamically to help cope with load during the peak. Such additional cell servers can be added and removed in a controlled manner by human operators or automatic monitoring tools at runtime.
Hardware setup.
A PikkoServer architecture support all types of hardware interconnects. It is mainly a question of latency versus cost. If you want minimal latency addition, host PikkoServer plus 10-40 cell servers on one modern hardware with a total of 20-50 cores. This is a powerful setup for high player density zoned MMOs that demands a minimum of latency. If minimal hardware cost is more important, host PikkoServer on an 8 core machine and the cell servers on multiple 4 or 8-core blade servers, all connected by a Gigabit LAN switch. The latency for this distributed setup will be a handful of extra milliseconds.
Trading.
Trading via “auction houses” are mainly a database performance issue. PikkoServer does not demand any specific database product, therefore the performance of the auction house is completely up to the database product and the skills of the database programmers building the auction house features. We recommend using a modern scalable database like Riak for auction houses. For this reason we have created uGameDB, a custom Riak client for Unity that is especially engineered for game developers. You can read more about uGameDB here.
Trading with nearby avatars in the game is handled like any other interaction and PikkoServer has no specific limitations for this kind of activity.
Zone-to-zone transfers.
The zone-to-zone transfer mechanism is very robust. We have built the transfer mechanism to handle 300 player transfers per second when each zone is powered by a PikkoServer and several cell servers. And there is no limit set for the number of zones PikkoServer can handle. Not to be confused with cell-to-cell transfers which are called handovers and are seamless. Handovers are even faster than zone-to-zone transfers.
Zone management - the Mast algorithm.
PikkoServer uses a model similar to how cellular networks work for determining the regions handled by each cell server. In a cellular network, a mobile phone is always connected to the closest physical mast, and is automatically switched between masts when moving around. The same principle is used for cell servers. Each cell server is assigned an associated "mast" in the virtual world by PikkoServer, and players and other game objects are then handled by the server with the closest mast. When a player moves away from its current mast toward another, a handover is initiated by PikkoServer and the player is moved to the new server.
The regions handled by cell servers in each zone are not static, they are continuously changed (five times per second to ensure smooth incremental changes) to adapt to the current situation in the zone. PikkoServer moves the masts around in the zone in order to balance the amount of players handled by each cell server. This dynamic nature of the masts makes for a very flexible and scalable architecture that always adapts to the current gameplay.
For masts to be able to move around quickly, the cell servers governing a zone need to have the entire zone loaded into memory for the duration of the game. In this way, they will be able to handle any part of the zone equally well as any other and will be able to accept players entering from anywhere. Although it is possible for cell servers to only load the part of the zone that currently contains players, and dynamically load in new parts as players move around.
Scaling handovers.
The amount of handovers (movement of object from one cell to another) required for a game depends on movement patterns. Cell servers tries to follow movements to minimize handovers and inter-cell communication (communication to an object on another cell is inter-cell communication). It is important to know how the number of handovers and inter-cell communication scales with differnt game patterns. As an example we will answer how the number of handovers depends on the density of players in our world record game with a scripted load testing setup (configured using uTsung) where we have setup:
1) 2-dimensional playable area, aspect ratio is 1:1
2) 999 scripted players, 1 observer
3) all players move in random directions with a constant velocity and
4) all players are evenly distributed over the playable area
3 and 4 makes it impossible for the cell servers to follow any meaningful groups of objects which gives:
5) the cell areas will be evenly distributed with equal size, and they will not move
Let’s analyze this setup on this 2D grid with the length unit being ‘1 m’ (similar analysis goes for a 3D space as well).
Let
be the number of players. We have
as the average player density within the playable area. Where 
denotes the size of the playable area. We can estimate the total length of borders between cells within the playable area for 8 equally sized cells to
.
And let
be the average player speed. Using the assumption that all velocities are uniformly distributed in all directions on the plane we can compute an average velocity perpendicular against a border element from one direction
The number of players crossing a border element from one direction per second is then
.
As borders are crossed from both directions we have the total length of border elements being crossed as
.
This gives us that the total number of handovers per second is
Note that both 
, 
and 
are functions of . This equation can be used to find how the number of handovers depends on the density of players for different sizes of the playable area using
.
With this we obatin an equation for as a function of
.
Or to simplify
.
A similiar setup in 3D space (instead of on a 2D grid as in the world record game) ends up with
.
That is, the number of handovers scales with the square root of player density in this setup and with the cubic root in an analogous free space setup. Below we have plotted this relationship for the world record game setup in a graph for player densities corresponding to a playable area of size:
Try it out for yourself with our Octave-script handoverdensity.m.
To illustrate this we have recorded an early test we made where 1000 players moved towards the center. This test confirmed that the number of handovers didn’t go haywire as we shrunk the playable area to 2% of the designed playable area, and the required speed of the data stream to the client stayed constant: video showing 1000 clients going towards the middle (lots of explosions makes the camera shake a lot at times).
Client side limitations.
The client side limitations are in manly two areas: One is simulating physics for a large number of objects (often CPU bottleneck) and the second area is rendering many objects and many visual effects at the same time (often GPU bottleneck). There is a long list of optimizations that can be done in the client for handling a large number of moving objects and effects per second together with animations. To show examples of this, we added such client load to our world record game. You’ll see video demonstrations of this very soon.
For the client to be able to handle large seamless game worlds, it helps if the physics engine support double precision floating point numbers. Various tricks can be pulled off to make it work without double precision but it is harder. For instance, if your game has a map that is 10 km wide you can’t rely on getting more than 1 cm-precision in the physics simulations 10 km from the origin if your physics simulations run on single precision floating point numbers.
Server side limitations.
The server is made out of two parts, PikkoServer and several cell servers. Let’s look at the PikkoServer limitations first. PikkoServer is extremely scalable as it takes advantage of the parallel nature of the Erlang language to the fullest. And as it works as a layer handling network traffic it is ideal for doing very sophisticated network optimizations. Together with our Stream product (a unique bandwidth optimization module), it allows individual optimized packets to be sent to each client depending on their line of sight, gameplay priority and client bandwidth responsiveness. This selective bandwidth limiting and traffic prioritization can be completely configured by the developer according to the specific needs for the game. When PikkoServer can’t access more CPU to handle load, it will start to drop outgoing low priority updates, and decrease the resend rate for outgoing traffic. The players might experience lag or unresponsiveness in this situation. A solution to this would be to slow down the game tick but that would be up to the game developer to implement. PikkoServer, however, is designed to recover very fast after such peak loads. Also note that this need only affect clients observing the most extreme action, other clients further away may keep on running as if nothing happened. More on this below.
The other server part in the architecture is the cell server(s). The set of cell servers is a system of distributed game servers with server side code for the game. They handle physics, database access, gameplay logic and more. The behavior of these servers under peak load is mainly up to the gameplay programmers. The work for doing inter-cell communication, like handovers of objects, distributed raycasting and area effects spanning several cells is not a major impact on the cell server load. The majority of the load lies in collision detection, local raycasting, database access and gameplay logic. That is why the behavior of these servers under peak load depends on the game design. However, we recommend handling the peak loads by adding more cell servers during the peak. These additional cell servers can be added and removed in a controlled manner by human operators or automatic monitoring tools, such as our Zone product.
Basic guidelines for building the cell servers are documented here.
Engine integration.
PikkoServer operates on the network protocol and works pretty much in the same fashion as a “man in the middle attack”. This architecture makes it easy to integrate into any engine with only minor changes to the game servers. Therefore, a PikkoServer integration is estimated to take not more than six months to do.
Licence.
If you are interested in licencing PikkoServer for your game, please fill out a licence inquiry at the Buy page.