Multiplay currently operates two different scaling models, Allocations and Reservations.
This document pertains to the Reservations system, detailing its basic usage and operation of the Reservation system and it’s subtypes and variations.
The Reservation system like the Allocation system using the same Multiplay API which is asynchronous, so that in the unlikely event that Reservations a long time to complete they do not consume resources. The Multiplay Allocations API complies with the RESTful Architectural constraints and is idempotent.
This guide will make many references to the Allocation documentation here
What is a Reservation?
Simply a reservation is a allocation which is retroactive, a gameserver that is already in use or soon to be in use by players would be reserved with the API.
The Reservation call is a API call that is decentralised, it is performed by the gameserver itself when it has hosted players or has started a session, there is only a confirmation of success from the API and no second call.
What is a Un-reserve?
A unreservation or unreserve is a API call to tell Multiplay that a gameserver has become idle and is ready to accept players again. Multiplay takes no action directly based on that information, but will adjust the fleet accordingly.
What is a Fleet?
Multiplay uses Fleets to encapsulate a set of regions, locations, providers and profiles. This allows Multiplay to manage multiple environments with unique settings for each region, while maintaining a core template.
- A fleet is a container for settings and a collection of regions, in turn each region can have many locations.
- A fleet can have custom regions with only one location, allowing for a very granular approach to player base regional splits.
- A fleet normally operates one game but can operate two or more.
- A fleet must have assigned profiles, not all profiles are fleet enabled.
- A fleet generally contains a split of Bare Metal and Cloud based VMs.
- In the event there is no Cloud provider available in a defined region, it is possible that a Fleet can contain only Bare Metal Machines, it is also possible for a Fleet to only contain Cloud VMs as well if required.
How do we scale based on Reservations?
Reservations unlike allocations are based on established data, for example the gameserver ID is used in the request, so Multiplay already knows the needed data to make capacity adjustments. There is a trade offs for reservations, because we are dealing with retroactive calls, time to scale is not as reactive as the allocation method, Typically this is some what countered by the pool gameservers being larger than allocations, this also has a direct cost implication. When a Reservation comes in the gameserver is marked as in use, this could mean the region and location that the gameserver is in will scale in cloud as a result. Similar to the allocations we will base the scaling logic off the most cost effective requirements, however as we are not directing the flow of matches with the reservation system as a result of this the reservation is prone to fragmentation, which i will expand on later in this guide.
Matchmaking with session based games and Reservations
Reservations works when there is little to no presence of a centralised service such as a matchmaker, this scenario is normally the case when server browser solutions are used, such as the Steam, or a light weight service. Reservations are however not limited to these models but we strongly recommend using the Allocation method if you have a central matchmaker service. This is because a although the scale up of capacity is as efficient as the allocations system, scaling down we see fragmentation. This is because Multiplay is not controlling the game instance usage order, the player is and also the player is in control of the session length. With the allocation model matches are refreshed at a faster rate and the efficiency increases as a result.
To this end the following needs to be considered when creating a game to work with reservations:
- Is a player allowed to play alone in a game server?
- If the game is server browser base this maybe a serious consideration, as multiple solo players would likely trigger scaling.
- What cases a gameserver as “in use”
- And as a result Multiplay could decide to take action, so if you do not have a session based game with a fixed number of players to start, “in use” might be a full gameserver
- How will you support multiple game modes and configurations?
- Reservations can support multiple profile and settings but they cannot be dynamically swapped, because of the retroactive behaviour as a result we have seen “first player configuration” meaning first player to join configures the gameserver, either automatically or dynamically.
- How will a gameserver clear down a incomplete match/session.
- How often does the gameserver heartbeat to the server list.
Depending on the model and the design of the game we also have to acknowledge that players could be matched into this model by a matchmaker, although this is a un-efficient scaling method it is still viable for a centralised service. To that end we also need to ensure that the matchmaker can group players together a pick a gameserver from the pool without interacting with Multiplay.
Things to consider using reservations
- You have less control over the gameserver’s configuration and settings. The gameserver itself needs to be more robust to handle custom modes and settings and change on the fly.
- The gameserver will need to potentially have to change settings with players present on the gameserver if there is no centralised service, this could disrupt players
- Gameserver responible for scaling, decentrailised authority.
- Gameserver must report into a serverlist or matchmaker to direct players and heatbeat to the same service
- Serverlists may have the following impact on your game
- Fragmentation of players
- May not suit the game style or expected player experience
- How to recover from gameserver crashes
- Reservation removes some of the cost opimizations from Multiplay, this lowers effciency.
- In some cases the choice of the player to pick a gameserver in the cloud can directly influence cost, Multiplay is unable to direct players away from VM’s
- Race conditions when players are joining as Multiplay cannot redirect players from shutting down Gameservers
- Need a bigger idle pool than allocations to support the re-active approach of scaling
- Is your serverlist going to be gamemode and region aware? if so how is that information returned to the server list from the matchmaker?
Unreserving a gameserver is telling Multiplay that a gameserver is not in use. This needs to be done at two points, the first at the end of the player flow when the gameserver is no longer in use, but also during startup of the gameserver, this is because we need to unreserve the gameserver if the gameserver has crashed.
WARNING: If a gameserver crashes Multiplay will restart the process using the same profile using criteria such as crash amount and frequency to determine if it should in fact be restarted. Multiplay will never take action to unreserve a gameserver, the gameserver is authoritative so action cannot be taken. This is why the gameserver must unreserve on startup to reset its state this assists with crash recovery!
Reservation flow Diagram
Reservations and Zero Downtime patching
Zero downtime patching with reservations operates on a single API call, the call specificies the profile to switch to, that has the new version associated to it. The profiles are then switched by Multiplay either when there are no players or ultimately at a fixed point defined when calling the API.
Reservations fleets cannot run different version and as such the API requires a force switch time, which is a force switch of the profile.
Extra options and features
- Auto Restarts can be configured on a per profile basis, so as to not disrupt players. Auto restarts will attempt trigger when there are zero players using query and no allocation and can make use of graceful restarts if supported by the game.
- Multiplay variables can be passed through to the gameserver, such as city, ping_site, serverid, machineid, port and ip. These can be used to report back information and unique ID’s without the API if the gameserver is designed to do so.
- Graphing and data management can be done based off the implementation of query to give very accurate data. This also improves our ability to monitor the gameservers on an individual level.
- Logging is all centralised. It can be downloaded or viewed on the website front end.
Requirements for the game server
The Game server must ingest the API end point, API keys and parameters via either a config or the CLI (recommended). Multiplay will pass in the following data, $$variable$$ being a platform level variable.
Command Line params:
-apialloc=https://api.api/cfp/v1/machine/$$machineid$$/reserve_server?serverid=$$serverid$$ -apidealloc=https://api.api/cfp/v1/machine/$$machineid$$/unreserve_server?serverid=$$serverid$$ -apiaccess=<key> -apisecret=<key>
It is also required that to facilitate effective crash recovery, the game server does a blind Unreserve when it starts up.
API References (in order of use):
- Game Servers#Reserve
- Game Servers#Unreserve
Useful API calls for gameserver validation:
- Game Servers#ServerStatus
- Game Servers#ServerResources
- Game Servers#ServerStatus