Multiplay currently operates two different scaling models, Allocations and Reservations.
This document pertains to the Allocation system, detailing its basic usage and operation of the Allocation scaling system and its subtypes.
The Multiplay Allocations API is asynchronous, so that in the unlikely event that Allocations take a long time to complete they do not consume resources. The Multiplay Allocations API complies with the RESTful Architectural constraints and is idempotent.
What is an allocation?
A simple definition of an allocation would be: a proactive request to obtain a Game Server Instance from Multiplay which can then be used by players.
In more technical terms, an Allocation is an API request for a Game Server Instance. Upon receipt of the API request, our orchestration system finds an unused Game Server Instance and flags it as used, a second API request is required to return a Game Servers information.
What is a deallocation?
A De-Allocation is the removal of an Allocation from our orchestration platform, this is completed via a single API call. A De-Allocation, does not directly shutdown a Game Server Instance, however if the capacity is no longer required and the instance was a Cloud instance it can then be shutdown and removed at a later day. Once deallocated a Game Server Instance can be reused by a subsequent allocation request..
What is a Fleet?
Multiplay uses Fleets to encapsulate a set of regions, locations, providers and profiles. This allows Multiplay to manage multiple environments with unique settings for each region, while maintaining a core template.
- A fleet is a container for settings and a collection of regions, in turn each region can have many locations.
- A fleet can have custom regions with only one location, allowing for a very granular approach to player base regional splits.
- A fleet normally operates one game but can operate two or more.
- A fleet must have assigned profiles, not all profiles are fleet enabled.
- A fleet generally contains a split of Bare Metal and Cloud based VMs.
In the event there is no Cloud provider available in a defined region, it is possible that a Fleet can contain only Bare Metal Machines, it is also possible for a Fleet to only contain Cloud VMs as well if required. How do we scale based on Allocations? Allocation scaling choices are based on multiple criteria. Our orchestration platform makes decisions based on cost, velocity, locations, spin up times, instance types, wider internet issues and location reliability.
When utilising our API there is no delineation between Bare Metal and a Cloud VM when it comes to an Allocation, our orchestration platform makes intelligent decisions on which capacity to return based on the criteria outlined above. Multiplay ensures type, density, fragmentation and the availability are all taken care of when we return a successful allocation.
Matchmaking with Allocations
Enabling a matchmaker with allocations is a relatively simple process, although understanding player flow i.e. how you expect players to be placed into a match from matchmaker to game server is key, while also considering regional placement, profile request and UUID tracking.
To this end there are a few questions you need to worry about when selecting or designing a matchmaker:
- How are players grouped together, either before or after Allocation?
- When is an Allocation called?
- How is the Game Server connection information passed to the Clients?
- How does the Game Server end the session?
- How are custom requirements configured, such as map/mode?
- Our profile system handles the above but the matchmaker needs to request the corresponding ProfileID.
- How will the matchmaker track an Allocation and the players session by the UUID?
- The UUID can be accessed on disk per gameserver when a allocation can be completed.
- How do you know is a specific allocated gameserver is ready?
- If an Allocation is active for too long, an alert will be raised with the 24x7 Multiplay Support Team.
- When is a De-Allocation processed?
The example below outlines a player flow where they the players are matchmaking into a game server instance supporting a lobby. Players are grouped together on the game server instance before play begins.
- An Allocation is called when a new session/lobby needs to be created.
- Players are grouped together on a game server instance in a lobby.
- When the matchmaking flow begins, the allocated game server information is passed to the players
- The matchmaker decides whether to place them in an existing session or create a new session based on the game mode and available sessions requested by the player. Alternatively, if the game supports in session joining, they could be funnelled to an in-progress session.
- The game server ends the session and reports to the matchmaker when all players have left, the server is not responding or the match has ended gracefully.
- When a Game Server ends gracefully the matchmaker is informed the session is over and a De-Allocation is triggered.
Additional scenarios to consider:
Multiple Game Modes
- Custom requirements are configured solely off of a ProfileID, in this example we have 3 profiles, Capture the Flag, Deathmatch and Team Deathmatch. The map is decided in the lobby and players start the session either through readying up or a timeout.
- The matchmaker will request an Allocation with the required ProfileID.
- Multiplay will provide a server with the necessary configuration based on the requested ProfileID, returning game server information to the matchmaker.
- The matchmaker will then forward on that game server information to the players.
- The UUID referencing this session and game server will be stored in the matchmaker database, recorded as an active session, if this changes the matchmaker will issue a De-Allocate.
- The Game Server talks to the matchmaker when a session finishes, changing the state.
- If a Game Server instance does not return a heartbeat in 120 seconds the matchmaker will De-Allocate the assigned UUID.
- How is an Allocation handled which are either orphaned or have yet to receive players?
- If players timeout or never join the session.
- If the Allocation is present but not required.
- If a Game Server Instance crashes how will the game server instance De-Allocate?
- In the above example, this is handled with the heartbeat timeout.
Now we have the basic information on how the system will work as well as some error handling we can start to map the actual functions to our API calls:
Allocation flow Diagram
Multiple Allocations calls
In a production matchmaking environment you should not issue a allocations call for each UUID. The best approach to this would be have a thread constantly looping all pending UUIDs as they can be batched together in one API call.
Allocations and Zero Downtime patching
Zero downtime with allocations is very simple to do. Your matchmaker needs to support multiple profile ID’s, a profile ID contains all configuration data and version information. So when a patch is released and updated your matchmaker can start allocating the new profile ID for players with the new matching client version. This method can support multiple versions over many profiles.
If the profiles being used for allocations are running version A, you would update and install version B, confirm the roll out of the patch, then start allocating profiles using version B. This would gracefully move capacity without dropping players.
Why the game server process should not exit itself after a session
If you decide to self terminate the gameserver at the end of the match (which can be called a destructive clear down), it will prevent optimal use of resources as the process initialisation is typically more expensive than cleanup. For example rather than restart the gameserver, return it to a lobby state without self termination.
Extra options and features
- Auto Restarts can be configured on a per profile basis, so as to not disrupt players. Auto restarts will attempt trigger when there are zero players using query and no allocation and can make use of graceful restarts if supported by the game.
- Multiplay variables can be passed through to the gameserver, such as city, ping_site, serverid, machineid, port and ip. These can be used to report back information and unique ID’s without the API if the gameserver is designed to do so.
- Graphing and data management can be done based off the implementation of query to give very accurate data. This also improves our ability to monitor the gameservers on an individual level.
- Logging is all centralised. It can be downloaded or viewed on the website front end.
- OS or machine level maintenance is completed without disrupting a fleet or a region.
API References (in order of use):
- Game Servers#ServerAllocate
- Game Servers#ServerAllocations
- Game Servers#ServerDeallocate
Useful API calls for gameserver validation:
- Game Servers#ServerStatus
- Game Servers#ServerResources
- Game Servers#ServerStatus
What is a UUID
You can check the Wikipedia entry here: https://en.wikipedia.org/wiki/Universally_unique_identifier