What are good approaches for smoothly rolling out content updates?

Game Design
,
#1

Rolling out updates in Roblox can be a bit trickier than usual due to the nature of the server infrastructure: Each game has multiple servers running and not every server may be up-to-date. To get all players into new servers, the only options I know are available are:

  • Using ‘Shut Down All Servers’.
  • Using ‘Migrate To Latest Update’.
  • Waiting for old servers to naturally die out.
  • Teleporting all players into a newly created server instance and then back.

However, there are some concerns regarding the user experience with these options.

The first two options are not the best user experience because pressing those buttons suddenly interrupts gameplay for the majority of the player base. The third option can lead to an awkward situation where players want to experience the new update, but they are put into old servers. The last option is arguably the only option that does not harm the user experience because you can notify players before the teleport happens, but implementing such a system can be complex or expensive because the developer has to implement some sort of system to broadcast this action to all out-of-date server instances, and on top of that teleports may fail.

Additionally, with options two and three there is the concern of player data being corrupted. If a new content update adds new items that have to be stored in the player’s data, their save files have to be modified or upgraded. When old and new servers are running simultaneously there is a chance (albeit very slim) that players with upgraded save files join older servers. And so those servers have to be prepared to deal with such save files.

Given the concerns I mentioned above, I would like to know how other games tackle these problems. What approaches does your game take to keep player data safe when a new update hits? What do you do to smoothly transition players from old servers to new servers without harming their experience? I’d like to hear from you all!

4 Likes
#2

My favorite solution that I haven’t gotten to try out on any kind of large scale (I can’t justify spending a ton of ad money to get a hundred players into a game or something) is a combination of options three and four.

Essentially, the servers are all occasionally broadcasting their PlaceVersion and JobId through MessagingService, and saving all incoming signals, sorted by PlaceVersion. When a server receives a PlaceVersion higher than its own, it phases itself out and forwards players that join the server to one of the other servers with the latest PlaceVersion. Existing players can remain in the same server but are given the option to jump to a new one. Eventually all the players will leave the old server and it will close, with nobody having their play session interrupted.

That’s the theory, anyway. Like I said, I can’t exactly try it out very easily.

Making sure save files are cross-compatible with other versions is definitely possible, it just requires foresight (saving absolutely everything in the player’s data folder, setting up inventory systems to ignore unsupported objects [but not delete them or error out], etc.)

1 Like
#3

Personally, I have each server handle its own updating.

Say you have a basic game loop:
Intermission -> Round -> Results -> Repeat

Before the intermission executes, I make a call to GetProductInfo to get the update timestamp of the place. If it doesn’t match the cached version from when the server started, then I can logically conclude that the game has been updated. From there, I teleport players to a new server.

local CurrentVersion = game:GetService("MarketplaceService"):GetProductInfo(PLACE_ID).Updated

local function Update()
   -- Move players to new server here
end

local function Round()
   -- Do round stuff here
end

local function Intermission()
   local NewVersion = game:GetService("MarketplaceService"):GetProductInfo(PLACE_ID).Updated

   if NewVersion ~= CurrentVersion then -- Game has been updated.
      Update()
   end

   wait(30)
   Round()
end

This allows servers to finish up their rounds before moving players to newer servers, making updating relatively smooth for players.

Your data system should have caching so that data cannot be loaded / written to until it is unloaded and saved in other servers. A “session lock”, if you will. UpdateAsync() should be useful for accomplishing this.

3 Likes
#4

(1) Shut Down All Servers

I have personally used this without any issues for 3 different games, for a combined total of 200+ shutdowns across around 1.7bn play sessions. It seems very reliable and I haven’t had issues with players being able to join old servers.

As well as the gameplay interruption problem, the “Reconnect” button is not as effective here because it is not guaranteed that there will be new servers immediately ready for joining at the moment the old server shuts down.

(2) Migrate To Latest Update

I haven’t personally used this one more than a few times yet because (a) old habits, and (b) probably-irrational worries about it not working. However, this should, in theory, solve the reconnection speed issue that Shut Down All Servers has because new servers are opened in advance of shutdowns.

You mentioned concerns of player data corruption in relation to this option. This shouldn’t be an issue with Migrate To Latest Update because old servers are marked as unjoinable - players simply will not be able to join them once you trigger a migration (source).

Something you didn’t mention is that this option has a targeted 6-minute window for shutdowns. This implies that Shut Down All Servers probably tries to do it more quickly. Perhaps a faster, but slightly more arduous reconnection process (Shutdown All Servers), is preferable to receiving the update 6 minutes after other players (Migrate).

(3) Let Old Servers Die

I think a good general principle is that this option should be avoided unless:

  1. The update is very minor, for example a small patch, correcting a typo, etc.
  2. The game can robustly handle players joining older servers with ‘new’ content, e.g. a new pet that the old server doesn’t recognise.

It might be non-trivial to support arbitrary content being added to the game and being present in old servers that are not aware of it. Even if you do manage that, I’d generally advise against using this option for content / non-minor updates, because these kind of updates are something that players are keen on seeing as soon as they are available. Having to hop around until you find a new server with the update is a terrible experience.

(4) Soft Shutdown via TeleportService

I have never used this method - although I am aware that there is a popular piece of code that supposedly accomplishes it.

The use cases for this should be entirely covered by Migrate To Latest Update. As well as that, the method may be outdated and I haven’t seen any data on whether it works at scale. I would also be interested to hear from others here specifically about using this option at scale and how they handle TeleportService outages.

A note on -Service solutions

As a general principle, in relation to handling shutdowns etc I am cautious about using:

  • DataStoreService (e.g. some kind of versioning for data / session lock),
  • MessagingService (e.g. informing other servers of my server status/version),
  • TeleportService (e.g. soft shutdown),
  • MarketplaceService (e.g. check latest version string, compare to my own).

This is because these services do experience (sometimes severe) outages and I am not keen on adding large amounts of extra complexity in order to handle the various potential things that can go wrong. DataStoreService is pretty much a necessary evil, but I’d rather avoid extra calls where possible.

It’s also especially difficult to handle failures from these APIs because there isn’t a consistent and reliable way to differentiate failures. You have to use pcall and then attempt to parse the error message to get any meaningful information - but there are no guarantees about the content of the error messages.

The closest you can get to specific-case handling of errors with these services is with the TeleportInitFailed event which supplies a TeleportResult Enum, but not all teleport-related failures will fire this event (failures can occur after Init stage of teleports).

3 Likes
#5

In one of my games we use a simple soft shutdown system based off of that code. Our game is relatively small, with usually around 150-200 concurrent players, so I can’t say much about how it scales.

Perhaps Migrate To Latest Update (MTLU) is a better solution, but I am doubtful of how well it retains players, which is why we’re using this custom soft-shutdown system instead. MTLU requires user input and decision making to proceed, and also from my feelings when using it just interrupts the experience in the “wrong way” (I can’t really put the feeling into words – maybe its just cause the popup has no button hover effects, every time I get one of those it feels like the game is broken even though its not lol, or maybe its because it gets rid of all momentum in a way that having a custom load/teleport screen + two teleports in a row doesn’t). I would really like it if Roblox implemented a better system that runs completely on its own without waiting for user input.

The code

This is the code in the main game, uhh it doesn’t have anything to handle failed teleports, probably should retry then kick in such cases.

-- runs with game:BindToClose()
local teleportService = game:GetService("TeleportService")
local place = 4925641595
local code = teleportService:ReserveServer(place)

game:GetService("Players").PlayerAdded:Connect(function(player)
	script.SoftShutdown:Clone().Parent = player:WaitForChild("PlayerGui")
	teleportService:TeleportToPrivateServer(place, code, {player})
end)

local players = game:GetService("Players"):GetPlayers()
for i = 1, #players do
	script.SoftShutdown:Clone().Parent = players[i]:FindFirstChild("PlayerGui")
end
teleportService:TeleportToPrivateServer(place, code, players)

Then next is the code in the middleman place to teleport players back to the main game. This code could also perhaps be improved to teleport players in the groups they arrive in so that they all stay in the same server.

game.Players.PlayerAdded:Connect(function(player)
	wait(5)
	while true do
		local s,m = pcall(function()
			game:GetService("TeleportService"):Teleport(4553413576, player)
		end)
		if s then
			break
		else
			game.ReplicatedStorage.SendError:FireClient(player, m)
		end
		wait(1)
	end
end)
1 Like
#6

What I do for Vesteria and what I think is the best solution for multi-place games is to simply hook into BindToClose, save everyone’s data and teleport them back to the main menu with a custom teleport screen that explains what’s going on. We get to shut down all servers and players get back into the action fairly quickly without requiring the user agency of actually re-joining the game. The downside of this obviously is that people might not end up in the same servers as before, but I’ve found that Roblox matchmaking is actually decently consistent about matching people back together.

7 Likes