I am working on a building game that saves instances to DataStores, and I will likely need to update the serialization format in the future. What is a safe way to migrate a player’s saved data to an updated format? And how can I write comprehensive migration tests to be extremely confident that there won’t be data loss?
1 Like
I’ve included a ‘version’ number in all of my save files which I plan to use if I need to do any migrations in the future.
I already have a flow set up that compared the saved file with the template file, and merges them together in case the template file has new additions. This also checks the ‘version’ of the save data. If the version is lower than the version of the template file, I’d route the flow through a ModuleScript programmed to take the data from the ‘old’ format to the new one. These can be written to ‘up’ the version number of a save, one migration at a time.
With any sort of data manipulation flow like this, you’d want to store a copy of the ‘old’ table before migration starts. Then, at the end of the process, you can compare the data and/or run more comprehensive tests written to precisely compare your serialized data. The result of these comparisons should determine whether;
- Data has merged/migrated successfully; mark the savefile as “safe” so it can be written to DataStore if player leaves.
- Merge/migration failed; don’t mark as save and provide error message to the player. If they leave or are kicked from the game, data is not marked as safe and thus discarded.
(Note; you need separate “safe” flag for “is data loaded?” and "can data currently be altered?. So that if the file cannot currently be altered when the player leaves, it will still be saved.)
2 Likes
If the player’s data migration fails, are you suggesting I still allow the player to play anyway but not save their data?
What’s the difference between loaded data and alterable data?
Hey, thanks for the question!
Managing player data at-scale can often be a logistical nightmare, even more so if the fundamental structure of that data needs to be updated - or, even worse, if changes need to conditionally be made for specific players!
At a glance, one might assume that the solution to these problems is to simply take the game down for maintenance, then run a bulk-operation on all player data for a few hours, then bring the game back online once the operation completes. This ensures that all player data is in a format that the game knows how to read, and that all elements of the data contains proper values. However, this comes at a huge time cost, especially if the data is hosted on a 3rd party database (as is the case in Roblox datastores), resulting in rate limiting. This means your game would be down for hours, possibly even days for what would otherwise be a small patch update. This problem worsens ten-fold if something goes wrong during this process! This is obviously not ideal!
Another common approach to this problem that I’ve seen is to run a diff between the two data structures, and merge them into a 3rd, “updated” data structure. However, this is an extremely complicated process and it is very easy to get corrupt data by doing this! If there’s a bug in the diffing algorithm, or there’s some special case, data will be lost! This is an extremely generic and thus rigid process and should not be used. It will be a ticking time-bomb that will wreak havoc on your project and playerbase.
Ok, two solutions that have huge costs. What can we do here that gets the job done, while having almost no time & computation cost?
Introducing… drumroll… contextual format migration (similar to what @Wsly mentioned)! Unlike the other two solutions, this solution doesn’t rely on data being up-to-date, and it’s flexible & robust enough to support nearly any usecase, regardless of project! It also has a low time-cost and low computation cost. That’s great! How do we do this?
Essentially, you want to assign an incremental version to each format you go through during the development of your game - format version 1, format 2, and so on. Here’s an example of some data format changes from a potential RPG game:
Version 1
{
_FormatVersion = 1,
Coins = 0,
XP = 0
}
Version 2
{
_FormatVersion = 2,
Currency = {
Coins = 0,
Gems = 0
},
Statistics = {
EnemiesDefeated = 0,
XP = 0
}
}
Let’s take a look at what happened here. When the game was first made, it only had 2 fields in the player’s data : Coins and XP. Makes sense! However, as the game became more sophisticated and had more features added to it, we wanted to add more fields to the save data! Coins was moved into a Currency table, and a new Gems currency was added! We can also see here that we’re tracking some basic player statistics such as EnemiesDefeated. XP was also grouped together with that! All of this was done in a single update to production, so it is now the second data format.
That’s great and all, but how do we convert player data from version 1 to version 2 when we update the game?
The method here is actually quite simple! You simply create a FormatConversions module. Said module contains functions that convert from format version x to format version y. Your data system takes the old data, and runs it through the function that bumps the data to the next format. Version 1 to 2, 2 to 3, and so on until the data is in the latest format that the game knows how to read from & write to. Here’s an example of what that module might look like using the same example as above:
local FormatConversions = {
["2 -> 3"] = function(Data)
-- Add "deaths" statistic
Data.Statistics.Deaths = 0
return Data
end,
["1 -> 2"] = function(Data)
-- Update currency
Data.Currency = {
Coins = Data.Coins,
Gems = 0
}
Data.Coins = nil
-- Update statistics
Data.Statistics = {
EnemiesDefeated = 0,
XP = Data.XP
}
Data.XP = nil
return Data
end
}
return FormatConversions
As you can see, each function takes data that’s in a format it can work with, then bumps it up to the next version. Your round system would chain-call these to iteratively update the player’s data until it was up to date! The function 1 -> 2 is called, then the data from that is piped into the function 2 -> 3, and so on. With this knowledge, you may ask “where do we store the current format for NEW players?”… since, after all, new data has to be created for new players that the game can work with! You simply store this information in a CurrentDataFormat module, like so:
return {
_FormatVersion = 3,
Currency = {
Coins = 0,
Gems = 0
},
Statistics = {
EnemiesDefeated = 0,
Deaths = 0,
XP = 0
}
}
When a player joins, your data system checks if any data exists for the player or not (they’re returning or new to the game). If the player is new, their save data is simply created from the current data format! They get all of the values in the format table. If it’s a returning player, their data is simply taken from the datastore and chained through the conversion functions!
By doing this, you ensure that all players have the exact same data format, regardless of how long it has been since they’ve played the game or how often the structure has been changed. In the event of a failure (something goes wrong during format conversion, data fails to load, etc), you can simply give the player “default” data (the table from the CurrentDataFormat module), and mark them as having unsavable data. This way, the player can continue to play your game despite having temporary session data! When they leave, their data is simply discarded. Worth noting, it’s a good idea to disable devproduct purchases for the player while they are in this state, so that player robux are not wasted.
This is just the surface of what you can do with this approach! Since the conversion functions are… well, functions, you can run any code that has the context of the conversion being done!
To further elaborate, here are some scenarios developers can often run into with a live game, and the solutions to them using this system.
Scenario : You have OCD
I know, I know. It happens. You named a field something and the name is bugging you. Luckily, because of this system, we aren’t stuck with that field name for all of eternity!
Old format:
return {
_FormatVersion = 1,
Currency = {
myCoinz = 0, -- This name triggers me
Gems = 0
}
}
Conversion function to fix this:
return {
["1 -> 2"] = function(Data)
Data.Currency.Coins = Data.Currency.myCoinz
Data.Currency.myCoinz = nil
end
}
End result:
{
_FormatVersion = 2,
Currency = {
Coins = 0, -- Much better!
Gems = 0
}
}
Scenario : A typo made it to production
It happens. A typo made it through code review, testing, and onto production. And unfortunately for you, all players in the game that joined a server to play the new update now have 500000 additional coins. Oops! Luckily, this is easy to fix!
Data format in the bugged update:
return {
_FormatVersion = 5,
Currency = {
Coins = 500000 -- Oops! A dev left this debug value here by mistake :(
Gems = 2
}
}
Conversion function in the bugged update:
return {
["4 -> 5"] = function(Data)
Data.Currency.Gems = 2 -- We added gems to the game and want all players to start out with some!
Data.Currency.Coins = Data.Currency.Coins + 500000 -- Oops! A dev was testing something in a test environment and this was never removed. :(
return Data
end
}
Player data with the problem, since they joined to play the update:
{
_FormatVersion = 5
Currency = {
Coins = 507180, -- The player's old coins were added to the 500,000 typo'd coins!
Gems = 2
}
}
Player data without the problem, since they haven’t joined in a while:
{
_FormatVersion = 3
Currency = {
Coins = 14207 -- No problems here! They've earned all of this!
}
}
Because we store all of the previous format conversion functions, we can run contextual logic to subtract 500,000 coins from only the players that were mistakenly given the extra coins! You would want to push an update that migrates player data via this function:
return {
["5 -> 6"] = function(Data)
Data.Currency.Coins = Data.Currency.Coins - 500000 -- Update 4 -> 5 mistakenly adds 500000 coins. This undoes that.
return Data
end
}
Scenario : Players were accidentally given an item they shouldn't have
Let’s say your game has a dev-only sword that one-shots everything in the game. It sure would be a shame if players were to get it in their inventor-- oh no! Some players got the item in their inventory due to a vulnerable RemoteFunction that they’ve exploited, and now they’re wreaking havoc in pvp! Luckily, this is easy to fix after you’ve patched the vulnerability, with one simple migration function:
return {
["14 -> 15"] = function(Data)
if Data.OwnedItems["DevSword"] ~= nil then
-- The player was literally exploiting and got the sword because of it
Data.OwnedItems["DevSword"] = nil
Data.Moderation.Banned = true
Data.Moderation.BanReason = "Owned the dev sword, which was only obtainable by manually invoking a remotefunction with a specific parameter value that the game does not do on its own."
-- You could fire an event or something here to ban the player and kick them!
end
return Data
end
}
Overall, contextual format migration is robust, fast, and efficient. It fits the majority of usecases a game will face, and it accounts for all players, regardless of when they last played!
6 Likes
Thank you for the thorough response! I really like the simplicity of this; mapping from one format to another removes the uncertainty of merging formats on-the-fly (for as nice as that would be 
). And the scenarios you’ve provided are very illuminating because they’re (unfortunately) realistic.
This clears up the question I had for @Wsly 
Do you recommend testing for a failed migration after each individual format conversion or at the very end?
1 Like
Ideally you want to abort the load process as soon as it errors. So, per-function / each individual format. If update 2 -> 3 works, but 3 -> 4 encounters an issue, you abort and give default data that is specific to that playsession. Doing this also allows you to hook in some analytics, so you can look at your analytics dashboard and say “Huh, conversion function 3 → 4 seems to be really broken today! Let me take a look”.
1 Like
Also! One more caveat : If a player joins a new server with an update, then they leave and join an old server, your data system should account for this! It should say “hey, I don’t know how to read this data format since it’s in a version above my own”, and abort - once again, giving player default session-only data.
This type of thing can happen while you’re deploying an update to production, but it’s slow to roll-out for some reason.
1 Like
Today’s announcement of Datastore APIs for open cloud suggests that one use case is data migration.
Data migration script : script that reads from current datastores, changes the schema, and writes to a new datastore.
What would be a good way of doing this? An immediate problem I see is trying to migrate a player’s data while they’re playing and writing to their datastore key from in-game.
This generally might not be a good idea! I outlined why in my reply above.
1 Like
Generally speaking, yes! For some games (i.e. sandbox building) it wouldn’t make sense to ‘start fresh’ but if you have a round-based game, players likely wouldn’t mind as much to not have their data.
Like Jayden mentioned, just give them a fresh copy for this session and discard it afterwards. Do make sure to notify the player though. I think it’s worth disabling Robux purchases as well, and displaying a banner in your shop.
Depending on your implementation, you want to track numerous things. One is “Does this player have a data table I can write to?” which is important if you want them to use a dummy table for this session. Game systems would still write and read from this table as usual, and this boolean can be used to check if the table is currently alterable.
A secondary boolean is used to store if the table is succesfully loaded data or a dummy table. That boolean is checked when playes leave the game or the game attempts to save the table mid-session.
1 Like