Overview of how persistence (save / load) works

To understand some of the design decisions in the persistence logic, it is useful to review the objectives of the persistence logic:

1. Allow the game to be saved to a file, and then later reloaded from said file

2. Work with as many major implementations of Lua as possible (5.1, 5.2 and 5.3 from lua.org).

3. Have a platform independent file format (endianness, size of data types, Lua implementation, etc.)

4. Have a file format which is somewhat forward and backward compatible between revisions

The implication of the second objective is that the persistence library must use only the Lua 5.1 public API. There were two obvious paths to go down for implementing persistence:

1. Write code for persisting every part of the game to binary, and then de-persisting it again

2. Take a snapshot of the Lua universe, and write code for persisting every type of Lua value to binary, and then de-persisting it again

The second option was chosen. The reason being that there are only a small number of different Lua data types, whereas there are a much larger number of game parts. There are also new game parts being implemented as time goes on, whereas Lua data types are fixed in number. With this decision made,LuaPluto was one option which seemed to fit the bill, however further inspection revealed that Pluto did not satisfy our objectives:

• Pluto uses the private Lua internals, limiting it to lua.org's Lua 5.1 and other very similar implementations of Lua. As a result, Pluto is incompatible with LuaJIT 2, and is probably incompatible with with lua.org's 5.2.
• Pluto persists functions to Lua byte code, which kills platform independence (and introduces security concerns - malicious byte code can be dangerous).
• Pluto's writing of binary integers seems to have no regard for different endianness and different sizes of the underlying types on different platforms.

Despite these problems, the implementation of Pluto is nice, and so there are similarities between Pluto the bespoke persistence library implemented for CorsixTH.

persist.dump(value, permanent_objects)

Takes a single Lua value, and a table ofobject =>key relations, and returns a string representing value.

persist.load(data, permanent_objects)

Takes a string previously created bypersist.dump, and a table ofkey =>object relations, and returns the value originally passed topersist.dump .

For simple values (numbers, strings, etc.),persist.dump will just prepend a minimal header to the value. For complex values (tables, functions, userdata, etc.),persist.dump will persist the value, as well as (recursively) everything that the value references: keys and values and metatables for tables, upvalues and environments for functions, metatables and environments for userdata, etc.

persist.dofile(filename)

Equivalent to the standarddofile function, with the added effect of interpreting--[[persitable:name]] on functions. See thesection on persisting functions for further information.

In following the massive web of references from the original object,persist.dump will often encounter objects which should not, or cannot, be persisted. The solution to this is the permanent objects table. Whenpersist.dump stumbles upon an object which is present in the permanent objects table, it will persist the key associated with the object instead of persisting the object, and will not follow any references from the object.

In order to be de-persist the value,persist.load must be given an inverted permanent objects table, so that it can turn keys back into values. For every permanent key persisted bypersist.dump, there must be some value associated with that key in the table passed topersist.load, though the value does not need to be the same as the one originally encountered bypersist.dump. For example, if the application is closed between a savegame being taken and it being loaded, then all of the permanent objects will naturally be different, though they should represent the same things.

Note that while permanent objects are not recursively persisted, if some other persisted object contains a reference to something inside of the permanent object, then that part of the permanent object will get persisted.

As can be seen in the above table, the are complex requirements on functions and on userdata, which are explained in more detail below.

C functions must be permanent objects - they cannot be persisted any other way. Lua functions can be persisted as permanent objects, or as explicitly named persistable functions.

If a Lua function is instantiated exactly once (which is usually true for top-level functions in source files), then it can be persisted as a permanent object. To do this, it must be given a name which is unique across all other permanent objects, and unchanging in future revisions. Global functions and class methods are automatically marked as permanent objects, using their existing names. Local functions can be marked as permanent objects by defining them using syntax like the following:

local f; f = persistable"unique name"( function(x, y)  return x^2 + y^2end)

Where the original, non-persistable function was:

local function f(x, y)  return x^2 + y^2end

If a Lua function is instantiated a variable number of times (which is usually true for anonymous functions and local functions defined within other functions), then it can be persisted, as long as it conforms to certain rules. Firstly, the file containing the function must have been loaded bypersist.dofile (rather thanloadfile,require, etc.) - unless you are doing some major restructuring, then this point will not need to be considered. Secondly, the function must be given a name which is unique across all other non-permanent persistable functions. This name must be given as a comment immediately before thefunction keyword, in the format--[[persistable:unique name]]. As an implementation detail, there can only be one function with a persistable decorator per source file line. Lastly, the function cannot have anystrong upvalues.

Note: The termclosure means an instance of a function, along with a value for each upvalue of that function (and also along with an environment table).

An upvalue (a local variable used within a function which is not defined in that function) isstrong if it is written to by at least one closure, and is read from by any other closure(s). Otherwise, an upvalue isweak (that is, an upvalue isweak if it is only in one closure, or it is in multiple closures but not written to). The important point is thatweak upvalues can be separated with no observable differences in behaviour (if Lua 5.2 was set as a minimum requirement, then upvalues could be rejoined, allowingstrong upvalues to be persisted too).

In the following example,value is aweak upvalue, as it is read-from and written-to, but only by one closure:

function make_counter(starting_value)  local value = starting_value  return --[[persistable:counter_function]] function()    value = value + 1    return value  endend

In the following example,self is aweak upvalue, as it is only read-from, despite being in multiple closures (the fields ofself are written to, but the upvalue itself is not written to):

local self = {x = 0, y = 0}local --[[persistable:mover_x]] function moveX(x)  self.x = self.x + xendlocal --[[persistable:mover_y]] function moveY(y)  self.y = self.y + yend

In the following example,value is astrong upvalue, as it is read-from by one closure, and written-to by a different closure:

local valuelocal function set(...)  value = ...endlocal function get()  return valueend

When a non-permanent function is persisted, all of it's upvalues are persisted. One issue to watch out for is when the persisted function can call other local functions (excluding those defined within itself), as local functions which it might call are recorded as upvalues, and hence get persisted, and thus require marking as permanent or persistable.

Note: This section can be ignored if only writing Lua code, as new types of userdata can be made by new C code.

Userdata which are zero-size (like those created by the standard, though undocumented,newproxy function) can persisted, provided that their metatable (if present) does not contain a reference back to the userdata, and that their__depersist_size metafield (if present) is set to0.

Other userdatamust have a metatable in order to be persisted, and as previously, this metatable cannot contain a reference back to the userdata. The metatable must contain at least the following fields:

• __depersist_size - an integer to be passed tolua_newuserdata in order to re-create the userdata during depersistence. Given that the size of a userdata is often platform-dependant, userdata metatables are usually permanent objects (which also avoids references from the metatable back to the userdata). The presence of this field is what causes the requirement of the metatable not referencing the userdata, as that would create a circular reference before the userdata could be recreated.
• __persist - a C function which takes two arguments: the userdata being persisted, and aLuaPersistWriter userdata. If the userdata needs to persist extra data beyond it's metatable and environment table, then this function must swap the position of its arguments, cast theLuaPersistWriter userdata to a pointer, and then call methods on it.

For example:

  static int l_layers_persist(lua_State *L)    {        THLayers_t* pSelf = luaT_testuserdata<THLayers_t>(L);        lua_settop(L, 2);        lua_insert(L, 1);        LuaPersistWriter* pWriter = (LuaPersistWriter*)lua_touserdata(L, 1);        pWriter->writeByteStream(pSelf->aiLayerContents, 13);        return 0;    }

• __depersist - a C function which takes two arguments: the userdata being depersisted, and aLuaPersistReader userdata. If this function returnstrue, then it will be called again with just a single argument (the userdata being depersisted) after all other non-userdata values have been fully depersisted, which is useful if part of the depersistence process depends on other values which might only be partially depersisted at the time of the first call. This function should, in most cases, call a constructor on the userdata, as the memory contents of the userdata is undefined otherwise. If__persist wrote anything, then this function must swap the position of its arguments, cast theLuaPersistReader userdata to a pointer, and then call methods on it, being sure that in comparision to__persist, it calls the equivalent methods and in the order order.

For example:

 static int l_layers_depersist(lua_State *L)      {          THLayers_t* pSelf = luaT_testuserdata<THLayers_t>(L);          lua_settop(L, 2);          lua_insert(L, 1);          LuaPersistReader* pReader = (LuaPersistReader*)lua_touserdata(L, 1);          new (pSelf) THLayers_t; // Call default constructor          if(!pReader->readByteStream(pSelf->aiLayerContents, 13))                  return 0; // If a read fails, do not attempt any further reads          return 0;      }

The metatable may also contain:

• __pre_depersist - a C function which takes the userdata being depersisted as an argument. This function is called before__depersist, and more importantly, also before the userdata's environment table is depersisted. Most of the time this is not needed, but it can be very useful in cases where the "reference graph" has a cycle containing multiple userdata and the order of depersistence of said userdata is important. When such cycles exist, one of the userdata in the cycle must be depersisted first, and if that first userdata references a second userdata via its environment table, and then the second userdata callsLuaPersistReader::readStackObject() to get the first userdata, then the second userdata will be seeing the first in an uninitialised state, unless the first used__pre_depersist to initialise itself before its environment table was depersisted.

The most important thing is to ensure that any functions which might get referenced by a game object are persistable, as detailed in thesection on persisting functions.

If compatibility of savegames is desired between different versions of the source code (i.e. taking a savegame, then moving to a newer revision of the source, then loading said savegame), then there are other things to consider:

• Remember that any new fields on a table (i.e. ones which weren't used when the save was taken, but acquired a use in the new code) may benil after loading a savegame from a previous version. This must be gracefully handled at all places where the field is used. An afterLoad in the relevant file can be used to bring old savegames up to date with new fields.

  For example, if theHospital class didn't have areputation field, and a save was taken, then in some newer version of the code, thereputation field got initialised in theHospital constructor and used throughout the code, and then said save was loaded, then thereputation field would be nil in allHospital instances, despite there being no way of that happening normally.

• If new upvalues are added to a non-permanent persistable function (for example, the new code uses a local variable from a parent function which the old code didn't), then when an older savegame is loaded, accesses to these new upvalues will act like accesses to globals with the same names. Hence the introduction of new upvalues should be avoided if savegame compatibility is desired. Alternatively, if new upvalues need to be introduced, and savegame compatibility needs to be maintained, then a copy of the original persistable function can be made somewhere, and the new code given a new persistable name.

• Non-permanent persistable functions should not be deleted - move them to the end of the file or to a new file specially for deprecated persistable functions. They can be named _ (a Lua placeholder) and labelled in a comment with the current date, for future removal.

  If a non-permanent persistable function disappears from the source code, then any savegames which include an instance of that function will fail to load. Similarly, if a permanent function disappears or is renamed (included those which are implicity named by being global or class methods), then any savegames which include a reference to that function will fail to load, though references to implicitly named permanent functions are rare.

For major changes, the cost of maintaining compatibility with savegames of previous versions could be too large. For minor changes, trying to maintain compatibility is usually a worthwhile goal.

Another thing to bear in mind is that the memory addresses of Lua objects (i.e. the result of callingtostring on functions and tables) should not be used for anything important, as these addresses will change across a save/load.