Most programs depend on external assets, being images, 3D models, data files etc. Past some point, it becomes a burden to manage all the loadings, allocation and deallocation if some sort of management system is not used.
The role of a resource manager is precisely to ease up the management of assets. The very least function they offer is to ensure that resources are only loaded once when they need to be accessed by different objects. They also typically offer non-blocking loading for systems with a user interface, fallback mechanisms when resources do not exist anymore, various loading source (from files, network, pak files etc.).
I was first exposed to resource management systems when I was working on games through the reading of “The Beauty of Weak References and Null Objects” by N. Liopis (published in Game Programming Gems 4). Their usage goes however beyond the simple scope of game programming. A more detailed introduction can be found in “Game Engine Architecture” by J. Gregory for interested readers although it lacks several features that I have found essential with time. More specifically, I have found the concept of a null resource very handy as you can quickly spot, for instance, missing textures in 3D models by replacing them with something like a saturated red image. Also, I have found that the ability to automatically reload resources when their associated files are changed very handy as you don’t have to restart the program everytime you make modification to things like scripts.
The code associated to this post is released under a LGPLv3 license and can be found [∞] here. The version used in this post is v1.0.
The code comes with two programs: a demo program and a test program. The purpose of the demo program is to provide a user-interactive scenario whereas the purpose of the test program is to assess that the library works correctly on your specific machine. The library was validated on Win10 with Visual Studio 2022, Debian 10 running on WSL and on Debian 11 running on my vmware machine. A few bugs were noticed in some very specific circumstances with WSL and are reported in the README file.
The test program is limited to behavioural, high level requirements, checks. When the tests passes on your machine, it therefore commit to a specific behaviour of the library. Here is a list of these behavioural commitments:
- Resources are constructed from a string (file, url etc.) and are accessed as pointers. Currently supported format are raw strings or file names.
- Resources are only loaded once when used multiple time unless all resources access are destroyed, in which case the resource is loaded again.
- Resources can be loaded synchronously or asynchronously. Asynchronous resources are non-blocking and can be waited on and checked if they are ready. Asynchronous resources are null until they have been loaded.
- Loading an invalid (non-existing) synchronous resource will trigger an exception. Loading an invalid asynchronous resource will not trigger an exception.
- Null resources can be registered to the resources manager. When an asynchronous resource is accessed, it is defaulted to the null resources if the actual resource is missing or invalid. If no null resources has been registered, an exception is thrown.
- Resources can be forced to reload if necessary. Resources created from files are automatically reloaded when the source file has been changed. This also applies to removing a resource file or creating a file for a previously invalid resource.
- Resource objects can be saved to file using the standard storage system. This applies to both resources directly or to object derived/inherited of the resource classes.
The demo program is fairly easy to run. Just start a command line and type the name of the program followed by the name of the resource file to load, for instance test.txt. The program will then print the value of the file until you press CTRL+C. If the file does not exist or if the file contains non-printable characters, it will default to a “lorem ipsum dolor” message. When the file is modified, its content is automatically reloaded. The text will always be trimmed to 80 characters maximum (“…” are appended if the text is longer).
The active part of the demo program illustrates pretty well how to use the resource system:
// set null resource
resources::set_null_resource(std::make_shared<const snippet>("Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."));
// create resource
resources::async_proxy<snippet> res(std::string("file://") + std::string(argv[1]));
// print every 100 ms as long as we don't quit
g_exit.do_until([&](void)
{
auto message = res->get();
printf("\r%s", message.c_str());
for (size_t i = message.length(); i < 83; i++)
printf(" ");
std::this_thread::sleep_for(std::chrono::duration<double, std::ratio<1>>(0.1));
});
printf(EOL);
Concerning the implementation details, most of the resource manager is split between a resources::manager singleton class that contains a function to retrieve (or load if it does not exist yet) resources based on a string. The retrieval is not performed by the singleton class itself but by specialized template object of the resources::manager::bin class. The idea is that there is one specialized bin by resources types (texture, text file etc.) and the type is dynamically retrieved using the typeid of the specific resource.
Bins can either load the resources directly or defer the loading to a resources::manager::loader thread class. All operations are thread safe. To procure this thread-safe behaviour, all the data is actually inserted in a resources::manager::bin::cache class object that is allocated on the heap. In case of a reload, a new resource object is created and atomatically swapped with the original one in the cache object once the loading is done.
Resources themselves are accessed by proxy classes sync_resource and async_resource. During construction, they access the resources manager to get a pointer to the cache. When accessing the derefence operators (*, ->) they atomically load the actual resource objects. This is also the moment where the null resource is called if the resources does not exist.
Finally, files changes are monitored by a file_change_monitor thread class. Some technical trade-off had to be made on that system. On Windows, to my knowledge, you cannot track individual files directly but only directories. You then receive a notification via a Windows event everytime a modification has been made in the given folder, being the file you are watching or not. You then browse a list and check if the last write timestamp has changed from when you inserted the file in the watcher queue. This works for file change, creation and removal. Note also that the std function for getting the timestamp will throw an exception if the file does not exist. On Linux, things are different because inotify allows you to track file inode directly. This is more efficient because you only receive notification for the file you are watching and not other files in the same directory. However, the approach is limited because although you can detect file modifications and removal, you cannot monitor file creation. inotify will not work if the inode is not existing which prevents watching file that does not yet exists. It is of limited impact but it departs from the behaviour implemented on our Windows version which is more general (although slower). The same problem also happens when you delete the watched file and later re-create it. In this case, the inode id has changed and you will get a removal notification but no new modification after since you are not watching the correct inode. For these reasons, I chose to watch directories in Linux the same way as I did in Windows, even if it is less efficient.
That concludes the post of today! I hope you will find resources manager as helpful as I did :) I also tried to address my programming post a bit differently by focusing more on the behaviour of the program rather than on its implementation.
I would like to give a big thanks to Samuel, Arif, James, Lilith, Mehmet, Vaclav, Sivaraman, Jesse, Themulticaster, Jon, Marcel and Kewei who have supported this post through [∞] Patreon. I remind you to take the occasion to invite you to donate through Patreon, even as little as $1. I cannot stress it more, you can really help me to post more content and make more experiments! The device presented in this post was paid 100% through the money collected on Patreon!
[⇈] Top of PageYou may also like: