Implementing Rendering Interpolation in Pulcher
Published:
Here I introduce how rendering interpolation is done in Core of Pulcher. I haven’t really seen anything written up about this for 2D so I thought I might as well document it. Pulcher runs at a specified logical framerate, right now 90hz but I probably will set it to something lower in the future. The loop is simple and looks like
// -- update windowing events
glfwPollEvents();
// -- logic, 90 Hz
msToCalculate += deltaMs;
size_t calculatedFrames = 0ul;
while (msToCalculate >= sceneBundle.calculatedMsPerFrame) {
// -- process logic
++ calculatedFrames;
msToCalculate -= sceneBundle.calculatedMsPerFrame;
::ProcessLogic(plugin, sceneBundle);
// -- update render bundle
renderBundle.Update(sceneBundle);
}
sceneBundle.numCpuFrames = calculatedFrames;
// -- rendering / audio
...
For reference, while the map is being interpolated, the animation system is not being interpolated;
Now, even though the game runs at 90hz, even on my 240Hz monitor I can notice that the rendering isn’t quite updating enough for my brain to see a fluid motion. It’s difficult to describe but having played a lot of 2D games at high framerate, I can definitely pick up when framerate is becoming an issue visually.
The solution to this isn’t anything novel - interpolation of the current and previous logical frame. This introduces a frame of latency, which is regrettable, but it’s preferrable over having a capped rendering framerate. The alternative is to use a delta-time to calculate logic, and while this grants fluid motion, I can’t imagine it is stable or gives reproducible results. Maybe I can talk about this choice at a later date though.
The initial step to prepare for interpolation is to copy some subset of data over from sceneBundle, a bundle of information that can be used to retrieve scene info in Pulcher. This gets copied into RenderBundleInstance, tracked by a RenderBundle. The render bundle can then construct RenderBundleInstance from its tracked instances when given a ms-delta interpolation value (from 0 to 1):
struct RenderBundleInstance {
glm::vec2 playerOrigin;
glm::vec2 cameraOrigin;
glm::vec2 playerCenter;
// allows plugins to query plugin data
std::map<std::string, std::shared_ptr<pul::util::Any>> pluginBundleData;
// 0 .. 1, ms delta interpolation. only used from instances created by
// RenderBundle::Interpolate
float msDeltaInterp;
};
struct RenderBundle {
RenderBundleInstance previous, current;
bool debugUseInterpolation = true;
// constructs dummy render bundle (previous = current = scene), this is so
// that the first couple of frames of rendering are valid & do not have
// garbage memory
static RenderBundle Construct(
pul::plugin::Info const & plugin, SceneBundle &
);
// stores current into previous, and then updates current with scene
void Update(pul::plugin::Info const & plugin, SceneBundle &);
// constructs a render bundle instance from the ms-delta interpolation
// value, from 0 to 1. Most likely `accumulatedMs / totalMsPerFrame`.
// equivalent of pseudo-code `mix(previous, current, msDeltaInterp)`
RenderBundleInstance Interpolate(
pul::plugin::Info const & plugin, float const msDeltaInterp
);
};
Everything is simple enough, but the pluginBundleData is where most the logical operations occur. Since we want plugins to have control over how rendering is handled, they also are able to store their own data into a render bundle instance, and also control how that data is interpolated. In the base-plugin, this primarily means interpolating the animation entities. Some static stuff like the map do not need this since cameraOrigin are more than enough to render them.
The Any implementation looks like this;
namespace pul::util {
struct Any {
Any() = default;
~Any();
Any(Any const &) = delete;
Any(Any &&);
void (* Deallocate)(void * userdata) = nullptr;
void * userdata = nullptr;
};
}
We have an RAII type that wraps a function pointer to a Dealloc function. The c++ std::any solution is complex and overkill for our purposes. The plugin will know the type it stores, and since the only valid operation on the stored data outside of the scope of the plugin is to deallocate it, I defer to the simple/naive solution.
Using a std::map and std::string to store these probably isn’t the best solution, but at some point I’ll switch from strings to hashed strings, and probably replace all my std::map with something like std::unordered_map. The convenience of using both of these right now is that they simplify interfacing to display information through Dear ImGui.
One benefit of having things this way is that a multi-thread approach to processing/rendering the game is pretty feasible. Since data is copied over every logical frame, and then interpolated internally in rendering, the logical components of the game can be handled in a seperate thread. The only tricky part will be handling rendering the creation/destruction of specific resources and sound triggers, though I have an idea of how to approach both of these issues in the future.
Now we have three plugin functions necessary to apply this interpolation;
void (*UpdateRenderBundleInstance)(
pul::core::SceneBundle & scene
, pul::core::RenderBundleInstance & instance
);
void (*Interpolate)(
float const msDeltaInterp
, pul::core::RenderBundleInstance const & previousBundle
, pul::core::RenderBundleInstance const & currentBundle
, pul::core::RenderBundleInstance & outputBundle
);
void (*RenderInterpolated)(
pul::core::SceneBundle const & scene
, pul::core::RenderBundleInstance const & interpolatedBundle
);
Hopefully in the future, when there might exist mods or other plugins, then this should provide all the necessary functionality. The loop now looks like:
msToCalculate += deltaMs;
size_t calculatedFrames = 0ul;
while (msToCalculate >= sceneBundle.calculatedMsPerFrame) {
// -- process logic
++ calculatedFrames;
msToCalculate -= sceneBundle.calculatedMsPerFrame;
::ProcessLogic(plugin, sceneBundle);
// -- update render bundle
renderBundle.Update(plugin, sceneBundle);
}
sceneBundle.numCpuFrames = calculatedFrames;
// -- rendering interpolation
auto const msDeltaInterp =
msToCalculate / sceneBundle.calculatedMsPerFrame
;
auto renderBundleInterp = renderBundle.Interpolate(plugin, msDeltaInterp);
// -- rendering, unlimited Hz
::ProcessRendering(
plugin, sceneBundle
, renderBundle, renderBundleInterp
, deltaMs
, calculatedFrames
);
// -- audio, unlimited Hz
plugin.audio.Update(plugin, sceneBundle);
// ... in ProcessRendering
sg_begin_pass(pul::gfx::ScenePass(), &(passAction);
...
plugin.RenderInterpolated(scene, renderInterp);
...
sg_end_pass();
Now all that has to be done is making my relatively-complex animation system support interpolation. There are a lot of caveats that I might address at a future date, such as animation construction/destruction, when to do flat-interpolation for 2D animation such as flipping, UV coords, etc, but for now we are just trying to look at rotation and origin interpolation.
At some point I will talk about my animation system, probably after I polish and clean it up, as it is something I find very unique about developing in Pulcher. Having a toolset/library built on top of ImGui makes prototyping very fast & simple. Right now all the animation logic is handled through the base plugin, but several structs sit in the Pulcher library animation header.
the final piece of the puzzle is describing what should be stored in the pul::util::Any::userdata;
template <typename... T>
using InterpolantMap = std::unordered_map<size_t, T...>;
...
struct Interpolant {
pul::animation::Instance instance;
};
...
struct BaseRenderBundle {
InterpolantMap<plugin::animation::Interpolant> animationInterpolants;
std::vector<plugin::animation::Interpolant> animationInterpolantOutputs;
static void Deallocate(void * data) {
delete reinterpret_cast<BaseRenderBundle *>(data);
}
};
On a logical update, all animation instances will be stored in the InterpolantMap, with their entity ID stored as the key. When doing interpolation, the previous animationInterpolants will be iterated, and an attempt to find the instance in the current/future interpolant will be made. If one doesn’t exist, then the entity was destroyed at some time in the frame. I might also in the future make some additional iteration for the current interpolant, to check for entities created that frame, though if this will be necessary I’m unsure at this point.
I won’t describe the underlying animation system here, I might describe it in a different blog post though as it is a very nice tool. All that really needs to be said is that it stores cached matrices for each element it renders, and it can also emit a buffer of uv-coords/origin vertices after matrices have been cached.
The UpdateRenderBundleInstance looks like
PUL_PLUGIN_DECL void Plugin_UpdateRenderBundleInstance(
pul::core::SceneBundle & scene
, pul::core::RenderBundleInstance & instance
) {
auto & registry = scene.EnttRegistry();
InterpolantMap<plugin::animation::Interpolant> animationInterpolants;
{ // -- store animation information
auto view = registry.view<pul::animation::ComponentInstance>();
for (auto entity : view) {
auto & self = view.get<pul::animation::ComponentInstance>(entity);
// .. culling
// copy the entire instance
auto instance = self.instance;
// compute cache
plugin::animation::ComputeCache(
instance, instance.animator->skeleton, glm::mat3(1.0f), false, 0.0f
);
// move instance w/ the entity ID
animationInterpolants.emplace(
static_cast<size_t>(entity), std::move(instance)
);
}
}
{ // -- store data into the instance bundle
auto & instanceBundleDataAny = instance.pluginBundleData["base"];
if (!instanceBundleDataAny) {
instanceBundleDataAny = std::make_shared<pul::util::Any>();
instanceBundleDataAny->Deallocate = ::BaseRenderBundle::Deallocate;
instanceBundleDataAny->userdata = new ::BaseRenderBundle;
}
auto & instanceBundleData =
*reinterpret_cast<::BaseRenderBundle*>(instanceBundleDataAny->userdata);
instanceBundleData.animationInterpolants = std::move(animationInterpolants);
}
}
The Interpolate looks like
PUL_PLUGIN_DECL void Plugin_Interpolate(
float const msDeltaInterp
, pul::core::RenderBundleInstance const & previousBundle
, pul::core::RenderBundleInstance const & currentBundle
, pul::core::RenderBundleInstance & outputBundle
) {
// -- get previous/current bundle
auto & bundleDataPreviousAny = previousBundle.pluginBundleData.at("base");
auto & bundleDataCurrentAny = currentBundle.pluginBundleData.at("base");
// -- construct output any for this frame
auto & bundleDataOutputAny = outputBundle.pluginBundleData["base"];
bundleDataOutputAny = std::make_shared<pul::util::Any>();
bundleDataOutputAny->Deallocate = ::BaseRenderBundle::Deallocate;
bundleDataOutputAny->userdata = new ::BaseRenderBundle;
// -- retrieve baserenderbundle for each
auto
& previous =
*reinterpret_cast<::BaseRenderBundle*>(bundleDataPreviousAny->userdata)
, & current =
*reinterpret_cast<::BaseRenderBundle*>(bundleDataCurrentAny->userdata)
, & output =
*reinterpret_cast<::BaseRenderBundle*>(bundleDataOutputAny->userdata)
;
// -- forward rendering information
plugin::animation::Interpolate(
msDeltaInterp
, previous.animationInterpolants, current.animationInterpolants
, output.animationInterpolantOutputs
);
}
...
void plugin::animation::Interpolate(
const float msDeltaInterp
, InterpolantMap<plugin::animation::Interpolant> const & interpolantsPrev
, InterpolantMap<plugin::animation::Interpolant> const & interpolantsCurr
, std::vector<plugin::animation::Interpolant> & interpolantsOut
) {
interpolantsOut.reserve(interpolantsPrev.size());
for (auto & interpolantPair : interpolantsPrev) {
auto const interpolantId = std::get<0>(interpolantPair);
auto const & previous = std::get<1>(interpolantPair).instance;
// locate current, if it doesn't exist then this object has been destroyed
auto currentPtr = interpolantsCurr.find(interpolantId);
if (currentPtr == interpolantsCurr.end()) { continue; }
auto & current = std::get<1>(*currentPtr).instance;
// copy instance
auto instance = previous;
// create an interpolated instance to compute vertices from
instance.origin = glm::mix(previous.origin, current.origin, msDeltaInterp);
for (auto & piecePair : instance.pieceToState) {
auto const & label = std::get<0>(piecePair);
auto & state = std::get<1>(piecePair);
auto const & statePrev = previous.pieceToState.at(label);
auto const & stateCurr = current.pieceToState.at(label);
state.deltaTime =
glm::mix(statePrev.deltaTime, stateCurr.deltaTime, msDeltaInterp);
state.angle = glm::mix(statePrev.angle, stateCurr.angle, msDeltaInterp);
state.flip = stateCurr.flip;
state.visible = stateCurr.visible;
}
// compute vertices
plugin::animation::ComputeCache(
instance, instance.animator->skeleton, glm::mat3(1.0f), false, 0.0f
);
plugin::animation::ComputeVertices(instance, true);
// store output to be rendered later
interpolantsOut.emplace_back(std::move(instance));
}
}
I should mention that there is one thing I currently do not like with how I am approaching this; there is no custom allocator. This is a perfect use-case of such a thing too, as I will need to copy/allocate a lot of information over, but it will only be used temporarily, some for one logical frame, and others for just one rendering frame. I could essentially pass this custom allocator to the plugins during the update/interpolate functions, but I will have to deal with this at a later date when I overhaul everything to use allocators. For now I just use heap allocation.
then finally RenderInterpolated
PUL_PLUGIN_DECL void Plugin_RenderInterpolated(
pul::core::SceneBundle const & scene
, pul::core::RenderBundleInstance const & interpolatedBundle
) {
// -- get current bundle
auto & bundleDataCurrentAny = interpolatedBundle.pluginBundleData.at("base");
// -- retrieve baserenderbundle for each
auto
& current =
*reinterpret_cast<::BaseRenderBundle*>(bundleDataCurrentAny->userdata)
;
// -- forward rendering information
plugin::animation::RenderInterpolated(
scene
, interpolatedBundle
, current.animationInterpolantOutputs
);
}
Now interpolation works & look like so:
