For Arcane Algebra, I wanted to utilize the immersive capabilities of VR to make players feel powerful while writing out their solutions. To do this, I used a convolutional neural network model trained on the MNIST dataset, to parse hand-written digits.

When given a 26x26 greyscale image of a number, the model will return a confidence value of what that number might be. By finding the value with the highest confidence interval, we can fairly accurately interpret a user’s handwriting.

I did initially run into a lot of issues with the accuracy of the model, despite it having a 99% accuracy rate with the training data. I intended on disguising this inaccuracy as an “unstable spell”, which would occur when the model did not meet a certain level of confidence for a number. When players cast an unstable spell, the monster they were facing would be sent backwards, giving them additional time to try writing their solution again.

With guidance from some of my peers, I found that this inaccuracy was caused from the way my images were being created. While the images I was feeding the model had thick lines and often took up the whole image, the training dataset always had the image centered and written with very thin lines. This difference in how the data was presented caused the model to behave unpredictably. Yet, it was resolved by adding some padding and decreasing the stroke width of the drawings.