Off-By-One Errors

Each year, sometime between Thanksgiving and Christmas, a large group of my friends gathers together for our annual "Friendsgiving" party. Since most people are preoccupied with family get-togethers and vacations over the actual holiday weeks, we tend to schedule our party on whichever December weekend happens to be the most convenient for the largest number of people.

Our parties have progressed quite a bit over the last decade. I met most of these people in college, so you can imagine how crazy the first iterations of these parties got. More than anything, it was an excuse to party (though I've never been one to drink, I still enjoy the company). These days, a vast majority of the group has settled down, gotten married, had kids, and some are still planning to have more! As such, the "parties" are more tame than they once were. Even so, it's always wonderful to see my friends in person, rather than chatting over Slack.

I had planned to pass my phone around and let people play my game. Most of them have seen the web version, but the game is designed for mobile devices, so I was hoping to let them play it on an actual mobile device. It was a good opportunity, but naturally, things didn't work out like I had hoped.

iOS Development Quirks

Hours before the party, I realized the game wasn't actually available on my phone, and that I needed to install it again. You see, for developers that have not yet paid for an iOS development license, apps installed to your device expire after about a week. Admittedly, I hadn't been working on my game very much over the last month, so it was not unusual to see that it needed to be reinstalled.

This is where the trouble began. When I plugged my phone into my laptop and attempted to install the game, I was greeted with an error message indicating that I could not install the app because my phone was running iOS 17. I had recently updated my phone to the latest version of iOS after being prompted to do so - I did not go out of my way to update to an early beta version or anything like that.

Though I was surprised to see the error, I figured I knew what to do about it: building for iOS is supported by Xcode, so I must simply need to update Xcode! I opened up the App Store, only to discover that I could not update Xcode. The description of Xcode within the App Store indicated that there was a newer version, but the App Store would not let me update it, nor did it give me any indication as to why. I tried restarting my computer, but was not lucky enough for that to fix it.

After a bit of Googling to no avail, I had a vague idea: I must need to update my laptop's operating system in order to install the latest version of Xcode. I checked updates to my laptop's macOS installation, and lo and behold, macOS Sonoma was available. I clicked the innocent-looking button to install the update, and I waited...

Thirty minutes passed, and I waited some more.

Another thirty minutes passed, and I continued to wait.

My wife could sense my frustration, but informed me that it was time to leave for the party. The progress indicator said that only 4 minutes were remaining, so I told her to wait for 4 minutes.

To Apple's credit, the macOS update did indeed finish in about 4 minutes. However, I still needed to install the latest version of Xcode. I opened up the App Store, and sure enough, there was now an update available. I clicked yet another seemingly innocent update button, and waited again.

The Xcode update took about 15 minutes, which felt like an eternity as my family sat and waited for me to leave for the party. My daughter, accustomed to me rushing her out the door, actually got into the car, only to be told that I was trying to finish something before leaving, so she came back inside and waited. I could sense her frustration as well.

Once Xcode was finally updated, I opened it up, hoping to quickly install my game so that we could finally leave. I was greeted with a popup for me to choose which platform I would be developing for. After choosing the iOS option, yet another progress bar appeared, indicating the download progress for the iOS SDK. So I waited again.

Another grueling 15 minutes passed by, and Xcode was finally ready. I switched back to CLion (because that's where I actually develop my game), and attempted to install the game, but the compilation failed because my application manifest did not declare a CFBundleDisplayName. After adding that property to Scampi's Info.plist, I tried once more, only to receive this error from ios-deploy:

[ !! ] Unable to locate DeviceSupport directory with suffix 'DeveloperDiskImage.dmg'. This probably means you don't have Xcode installed, you will need to launch the app manually and logging output will not be shown!

A quick Google search indicated that this is a widespread issue, and that the ios-deploy project does not intend to support iOS 17+.

I desperately switched back to Xcode to see if I could install the game from there instead. I clicked the "run" button, and the app quickly compiled and started up just fine, but the frame rate got very slow very quickly - an indication that I was running a "Debug" build of the game (rather than a "Release" build). I switched Xcode's configuration to install the Release build instead, but the game crashed instantly, with no obvious error message.

I gave up. I reverted back to the Debug build so that I could have something to show, and we left for the party, very fashionably late.

Avoiding the Subject

Since my wife was driving, I tried to play the game on the way to the party just to see how bad the experience would be. While the ShopScene was just fine, the InfiniteRunnerScene became completely unplayable within 5 seconds.

This is unfortunately just a result of a completely un-optimized Debug build performing a ton of physics calculations too frequently. The fixedUpdate() method will always try to be executed 50 times per second, and if it can't, then the engine becomes permanently "stuck" trying to catch up to the desired number of fixedUpdate() iterations. Some engines have some "short-circuit" logic to prevent that sort of death spiral, but mine does not.

In any case, I simply couldn't show it in that state to anyone at the party, and I was deeply disappointed.

All of my friends know how much effort I've put into the game over the last year, so the topic was bound to come up in conversation. I mostly dismissed any questions about it, stating that I hadn't been working on it recently, which was true. Internally, I was screaming in pure disappointment.

One of my friends is working on his own game using Godot, and we discussed the possibility of using my engine for it instead. We had a very interesting conversation, touching on the pros and cons of using your own engine, requirements for his game that my engine does not yet meet, and ultimately posing the question: what benefit would my engine have over just using Godot? We didn't come to any formal conclusion, but I really enjoyed the conversation.

After the Party

The title of this section is a reference to a song by The Menzingers, but my week was definitely not as glamorous as the song.

As I mentioned before, the game would crash when running a Release build, but not a Debug build. It took me entirely too long to figure out what the issue was.

I spent several nights fiddling around with different settings for the app in Xcode, assuming that the issue was due to the iOS 17 upgrade. I destroyed a bunch of caches to make sure the build wasn't being polluted by any code intended for older versions of iOS. I sprinkled logs throughout the game, in an attempt to discover where exactly it was crashing. I added exception handlers, hoping to bubble up the problem into a nice readable log message (which didn't work at all).

The logs that I added didn't seem to be printed out in the order that they got executed. That's not completely out of the ordinary; it's entirely possible that the logs are flushed asynchronously from the execution of the application. While it did make it somewhat difficult to debug, I noticed that the draw() method was only ever executed once, even though the update() method was executed multiple times. This led me to believe that the issue was rooted within the renderer.

I switched the Engine's default scene to the good old TestScene, just to make sure it wasn't something specific about the game's dynamic renderable creation. When that failed, I opened up the MetalRenderer, and commented out all of the feature renderers, and updated the getEntityIdAt() method to always return an empty std::optional (since the SelectableFeatureRenderer is never created).

The game didn't crash! It wasn't displaying anything, but at least it didn't crash!

I uncommented the feature renderers one by one, testing that the game continued to work each time. I managed to uncomment all of the feature renderers without crashing the game - only the getEntityIdAt() method was still modified. I reverted that change, and the game crashed once more.

For whatever reason, I finally noticed a small window at the bottom of Xcode that printed the game's logs. I had been using the Console app on my computer to view the game's logs this whole time, but apparently Xcode shows them directly (which I suppose is something CLion can't do). Xcode also prints something that the Console app does not: Metal API validation errors.

_validateGetBytes:98: failed assertion `GetBytes Validation
(origin.x + size.width)(4294967296) must be <= width(390).
'

This is the exact moment that I made this error, way back in August. The provided x coordinate is a floating-point number, which gets multiplied by the screen width to convert from a "normalized" screen coordinate (0.0f to 1.0f) into an actual pixel coordinate (0 to the screen's width). When the normalized coordinate is 0.0f, then the resulting multiplication is 0, which is then decremented by 1, resulting in an underflow to 4294967296 (uint16_t's maximum value). This is an unusual example of an off-by-one error.

The line below it does something similar for the y coordinate, except the y value is inverted, such that this issue would only occur when y == 1.0f, which should never happen.

On mobile devices, the logic to detect "hover" states always samples at { 0.0f, 0.0f }, which causes this error on the very first frame. The solution is to simply remove the subtraction for the x coordinate conversion, but leave it for the y coordinate conversion.

Running From CLion

I needed to figure out how I could continue to install the game from CLion, since ios-deploy was no longer an option. This comment states exactly what Apple intends for developers to do, so I whipped up a new device.sh script within Scampi's scripts/ directory to do just that, and updated the "run" configuration in CLion to invoke the script.

When testing the new run configuration, I expected to receive an error message stating that my device was not plugged in. Instead, I received an error message stating that my device was not unlocked, and my phone's screen lit up at the same time! Does that mean the game is being installed and run wirelessly?! I unlocked my phone and tried again - sure enough, the game opened up! Well done, Apple.

Unfortunately, I haven't found a way to enable app logs from CLion in the way that Xcode does. I don't really need to solve that right now though, I'm just relieved to have the game in a working state again.