Skip to content

Music and Sound Effects

Happy New Year.

I've been putting off this part for a while. Partly because there was still a lot to be done in the code, but mostly for personal reasons. I love music - I love recording and producing it; I love writing it; but most of all, I love the collaborative effort of multiple people pouring their souls into something. Since my brother died, I haven't written or recorded anything. He had such a gift for it, and I loved collaborating with him the most.

I've been recording music since I was in high school. I started off using a program called Cool Edit Pro, which was eventually acquired by Adobe and rebranded as Adobe Audition. Around the same time, I played around with a sequencing program called Fruity Loops, which was also rebranded as FL Studio. I would sequence drums in FL Studio, and export the results as a WAV file, and import it into Cool Edit Pro to record guitars, bass, and vocals. In hindsight, there were much easier ways of going about this process, but I was a clueless teenager mostly mimicking my brother.

Sometime in college, I bought my first multi-input interface and made the switch to using Avid Pro Tools. I made some cash and scored some free meals by doing recording sessions for local bands, but mostly I just had fun experimenting on my own.

Since graduating college and starting a career, the amount of music that I've made has declined significantly, but I still put a lot of time, money, and effort into building up my own home studio. Guitars, amps, drums, recording hardware, powerful computers - things that seemed so unattainable as a teenager became just a matter of saving up for a couple of paychecks as an adult with a programmer's salary. My brother loved to come visit and just spend an entire weekend in my basement recording songs that were stuck in his head. The last song we recorded together was a cover of Hybrid Moments by the Misfits.

I opened up Pro Tools for the first time in well over a year, and there it was under the "Recent" tab.

Hybrid Moments

I cried.

20.1 The Soundtrack

I don't even know how much detail I should go into regarding the process of writing and recording music. I once considered myself pretty good at writing music, but musical creativity is a muscle that must be exercised, and mine has atrophied quite a bit. I've learned a lot about music theory in my late-20s and early-30s, but I've done very little in the way of writing and recording outside of my brother's infrequent visits.

We have an Alesis Recital keyboard in our home, originally purchased for my daughter back when she was taking lessons. She doesn't take lessons anymore, but she still loves to play, often learning songs from movie soundtracks, or transcribing music that she plays on her cello for her school orchestra. She really is incredible, and I find myself jealous of her ability to learn so quickly.

The keyboard isn't particularly expensive or high quality, but it's sufficient. I often fiddle around on it, and I tend to learn something new about pianos every couple of months just by messing around on it. I primarily play guitar, and I have learned to play drums over the years as well, but piano is still a relatively new experience for me.

The keyboard is set up in on of the front rooms of our house, next to my wife's quilting station. She happened to be working hard on a quilt this week, so I happened to be sitting at the piano more often than I normally would. I had a bit of a "eureka" moment regarding chord structure on the piano. Previously, I would just play whichever keys sounded right together, but I started to notice visual patterns in how my fingers were spaced apart. I understand notes, chords, how they relate to one another, and have even visualized it using the circle of fifths, but I've never seen it within the keys of a piano before.

The First Attempt

I started off thinking that an electronic sound would be appropriate for a space-based runner. I had never done MIDI sequencing in Pro Tools before, but it wasn't too complicated to figure out. I spent way too much time scrolling through the different pre-packaged "voices" before finally whipping up some arpeggios over a bass pedal, using my new-found knowledge on the piano.

I spent a day or two extrapolating more "parts" out from the original tune, when my wife made an interesting comment: "It's good, I just can't sing it." She wasn't wrong, arpeggios are not just hard to sing, they are also hard to remember. I scrapped the idea and started over.

The Second Attempt

I simplified the idea greatly, using the same bass pedal, but playing a lead motif that is capable of sticking to the player's brain for hours after they stop playing. Again, I extrapolated more chords into a bridge, and structured the song in such a way that it would loop endlessly.

The lead was indeed memorable, but it was also abrasive. After listening to it berate my ears over and over for a couple of days, I just didn't want to hear it anymore. My wife didn't hate it, but she wanting something more from the song. It really wasn't fleshed out enough, but I couldn't bring myself to work on it any further.

The Third Attempt

I decided to go back to my roots and play something on guitar. I discovered that my Pro Tools license was actually expired, and so my virtual guitar amp, Eleven MK II, wasn't working correctly. I spent several hours trying to get that to work before caving and upgrading Pro Tools and all of my plugins to their latest versions. Evidently, my license just wasn't immediately available, and after a chat with Avid support, everything started working again.

I spent a lot more time playing around with different guitar amp "voices" and effects. I ended up choosing a clean tone with a lot of delay - it felt like it fit the space theme nicely. The song I came up with had a less abrasive lead on guitar, a bass line played on an actual bass guitar, and a rhythm section sequenced using an electronic piano voicing. The song alternated between two sections, obviously meant to repeat indefinitely.

The result was alright. I didn't love it, but I didn't hate it. My wife said it sounded "sad", even though it was played in a major key.

I whipped it up into the game so that I could hear it while playing the game, just to see how it felt in the context of the game. Doing that made the song a definite "no", it just didn't fit.

Market Research

There is an incredible YouTube channel called 8-bit Music Theory that does a fantastic job breaking down video game music and explaining why the songs work so well in the context of the games that they are in. I've been most fascinated by his videos breaking down the major modes and their uses in video games.

Specifically, I've been intrigued by the Phrygian mode. He uses examples from the Final Fantasy and Pokémon series, but I've also identified its usage within The Legend of Zelda, and even the Power Rangers theme song. Honestly, I feels so played, like the creators of all of these forms of entertainment knew exactly how to hook my younger self in the mid-to-late 90s.

The Fourth Attempt

Well, I don't even know how long it's been since I wrote the previous blurb about the major musical modes - it must have been at least a week, maybe two. That's not to say I haven't gotten anything done. On the contrary, I have accomplished a lot during this time - I've just been neglecting writing about it!

I wrote and recorded another song, but this time without my guitar. I was slightly disappointed at first, but I've gotten a lot of positive feedback this time around. The song has even passed the ultimate test: it's gotten stuck in the heads of both my wife and my daughter, and they weren't even annoyed by it!

The song is actually split into three tracks: one for the title screen, one for the game itself, and a third for the "game over" screen.

The title screen music foregoes the use of any drums, while looping over the backing arpeggio that is consistent through the rest of the song. The arpeggio is paired with a rhythm synthesizer that sounds a bit like an organ. Both the rhythm and the arpeggio alternate between an E-minor chord and its flat second - that's the Phrygian mode! Overall, the title screen music establishes an intense tone that is firmly rooted in a space theme.

The game play music is further split into three parts. It starts with the same rhythm and arpeggio chord from the title screen, this time coupled with a drum track. After a couple of bars, it transitions into a section section consisting of an additional lead arpeggio, which itself moves along the root chord that the backing arpeggio establishes. The third section veers away from the intense Phrygian sound in an attempt to provide some relief to the player. The backing arpeggio continues while the lead switches to a more traditional melody. The song lasts for exactly one minute and repeats seamlessly.

The music that plays after the player dies is much more somber, consisting of only the rhythm "organ", playing the same rhythm that backs the more traditional section at the end of the game play song, but without any drums, bass, or leads. This track does not repeat, and is designed specifically to provide tension that is resolved by both the title screen music and the game play music. It leaves the player hanging and wanting to play more.

The whole thing is designed to be an entire experience, alternating between an intense and anxious mood and providing comfort and relief - but not too much.

Musical Arrangements in Code

I hadn't actually written any support for playing or managing long-running music tracks in the existing AudioManager, so I spent quite a bit of time adding all the functionality I needed. I started by essentially copying and pasting all of the code that supported "effects" into a "song" paradigm, adding a new SongType enum, as well as a Mode enum to support playing a song Once or on Repeat.

The AudioEngine implementation utilizes the handy scheduleBuffer: atTime: method of the AVAudioPlayerNode class in order to precisely schedule loops of the current song. I tried doing this in several different ways, but ended up loading the music files into buffers when the app first starts up, and scheduling those existing buffers, rather than trying to load the files on-the-fly. Using buffers directly will prevent any loading times that could be experienced at runtime, and scheduling by specific sample counts will hopefully prevent any floating-point errors that can occur using timestamps.

The OpenAL implementation is unfortunately a bit more complicated. OpenAL doesn't support callbacks that fire when a track has finished playing, so I introduced a new poll() method to the AudioManager interface that the Engine calls once per frame. The OpenALAudioManager uses this new method to check the state of any currently playing sources and update a local state appropriately, which can then fire a callback if needed. I also use this method to keep track of how much time has passed so I can repeat the currently playing music track. Since this implementation is based on time rather than samples, its susceptible to floating-point errors and sometimes sounds a little off. I'm not focusing too hard on the browser version of the game so I'm going to ignore it for now.

Much more annoying than the actual AudioManager implementations were the changes required to support lifecycle events in Scampi and Pesto (I pretty much ignore any concept of a "lifecycle" in Alfredo).

In Scampi, in order to pause the track scheduling and correctly suspend audio playback, the AVAudioPlayerNodes and AVAudioEngine should be paused and re-started whenever the app goes into the background. I added new pause() and resume() methods to the AudioManager and implemented them in the AudioEngine implementation, and tried a few different methods of calling them from the ScampiAppDelegate. No matter which lifecycle events I used to pause and resume playback, the game seemed to get back into a state where the fixedUpdate() method was ticked based on the amount of time the app was in the background - I thought I fixed this already!

I ended up adding a big fat conditional return; at the beginning of the Engine's tick() method using a new isPaused() method of the LifecycleManager (which has mostly gone unused until now). Pausing and resuming the LifecycleManager, along with the AudioManager and the TimeManager, seemed to do the trick. With so many sub-systems being paused and resumed at the same time, I decided to just encapsulate all of those within new pause() and resume() methods in the Engine itself. I rearranged the ScampiViewDelegate and ScampiAppDelegate to just pause and resume the Engine accordingly, and all was well.

Pesto had completely different problems. I still had to pause and resume all of the audio sources in the OpenALAudioManager, but that wasn't a problem. Browser tabs prohibit audio from playing until the user specifically interacts with the web page. In order to try to keep this browser-specific nuance separate from the OpenAL implementation, I decided to just pause() the OpenALAudioManager as soon as it got created within Pesto's main() method, and resume() it from within some input handlers inside of the WebInputManager.

I gave Pesto the same treatment as Scampi by moving the visibilitychange callback from the WebTimeManager into the main() method itself, and changing it to pause and resume the whole Engine rather than just the TimeManager.

All of that was working correctly, but I was noticing some strange audio artifacts at the end of each music track. Come to find out, OpenAL buffers should only contain the actual PCM data, not the entire WAV file, so I had to update the WebOpenALFileLoader with some logic to extract the PCM data from the WAV file, which is luckily very well documented all over the internet.

I had almost forgotten that I decided to normalize all of my audio implementations around 16-bit WAV files with a 44.1kHz sample rate, so I made a few modifications to make that happen.

This commit shows all of the changes that I made, and actually includes the audio files if you're interested in giving them a listen! I didn't anticipate spending an entire weekend on it, but I'm so pleased with the result. You can experience the music in-game here, though you'll notice longer download times than before.

20.2 Sound Effects

I spent all of my free time over the last few days making sound effects out of basic primitive sound waves and arpeggios. A lot of the effects were surprisingly easy to make, while others took way more time than I had predicted.

Most of the sound effects sound like they came straight out of an old Super Mario game, but I can assure you I made all of these from scratch in Pro Tools! The resemblance comes from the basic fact that the old Mario games only used basic signal generators to produce their sounds, so it's not surprising that my results are similar.

All in all, I made 14 clips:

  • 2 sounds for pressing a button ("down" and "up" events)
  • 5 different sounds for colliding with an asteroid (1 for each size)
  • Collecting a powerup
  • Colliding with a mine
  • Healing a shield
  • Ship movement
  • The explosion when the player dies
  • "Ticking" for the XP bar's progression
  • Leveling up

Three of the effects really took up most of my time, requiring a lot of tinkering and iteration.

Swoop

The first was a "swooping" effect that plays whenever the player moves left or right. I started with more of a "swooshing" sound using some white noise, but decided to go with a sound that is more akin to Mario's jump. I'm not exactly sure why, but it just felt right.

Kaboom

The second was the sound that is made when the player dies. I knew that I wanted something "noisy", like the attack sound effects from the old Pokémon games, but I honestly didn't know how to achieve it in Pro Tools. I spent a while doing some research about how the Nintendo Gameboy handled its noise channel. The truth of the matter was fairly unremarkable: it lets you adjust the pitch of the channel just like the other channels. I decided to generate a sample of white noise, and then import that sample into a MIDI controller called Structure Free in order to adjust the perceived pitch. It worked surprisingly well!

Kablooey

The third was the sound of colliding with a mine. I wanted something "explosive", but not too similar to the sound of the player's ship blowing up. I went through the most iterations with this particular sound, finally converging on a combination of white noise, a wave sample that really sounds like you've done something wrong, and - completely different from anything else in the game - a big thumping kick drum to really emphasize the impact.

Some Code Fixes

There was a bit of boilerplate required to load the effects, but most of it was just placing the appropriate calls to play() the effects in their rightful locations in the game systems. I did make a couple of additional changes though.

In order to really drive home the explosion of the player's ship, I decided to add a stopSongs() method to the AudioManager. This allows the music to come to a halt while the explosion sound effect plays, and the somber "game over" music to start a few seconds later.

I noticed that the AudioEngineAudioManager would saturate its available effect sources pretty quickly, even when increasing the number of sources to 24 (up from 8), which resulted in some effects failing to play at all until the sources finished their current track and returned themselves to the pool. I decided to switch to a least-recently-used (LRU) pool, in which the sources are popped from the top of the queue when they are used, and immediately pushed to the back of the queue. If a source is popped before its current sound has finished playing, that sound is interrupted and the new sound is played from the beginning instead. The player would prefer to hear what is currently happening rather than the tail-end of what was happening in the past. I had already used this method in the OpenALAudioManager, I'm not sure why I diverged between them. In any case, not relying on the callbacks to re-insert the sources into the pool means I can get rid of the mutex - a double win!

In Pesto, if you play an effect and leave the tab (thus pausing the game), and subsequently move back to the tab, the previously played effect will play again. This is because I was calling alSourcePlayv() on the entire collection of effect sources in resume(), even though some of those sources weren't playing when the player left the tab. To fix this, I just had to check the currently stored state that we were already using to trigger callbacks.

I made some fixes to the timing of the scrolling tutorial text so that multiple instances of scrolling text should never overlap. I also made the tutorial asteroids always have a size of "4" so that the healing portion of the tutorial is more noticeable - it felt really awkward if an asteroid of size "1" happened to spawn and barely moved your shield's health bar.

I had to add a bit of extra logic to the LevelTrackingSystem in order to play the XP sound effect at a reasonably convincing rate. I started by just playing it every 1/20th of a second, but it felt very wrong when the player only got a tiny bit of XP, so I added some logic to make sure the amount of XP goes up by at least 1 before playing the effect.

There was actually already a bug in the LevelTrackingSystem in which the level would not increase if you happened to get the exact amount of XP required to hit the next level. The bug was purely visual and didn't affect the power that the player gained. In any case, it was just a bad usage of > instead of >=.

Some Feedback

Before committing everything, I did a final pass to normalize the volumes of all of the sound effects, while making sure to keep the volume reasonable relative to the music. I'm really happy with the final result. Other people seem uncharacteristically excited about it too - the addition of audio seems to have made my daughter perceive the game as a complete package now, rather than just another one of my random side projects.

I got some good feedback from a few friends as well. Some have noticed that the music doesn't quite loop gracefully in the browser, but I'm choosing to ignore that. Others gave more advice around the actual composition of the music - varying the bass line, more dynamics or even silence to build upon the tension/release cycle. The only actual negative feedback I got was that one person didn't like the drawn out "drone" of the "game over" track. I can't say I blame them, but I actually like it, so I'm going to leave that one in for myself.

Nothing is final, really. I could release the game and continue to iterate on the music in subsequent app updates. In any case, here is the commit that adds all of the sound effects, and you can hear them for yourself here. It's unfortunate that I can't convey the new audio in the form of screenshots, but I'd encourage you to experience it for yourself anyway.