Gamebreak: The Spectacle of Music

When you’re blind, live performances aren’t actually all that different from listening to music in a studio. There’s the crowd, of course, and we can usually detect when an artist is really singing versus using a dub track, but otherwise there is little difference. Sure, live concerts are still fun, as certain artists have a way of creating an atmosphere. The music is all around you, the crowd is going crazy, and maybe the artist throws in some chatter to get the crowd extra hyped. These things are all great, but there’s so much more that we blind folks miss all the time, and that’s what I want to talk about.

Concerts are more than just musical performances these days. They are visual spectacles as well, as artists go through frequent costume changes, lighting effects are used, stage platforms move, dancers do very specific routines to the song, and so on. All of this is generally lost on a blind audience member. We can’t even detect when the artist moves around, since we’re hearing the audio through the arena or stadium speakers. Those artists who begin their shows with visuals on a big screen with a rumble, or single long musical note in the background might not realize it, but we’re mentally skipping that part of the concert, since we cannot see it. All it is to us is an indicator that the music will be starting soon. I’m writing all this for context, and I promise I’m about to get to the point. Bear with me.

I have written many things about Netflix and their inclusion of audio description over the past few months. They’re just incredible about it. Nevertheless, last night I experienced yet another surprise. I was browsing through the catalog, looking for something new to watch after having just finished yet another amazing series, when I encountered the Taylor Swift Reputation Stadium Tour. I personally admire Taylor as an artist, and thought it was cool Netflix did something with this. Plus my fiancé actually photographed this tour when it was in Ohio, so I was doubly intrigued. I figured I might llisten to it, and genuinely didn’t expect anything more than crowds, music, and maybe Taylor occasionally talking to the audience. I was wrong.

Right from the jump, the entire event was audio described. Suddenly, all the spectacle of the show was there for me as well. The dancers who came out in military garb during “Ready For it,” the way Taylor made male dancers fall with a wave of her hand during “I Did Something Bad,” (an action that is relevant to the song itself), it was all there. Every costume change, every platform rising over the audience, and even specific mentions of fans that got some screen time, such as a fan in a carnival barker’s outfit. Finally, a concert which took me beyond the music. In a way, I was there with that crowd more than I have been at some concerts where I was physically present. It is difficult to describe, but it was a wonderful experience.

To be perfectly clear, I am aware that many theatres these days have audio description support for plays, which is also a wonderful thing, but I have not seen much mention of this being used in a big arena or stadium concert environment. I am also not saying this is the first time this has ever happened, only that this is the first instance of this I’ve seen. Netflix has once again gone above and beyond here. After all, there are still certain things that Netflix does not provide audio descriptions for, such as standup comedy specials. I grant you there wouldn’t be much to describe in that case, but hey, some comedians do use visual humor. It therefore means something to me that the extra step was taken to describe this show.

It is my sinceer hope that ideas like this are adapted into live shows more. Not just later, when they’re posted on platforms like Netflix, though that is a wonderful thing, but in the moment. If a blind person attends any kind of big live show or event, they should have some access to what’s going on. For plays, there is generally someone up in a booth describing the play live as it happens, while the blind people in the audience wear headsets to hear it. Why couldn’t this be adopted to larger events, and tours like this one? Have a live describer for concerts, and with something like, say, a wrestling event, patch us into the commentary channel. Of course, I am aware wrestling commentators talk about other things during commercial breaks, so synchronize the system somehow so we don’t hear too much. The point here is that I think it’s very doable. Sure, we have services like Aira, but the astronomical prices of that service don’t exactly make it an ideal solution. Best if the arena, and/or the concert promoter provides the service as part of the show.

For now, I hold the Taylor Swift Reputation Stadium Tour as the highest in live concert audio description. It’s basically perfect, as the descriptions are cleverly interwoven with the music so as not to step on any lyrics. We can sing along even as we take in the spectacle that is music today. I hope this article makes you think, and as always I encourage comment and discussion. Thanks so much for reading, and continue to be awesome!

GDC: The Panels

So I’ve been thinking. Why do just 3 blogs about GDC when you can do 4? I said before i’d talk about my panel, the people I met, and the awards ceremony, but then I thought “Why not talk about what you learned from the other panels you saw there?” So that is what this particular blog post is about. I think you’ll find some interesting things in here, especially where audio is concerned, as it was honestly my focus when deciding which panels I’d attend.

So first, let’s talk a bit about the audio design panels I attended. I admit a lot of this talk was technical, and over my head even as someone who has done some minor audio production. Still, there was information to be gleaned. For example, you know that sound they always use in horror movies and games to build tention? That kind of screeching that sounds halfway like a musical note? I found out exactly what that is. That, my friends, is a sheet of metal when dry ice is applied to it. Dry ice causes metal to make that sound. I thought that was fascinating.

Furthermore, though I was already aware this existed to some extent, the things done with sound morphing are also impressive. Try taking the sound of a refrigerator, and turning it into a futuristic forcefield. It’s a longer process than you might think, and the sound goes through several iterations of pitch change and modification to finally get that sound. Or how about gunshots? Sure, guns can be recorded live to bring a more realistic sound to a game, but volumes have to be normalized so as not to destroy the ears of gamers everywhere. However, we still want our shots to sound impactful, so then we add a bit of a base boost, or we increase the reverb effect that falls off at the end. It’s amazing how one small sound can undergo so much to sound just right.

And then, once all that sound design work is done, you have to worry about the mix. This was a completely separate panel. I won’t say too much, partially because again, it got pretty technical, but think about it. For good sound design, everything has to be perfect. How loud should dialog be compared to sound? When should this sound fade in? Should there be a crossfade here wen the player does this? Should the audio duck in certain situations to give dialog, or another sound, more attention? How should ambient sound be handled? What about footsteps? All these things have to be considered, and the answers to these questions may depend on the game. They most certainly aren’t all the same. There is tons and tons to think about.

That brings me to another presentation I attended, this one specifically talking about the audio in The Last of Us, a game that has won so many awards, and won even more during the conference. Anyway, the things they did for that game were astounding. They built a system that changes the state of a sound if it is being blocked by a wall or a door, for instance. There are sounds in the game that you only ever hear through walls, and they’re fully realized sounds, yet this system muffles them in the perfect way to make it actually sound as though a wall is blocking it.

But that’s only the beginning. Get this. They were aware that a lot of the storytelling in the game would not occur through cutscenes, but rather through conversations between the characters during gameplay. However, it’s very possible for you to get really far ahead of your companions, maybe putting a couple walls between you and them. This would of course make it difficult for you to hear their dialog. They considered making all conversations mono and perfectly clear regardless of position, but decided that took too much away from the emersiveness of the game. So what they did instead is build an audio enhancer that detects when a character is a certain distance away, and enhances their dialog just enough so, when the game is actually being played, dialog can only be so muffled, or sound so far away. Regardless, you should still be able to hear it just fine. And trust me, they demonstrated what it would be like if the enaancer was off, and it’s just about impossible to make out words. Definitely a great decision on their part.

And here’s another tweek you may not know about in The Last of Us. The entire mix is dynamically modified if the game switches to a combat state. If you attack someone or they attack you, a couple things happen. First, all enemy audio is enaanced so you’ll be able to hear them regardless of distance while still factoring in positioning. Second, the object detection of the audio is dimmed down a bit so you can hear enemies no matter where they are so long as you remain in that combat state. As soon as you’re out of it, everything goes back to the way it was, readying you for the next jumpscare.

Now let’s talk a bit about music. I went to several music-related talks, one of which was both music and audio, but all were interesting. A presentation from Austin Wintory showcaded his process for developing music for an RTS game. We heard all the different versions of the game’s themes that he came up with, the things that ultimately didn’t work, and the final product. It was amazing to get a peak inside the head of a composer that good.

However, it gets crazier. There was a presentation on the new Killer Instinct, which was the one that was both on audio, and on music. Firstly, I got to hear the original Killer Instinct announcer voice completely unmodified, which was neat. But where this presettation really shined was when they talked about their dynamic music system. In fact I was ashamed of myself afterward for not noticing these things earlier. For example, though every stage has its own music track associated with it, the track is not played from beginning to end. It is actually broken into several 1 or 2 bar pieces that are played at random throughout the fight, except under specific circumstances.

Then there’s the intros. Each character has their own intro animation that you see and hear before the fight starts. Well, all these intros have different lengths. However, so long as you don’t skip them, the moment each intro plays is actually timed to fit the music, so the intro for the song itself can be the same length regardless. Just another one of those little things.

Moving on, though, things get crazier still. Each track has its own little effect noises that are mixed in when one or both players have a full instinct meter. This effect, whether it’s a voice that comes in singing a few notes in the background or an extra little bit of guitar, pans appropriately based on which side of the playing field the player whose instinct meter is full is on. If both players have a full meter, the effect just occurs on both channels.

Then, you have the ways in which the music system rewards players. For instance, you only hear the chorus of any of these songs if someone performs a combo of 15 hits or more, and even then you only hear 2 bars of it. To hear the entire chorus, you have to consistently hit combos that large and keep it going. Also, a bit of an easter egg. The music system even rewards players for figuring out there is a mssic system, and trying to break it. If both players do nothing for a set amount of time, no button presses at all, you’ll be treated to a remix of hhe original stage song for that character from KI1 worked right into the new music. As soon as you start moving again, the song will transition back to the new stuff after the current bar, so as not to break the rhythm.

Then you have the tons of little musicll stings that occur whenever there’s a combo breaker, and you have the musical ultra combos, which are done using a number system and a bunch of small notes. It’s all awesome, and makes for a really dynamic listening experience.

And yet, nothing blew my mind quite like the presentation on the music of the Tomb Raider reboot. Granted I hadn’t yet heard anyone play the game, but even if I had, I don’t know if I would’ve realized the incredible, insane complexity put into the music system. Get this, guys. There is no “main battle theme.” No particuaar music that plays when you go into combat. Every single encounter, every fight, every scene, is individually scored. Yeah, that’s already incredible. But just in case that’s not enough, the scores are dynamic. Things happen in the music depending on what you do. I’ll give you 2 examples.

The first, and perhaps the best example they gave is one where you’re wrenching an axe free of this door so you can go through it. You do this by quickly tapping a button, but as long as you’re tapping fast enough, it takes you a set amount of time to pull the axe free. The score is ready for this. A long, high-intensity note, created mostly with horns, plays while you’re pulling at the axe, and the length of this note is exactly as long as it should take you to pull it free if you’re successful. Because of this, the score continues without pause if you are, and the note blends perfectly with the next bit of music, seeming like nothing more than an obvious lead in. However, if you fail, the music is also ready for that. The long note that started playing when you grabbed the axe will fall off at the end, and more, similar notes will play instead, highlighting the urgency of the situation as you theoretically try to wrench the axe free again. All this happening in the midst of a chase scene.
Another smaller example is a stealth one. There’s a scene where you’re sneaking up on 3 guys. Music plays throughout this scene, but not much of it to allow you to get into the quiet, stealthy spirit of that moment. Well, there’s one particular piece to the music in that area that only plays if you shoot the guy in the middle first. If you don’t, you’ll never hear it, not anywhere else in the game or anything. Remember, every single scene is individually scored. Pretty impressive, huh?

There was more of course. I attended a musical demo derby featuring game composers given an unscored version of the same couple games, and asked to rescore it. That was also educational, as I got to hear composers with different takes on the same experience. I also attended a panel on the voice industry, which as far as I’m concerned just prepared me for the future. Heheh. Anyway, doing these things, attending these panels was truly a wonderful and enlightening experience.I learned and experienced so much, and because of them, there are games I won’t look at the same way ever again. I hope you’ve enjoyed reading of my experiences, and do feel free to comment below. More is coming!