Rain and Terrain: How Animal Crossing’s Audio Design Helps Even the Totally Blind

Note: This post is a reblog of a post I wrote for the Audiokinetic Blog, and was published on June 17, 2020.

Animal Crossing is playable by the totally blind! These words have shocked many people to whom they have been spoken, and they even shocked me the first time I heard them. I couldn’t fathom how the blind could play this kind of game. I had certainly never played anything like it before. Nevertheless, now that I had heard it was possible, I had to find out how, and of course I had to try it myself. It shouldn’t come as much of a surprise to you, dear reader, that the primary reason Animal Crossing New Horizons is playable is its audio design. What may surprise you, however, is just how good it actually is. Sit back, relax, and I shall explain it all to you.

For any game the blind can play, we need a lot of auditory information. We need to know where things are so we can find them, or in some cases just use them as landmarks. Animal Crossing does this very, very well. Not only are there different sounds for each type of terrain, making it possible for us to tell the difference between the beach, or grassy areas, or the plaza, but there are plenty of environmental queues. Get this. If there is any wind blowing in Animal Crossing New Horizons, you can hear it blowing through the trees, but not as just another ambient effect. No, you can hear the wind blowing through the trees at their exact positions on your island. This is the first game I can think of where we can actually use the wind itself to discover where individual trees are located. It’s phenomenal!

One of the greatest and most helpful things, though, is your ability to make your own audio landmarks by placing certain items in important areas. My house, for instance, is on the edge of the south beach, allowing me to use the beach as a baseline. I then placed a permanently lit campfire just to the left of my house, ensuring that if I walk along the south beach, I will hear that campfire and find my house every time. To make life even easier, I later placed the shop on the other side of that campfire, so all I have to do is step outside my house, walk around to the left so the campfire is on my right, and there’s the shop door. A third example is my placement of my island’s museum, which is right next to a waterfall on the north side of my river. Why? Because the waterfall is very loud and easy to detect, making it the perfect natural landmark.

Don’t worry, though, I’m not even close to being done. There’s plenty more audio goodness to be discovered in this game. I’ve already talked about how we know where trees are, but I haven’t covered the act of collecting from those trees, or anything else really. There isn’t much to say except that when an item drops to the ground, it makes a sound as it lands. These sounds are also perfectly positioned. So when I swing my axe at a tree, I can tell sometimes that the fruit from that tree landed on the same side of the tree as I’m on, whereas the wood landed on the opposite side. The audio positioning is, truly, that good, and keep in mind this game doesn’t even use surround sound.

Speaking of collection, though, what about those balloons? What about bugs, fish, and fossils? Well, balloons are the easiest. In fact, the deaf/hard of hearing community has lodged a complaint that balloons can be heard long before they can be seen, which is true. As a balloon floats through the air, the sound it makes is very, very distinct. All we need to do is move toward that sound so it becomes loud, then center it in our headphones, and fire that slingshot. I have taken down several balloons in a single shot because they’re so easily trackable. Sometimes it will take several attempts, but usually I’m just not close enough, or the balloon floated behind me. I always get it eventually.

Bugs are a bit trickier. There are bugs that make noise, such as the honeybee, long locust and loudest of all, the mole cricket, but many, many of the bugs make no noise at all. So yes, this is a bit more difficult a problem to solve. What we’ve come up with isn’t technically audio related, but I’ll mention it here for the curious. The strategy for a blind person is to section off an area of your island where you plant many flowers, and leave many tree stumps. Bugs will often come to these places, making it technically possible to catch a couple if you go to that area and start swinging.

Now let’s talk fish. On the surface, (that’s a water pun), fish wouldn’t be easy for a blind person to catch. The instructions for catching fish specifically say that you have to cast your line in front of the shadow of a fish. And no, fish do not make sound when they’re in the water. However, if you use fish bate, it actually summons a fish right to you, or very very near you. And as it happens, the clams you use to make fish bate do make sounds when they’re underneath the sand. The tiny little water spurts they give off are all the indication we need to find them and dig them up. Again, sometimes this takes a few tries, as we don’t always hit the correct spot exactly, but it is certainly doable.

There are a few interesting audio notes about fish, though. While they don’t make sound while swimming, a fish interested in your line does make a series of sounds. One for nudging your bobber, another much larger sound for when they latch on and get hooked, and finally the sound of you reeling them in. This last one is the most interesting, though, and here’s why. It is actually different depending on the size of the fish you’ve hooked. Little splashes for little fish, big splashes for, well, big fish. So, to fish, we throw out our fish bate, then cast our line a few times around us until we hear the interest sound. We then wait for the hooked sound, and one press of A pulls the fish right in. It’s unfortunate that we basically can’t fish without bate, but at least we can do it at all.
So let’s talk about fossils. Unfortunately, fossils are very, very difficult for the blind to dig up, as the appropriate dig spots make no sound whatsoever. Therefore, I don’t have much to say about them, at least when it comes to digging for them on your own. However, the next topic will cover a solution for even this problem.

Next up in this compilation of auditory awesomeness, interacting with friends online. I couldn’t ask for this to be any better without asking for things like chat narration via text to speech, which I don’t think is very likely. At any rate, interacting with friends online is as easy for your friends as it is for you. You can hear every single thing anyone on your island does. Not omnipotently, of course, but the distance from which you can hear things is very long. You might think this isn’t a big deal, but many many games, even online ones, wouldn’t play sound for things your friends do, even if they’re near you, unless they directly affect you or your enemies. Firing a weapon, for instance, but not playing a sound for equipping a new one. Well, in Animal Crossing New Horizons, you can hear… everything! If your online buddy pulls out a different tool, you can hear that. If they shake a tree and pick up the branch and/or the fruit from it, you can hear all of that. Everything they can do, you can hear them doing it. This is especially helpful if your friend is helping you with the previously-mentioned fossil collection. Since fossils don’t make noise, just have a friend hop on, find a fossil spot, stand near it, and clap. You’ll be able to hear it from far away, track it, and then dig up your fossil. Done and done.

I’ve also used this trick with friends that have agreed to give me things. They’ll stand near me and drop items. I can hear them drop so I go pick them up, but if for whatever reason I miss one, they’ll stand at the spot where they dropped it and perform any of the many emotes in the game, all of which also have unique sounds, by the way. It’s a system that allows for true online companionship, even if both players are blind, except of course for the fossil example in which case one must be sighted. But if I want to hop over to a blind friend’s island and drop him some fruit, that’s no problem at all.

Now we move into our final segment, which I will lovingly refer to as the crazy segment. Consider all the things I’ve already told you. Consider how awesome those things are, and now prepare yourselves to be amazed all over again. Did you know that, if I listen to someone walk in Animal Crossing: New Horizons, I can tell whether or not they’ve changed their shoes from the default? Yeah. All those terrain sounds exist, but they change depending on what kind of shoes you’re wearing. Nope, I’m not kidding at all. I don’t have these sounds memorized personally, so I couldn’t currently tell you which shoe was which, but I have definitely witnessed this happening for myself. It’s just stunning.

And speaking of stunning, I’m still not done. Let’s talk about rain. When it’s raining in Animal Crossing: New Horizons, everything changes. Yes, everything. The sound of wind running through trees becomes different as the rain also pelts them. The sound of things dropping to the ground changes as the ground is soggier, your footsteps change for the same reason, (and trust me stepping off a plane onto an island where it’s raining and walking across a wet dock is very noticeable), and that’s still not all! If you’re inside a tent or building, you can hear rain hitting the roof. But even better, you can hear this from the outside too, meaning that buildings and tents are technically easier to find in the rain! I feel like I need to say again that I am not kidding. This is truly amazing stuff.

So there you have it. This doesn’t answer all of a sighted person’s questions about Animal Crossing, (I didn’t go into detail on how we use optical character recognition for instance), but it does answer all the audio-centric questions. Next time you visit your New Horizons island, maybe close your eyes for a second and just listen. See what you can figure out. Animal Crossing has outdone the audio design of many, many triple A games out there, and I want everyone to hear it.

is Fortnite Leading the Way Into the Future?

Today, February 2, 2019, an event took place in Fortnite. This time, though, it wasn’t one of their typical world and story events. This time it was, no joke, a live concert. But to me, it was so much more. What I witnessed when I witnessed this event was incredible potential. So incredible that I had to write about it, and believe me I never expected to write about Fortnite.

The concert was incredible. About 20 minutes beforehand, a mode called Showtime appeared in the game, wherein the circle stayed in Pleasant Park, the location of the concert stage, and gave everyone a countdown timer to get there, which was of course counting down to the concert itself. Marshmallow, the performing artist, even did a mic check beforehand. Sounds like such a small thing, but that clued me in to how epic this was going to be. Once that counter finally hit 0, everyone automatically became weaponless, and the show began.

Now you might think the show was nothing more than some piped in music, no kills, and people doing Fortnite dances, but you would be wrong. Yes, the music played, and everyone began to dance, but then it got crazier. Remember, this is all in a video game, so they can do more than just throw lasers at you. As the first beat dropped after the intro and the big pyro exploded, it bounced all the characters that were present up into the air. Later, just as the lyrics of the current song were “we can fly,” that’s exactly what happened to everyone. Suddenly everyone was flying, looking down at the spectacle beneath them. As the show went on, the DJ activated his mic, asking everyone to squad up and show off their favorite emotes. I see through this portion, as it’s highly unlikely he was watching anyone in particular in reality, since the concert was taking place in likely thousands of matches at once. Still, it’s the fact that the show was more than just music and light that I’m getting at here. I can’t even play Fortnite, but hearing the reaction of my fiancé as she experienced this event was truly an incredible moment.

That’s how we now get to the point of this blog. Now that I’ve witnessed it, I am boggled by the fact that technology like this isn’t being leveraged in tons of other ways. We are in a digital age, an internet age, and there are so many insane applications for this kind of thing! I understand that not everyone has amazing internet, but those who are playing a game that requires at least decent internet probably have the decent internet they need to play it. So how about leveraging this kind of tech to create things in MMO’s? Why must most games have precise, controlled world events that are always the same, just triggered by the developer at different times, when they could be doing just about anything? Why not use this technology to make the player feel even more a part of the game world than they already do?

Come on, devs! Wanna hype some big space military operation? Have your players head to a briefing location, and host a live briefing there like in Ernest Cline’s Armada. Live streamed dialog of some authority figure explaining what must be done, screens showing targets and operational concerns, all of it. I will no longer buy the idea that that cannot be done, because we have just witnessed it. And while you’re at it, devs, change your game world live like Fortnite does. Don’t just make a video and call it done, make actual changes. Don’t be scared, and don’t apologize for it. People might not like it, but they’ll be able to talk about how they were there for it.

Community is a powerful thing. Games recognize that to a certain extent, but how much more could emersion be increased if things like this are fully embraced? The craziest thing here is that Fortnite, of all games, appears to be the first. This game that started as such a small thing with a tacked on Battle Royale mode is now leading the way in interactive live events? Playstation Home tried to do things like this, true, but never took it to the level Fortnite has, and also never actually functioned all that well either. It’s incredible, and I hope people are taking notes. I’m certainly thinking other performers could find a way to use this, but like my previous Armada example, that’s really only the beginning here. Let me know what you guys think of all this, and where you think we’re going in the future. As always, thanks for reading, and continue to be awesome!

Pressing Buttons: Quicktime Events and Blind Gamers

Greetings readers! Right on the heals of my Shenmue discussion, I wanted to talk about quicktime events As a refresher, quicktime events refer to those moments when you’re watching what appears to be a cutscene, but you must suddenly press a button to achieve something. Failure to press the correct button by pressing the wrong one, or not pressing it in time, results in a failure of the attempted action, which can sometimes lead to the demise of your character. How, though, do blind people deal with these moments, and what do we think of them? That’s what we’re about to talk about.

First, it’s important to note that there are basically 2 types of quicktime events. The first is one where, regardless of how many times you retry an event, or how many playthroughs of a game you do, the button you need to press never changes. These are the ones blind people are sort of OK with, because we gamers don’t typically mind memorization. If we can memorize a quicktime sequence, that becomes the bit we feel good about when going through that section of the game.

The second type is the worst for us. Quicktime events where everything changes every time cannot be memorized, so we can only rely on, pun intended here, (blind luck) to get through those moments. My first tip to game developers who intend to put quicktime events in their game is to avoid this method. Giving us the option of memorization isn’t quite an accessibility feature, but it is a nice perk.

There is another sort of quicktime event type involving directional movement along with a button, such as pointing a cursor at the proper spot before executing your button press, but that’s another can of worms I don’t think we need to open. This would, in a way, be an even worse option than the random button presses, since we have no idea where a cursor would be in that situation. Telltale does this sometimes, and it’s so, so very agrivating.

So I’ve now given you an idea of how we feel about different types of quicktime events, but let us now approach the big question. When it comes to accessibility, if we’re actually talking about a game with accessibility features implemented, what should be done about quicktime events? My answer might surprise some of you. I’ve heard a lot from developers that the answer to blind accessibility is to remove quicktime events entirely, or make them skippable. This, I tell you now, is the wrong approach. Well OK, in my opinion it is. You’re bound to hear several different opinions on the subject, but hey, another key to accessibility is options. We love options!

Anyway, personally I believe the correct approach is to treat quicktime events like the rest of the game, and make them accessible. Don’t remove them and thus remove the challenge. Don’t make us skip them and potentially miss a great part of the story. To me, those are unacceptable options, and honestly, copouts. Make us feel the intensity of those moments like anyone else. Get some voiceover of the names of each button, or use text to speech. Apply this to the quicktime event so that the button we need to press is spoken right when or right before we need to press it. Do this, and you can even keep your randomized button quicktime events, because we’ll still be properly alerted.

If you don’t want to apply a voice to the button, apply a sound. Create a sound that is different for every potential button we might have to press, and play it at the time it is needed. We can memorize those as well. The important thing, though, is just to give us as close an experience to the one a sighted person has as possible. That’s what we want. We’re not asking for easy mode.

And that’s it, I suppose. Quicktime events are an interesting mechanic, and possibly far more elaborate than some thought, but they do not have to be bad things when it comes to accessibility. I guess that’s my point. With the first type of QTE, the one where buttons are never different, (Shenmue is an example of this), we can deal. Make them accessible, and we will love them. Thanks all for reading, and as always, continue to be awesome!

Microsoft E32016 Press Conference Quick Thoughts

Greetings again!
Well, turns out I do have something to say. The Microsoft Press Conference is now over, and I have mixed feelings about it. On one hand, the Play Anywhere idea is a good one. Moving forward, ordering a game for Xbox One or Windows 10 gives you the game on the other platform, and saves automatically transfer. The work they’re doing with cross platform play, being able to play not just cross platform on PC and Xbox, but on mobile devices as well, is also a great idea. Those are the things I approve of, along with a few of the games that sounded neat. The Happy Few, and Scalebond were especially interesting to listen to.
However, I was right. New hardware has been announced. The Xbox One S is a slimmer Xbox One, which is fine. New players can come in and get that one. Great. My real problem is the other piece of new hardware they announced. Project Scorpio. It is a new Xbox Console, with tons more power, capable of delivering high-fidelity Virtual Reality gaming without sacrificing performance or graphical quality, and so on. Here’s the thing that I don’t think Microsoft wants you to consider.
When they made the announcement, they tried to soften the blow by assuring you that all previous games and accessories would of course work with this thing. That’s all well and good, but let’s talk about moving forward. Starting basically now, new Xbox One games will be made with this new console in mind. “But,” you say, “We’ve also been reassured that new games will continue to work on the original Xbox One as well!” Yes, I’m sure they will, but think about this. Developers want to keep moving forward. They want to provide you with the best experiences they can. So they are absolutely going to focus on harnessing the power of this new system, and games played on a regular Xbox One are going to suffer for it. No matter what they say about all games working everywhere, there will come a time, probably sooner than later, where you will simply need this new system, or be forced to deal with what appears to be a broken game.
Now, this isn’t just me ranting at Microsoft. I don’t agree with the fact that it’s rumored Sony is doing this same thing. Gaming is already an expensive hobby. We as console gamers should not be forced to buy new hardware when it is not a new system, just a hardware and power upgrade. Now, PC gaming is a more expensive hobby, and I get that they already do this. You have to if you’re going to keep up with the constantly improving PC world. I just do not agree with the idea of consoles going that way. Many console gamers buy consoles so they don’t have to keep up with PC’s. I’m not a fan, and I’m definitely not sold. This is only my opinion, but there it is.
I’ll leave this post at that. Who knows, I might be back once the Sony Press Conference is over. We shall see, folks.

Video Game Console Accessibility for the Blind takes ONE Step Forward

Ladies and gentlemen, today, Thursday, November 12, marks another step forward in the accessibility of home video game consoles. Today, Text to Speech arrived on Xbox One! The most readily apparent fact is that now, both current generation consoles have text to speech capability of some sort, but this post is here to specifically discuss the feature on Xbox One. And so, without further adue, let’s begin.
Activating Narrator, (the name of the text to speech software on the Xbox One, and by no coincidence, the name used for Microsoft’s text to speech software within Windows), is super easy. If you have a Kinect, simply say Xbox, Turn on Narrator. Within seconds, you’ll here the words “Starting Narrator.” But hey, even if you don’t have a Kinect, activating Narrator is as easy as holding the Xbox button, then pressing the menu button. Awesome, right?
Now, those who have a Kinect do have one particularly annoying issue to look forward to, especially if their system is new. Narrator cannot be activated until the Kinect is calibrated for sound. The reason for this is that, since Narrator will be reading your Xbox for you, it will likely say the word Xbox several times. Xbox, of course, is the word that triggers Kinect voice commands, so the Kinect needs to be able to differentiate between your voice and other background noise, and the speech coming from the TV or speaker system.
Once narrator has begun, navigating the various screens is a pretty simple afair. At the top, you have several tabs, including home, community, games, apps, One Guide, movies and TV, and music. Once you’ve selected the one you want with the left and right arrow keys on the controller, you can scroll through that tab with the up and down arrow keys. If you reach an item that Narrator calls a section button, this indicates a deeper list of items that can be moved through with the left and right arrows. For example, navigate to the games tab, and scroll to New Releases. This will be a “section button.” Move left and right to hear all the games released recently, and if you like, press A to select one.
Pressing left from the first tab on the home screen brings you to the guide window, where you can read your notifications, access your friends list, read messages, and so on. All of this appears to be accessible. From here, friends can even be invited to the game you’re playing, or you can join one that they are playing.
This kind of basic navigation will get you through most menus. It will allow you to do everything from browsing your library and launching games, to purchasing new games and content from the store on the console. Yes, there is a working web store we can already use, but now, well, just imagine! At last, we can make those impulse buys just like everyone else!
There are, in fact, very few things that don’t work. Almost the entire community tab is completely inaccessible, which is too bad but not horrible. The one and only thing I really, really didn’t like is that the Achievements screen is also utterly inaccessible. Sometimes it will read enough to let you know there are things there, but you won’t know what those things are. That, certainly, is one thing that could be fixed.
Also, it should be noted that third party apps do not function with Narrator. Skype does to some extent, but then Skype is now a Microsoft app. Other apps like Netflix, Hulu, and so on, won’t work with Narrator at all, though I could see this being technically possible in the future. We’ll see.
Aside from some lag issues and the above mentioned unreadables, there aren’t many problems with the new Narrator feature. There are things that remain untested, such as whether or not the acquisition of achievements is spoken aloud, but I can confidently say that this was a very good first attempt at Xbox One accessibility from Microsoft.
I want to make clear here that I am in no way hating on the Playstation 4. The fact that both consoles are accessible to the blind is more than just good, it’s incredible. The fact that the Xbox One’s accessibility is, for the moment, quite a bit better should not be viewed as a competitive victory, it should be viewed as a step forward for all accessible video gaming. Competition breeds results. Perhaps Sony will take note, and the next time their text to speech feature is updated, perhaps it will be a massive, game changing update. Only time will tell, and right now, the future looks pretty bright for the blind.
One last thing. As long as it’s available, you can check out my audio demonstration of the Xbox One’s Narrator feature using this link. I hope you’ve enjoyed this post, and found it informative, but regardless, keep on gaming, and keep shouting out your desire for accessibility in both games and consoles alike!