Adventuring Blind: How I believe Point and Click Adventure Games Could All Become Accessible

Back in the day, perhaps about the mid-nineties, if you said you wer playing an adventure game, as long as the word text wasn’t in front of it, you were usually talking about just one thing. You were talking about the then popular style of game now generally referred to as Point and Click. There are bunches and bunches of these games, and not all old either. A few developers, perhaps most noteably Telltale, still make games like this, and some of them obtain a uge amount of popularity. But what is the point, you ask? Why does all this matter to the blind gaming community? Well, I’ve thought about this or a while, and I genuinely believe that pretty much every single point and click style game could be made accessible. Best yet, I think they could all use the same basic interface. So I’ve decided to blog about it here as something to mull over, and perhaps, just perhaps, something we might want to pursue.

Here’s the dream. One program that you launch in conjunction with whatever point and click game you want to play. The program detects what game it is, and loads up the necessary files it’s going to need. This program, you see, would act as a sort of overlay to the games themselves. It would probably have to be constantly updated, or alternatively a site could be created where woe would download the necessary addon packs for the games they wanted to play, but I really think it could work.

When the game and the overlay program are loaded, the overlay begins constantly monitoring the state of the game, presenting you with lists of options based on the context of what is going on. At first, this would be the basic new game, load game and so on, but wen you actually start the game, this overlay would then keep track of the room you’re in. This is where it gets fun.

What I see in my head are a couple of combo boxes, and maybe a few buttons. The overlay would use standard windows controls so as to easily be read by a screenreader. When you started your game, there would be multiple lists on screen. One would be the list of known objects in the room, another would be the list of known exits from the room, and a third would be your inventory, though perhaps to save on clutter, there could just be a button that pulls that particular list up. All of these lists would have to be allowed to change overtime, as quite often, both objects and exits are hidden until you perform certain actions, and you’re always gaining and losing inventory items. The inventory list in particular would likely have to allow for small submenus so items could be examined or combined and so on. Come to think of it, room objects should work the same way.

But here’s the thing. For the most part, (the rest I’ll get to shortly), that’s basically it. As I said, these point and click style games all worked basically the same way. You enter a room, pick up any objects in that room or solve any puzzles there, and you move on with the story while you do it. Of course, there is hidden depth here, which I’ll get to along with the reasons there would have to be just a little more with each game.

First, despite the simplistic interface, one must consider how this would have to work. The configuration files, or packs, or whatever you want to call them, would likely have to be created by the sighted, so we can’t really do this alone. Essentially, though, what you’re looking at is this. The sighted person would first reveal everything in a given room tat they could, then record their individual mouse positions right down to the pixel. Then, a name or short description, (small furry creature tensed to pounce), would have to be added. This is what the player would see as they browsed through the object list. There would be a variable that would tell the overlay if the object was hidden at any given time, so it wouldn’t appear in the list initially, but every time an object was interacted with, the game could recheck the room for interactable objects, (usually indicated by a change in the mouse pointer itself), and cross reference that with the positions marked by the configure to determine what object had been revealed. Inventory item lists may have to be storeed locally to work properly, but maybe not. That’s one point I’m not clear on.

So things start to seem a little more complicated, but here is where we come to another problem. It’s one that I feel could be resolved with a little work, but it is something that should be addressed. Some point and click games like to break format for short periods of time. The Walking Dead from Telltale has several examples of this with its action sequences, and a funny little game called the Book of Unwritten Tales has this as well with things like potion mixing. So there’s good news and bad news about this. The good news is the very fact that these things are very segmented. They’re a brief break from the format, which is then returned to. This would make them easily detectable. The bad news is there’s no universal way these parts of the games work, so they would have to be coded for accessibility individually. Still, I think this could be done for each game as a sort of subroutine of the overlay. Additional functions the overlay executes when one of these events occurs. This would potentially save on the developer having to recode anything at all, (unless they were the ones who helped with this of course), and what coding anyone else would have to do would be relatively minimal. Just enough that the part of the game in question could be got through.

Now of course I know that’s not everything. Surely there would be other little nuances to consider, such as how exactly combining of objects is handled, what is remembered within the program and what isn’t, and so on. And maybe, just maybe, all this is incredibly stupid. I’ve done a little programming, and I think I know enough about it to at least understand what would need to be done, even if I also don’t know enough to actually pull it off myself. But hey, I could be wrong. Still, as I said, this is something to mull over. A nice little dream that, if ever achieved, wouldn’t just make one or two games accessible to us, but potentially hundreds. Pretty cool, huh?


  1. Hi Brandon,

    I found your website through that Poloygon article you linked to in your last post which showed up in my Twitter feed. I haven’t had a proper look around yet but looking forward to it. I too found the article very well written and it covered a lot of ground (I think). I did some games accessibility research for my dissertation and, while not my area of focus, audio games and game accessibility in general was a big inspiration.

    That was three years ago – how time flies – and I’d love to say I’ve really made full use of my degree but I really haven’t. Well, at least not yet;-)

    I discovered point and click adventure games while I was at Uni and I’m an unproductive member of the Adventure Game Studio (AGS) forums. So after reading the article I was immediately thinking how adventure games could be made more accessible (in many ways I think it would have made for a more interesting dissertation project!) so your timing couldn’t be better.

    I’m no programming whizz and I don’t know a lot about AGS in particular, but I like to mull these ideas over.

    The system your describing sounds like an external application to the game it self – a sort of universal translator – so it would need the add on files (the accessible content) in a common format.

    Someone already made a plugin for AGS that dumps all the dialogue (text) in the code files into an easy to use editor and, I believe, allows this to be exported so there is the suggestion that something similar could be created. That’s just using AGS as an example, and I’m not well versed in what makes the engine tick. Alternatively the community could transcribe details of a game manually into an agreed format which is then read by the translator/interpreter, like how YouTube videos can be captioned.

    I’m not clear on how this program would monitor the game’s state and hotspots etc if it was a standalone program itself? Without knowing much about them at all, I presume things like Screen Readers hook into defined Windows states/controls to work so I think some implementation on the part of the game’s developer would also be necessary. Again, I don’t see why this couldn’t be a plugin instead that you include with your game, but that might be moving too much away from a universal solution.

    It makes a lot of sense to think of the components of Adventure Games in terms of lists as they’re full of them. They also use verb coins or mouse modes while are just lists of options too. I think they potential is there.

    One of the points I tried to get across in my dissertation though was that rather than tacking on accessibility options, we should be building games from the ground up with accessibility in mind and exploring the creative possibilities that provides.

    I think a universal option isn’t quite one way or the other, but I would like to see Adventure Games that wear their accessibility options boldly, complementing the gameplay (and not as a gimmick) while obviously performing an important function in the process. Of course, this is all hot air and I really need to put my money where my mouth is but the dusty cogs are whirring into life again..

    1. Brandon Cole says:

      Thanks very much for your comment! It’s good to get the perspective of someone who has done a little more programming than I have. In fact, as you look around the site, you’ll actually run across every single thing I’ve ever programmed except for 1 thing, but don’t worry, none of what I’ve done is particularly impressive. As for your comment, yes indeed I do envision this accessibility utility for point and clicks to be an application separate from the game, which is why I mentioned that it would either have to be updated when a new game was added to those made accessible by it, or there would have to be some hub where folks could go download the necessary pack for the game they want to play.
      About the monitoring of the game’s state, and mouse hotspots and so on, I did indeed take that idea from currently-existing screenreaders, which I admit is why I thought all that was relatively easy to do. You can script a screenreader to grab a mouse position, then send a click to that position when a condition is met, such as the pressing of a key. More complex scripts can be created to make a list appear, then send a click to a different position based on the item selected. However, I didn’t want to restrict this program to one particular screenreader, which is why I didn’t specify screenreaders initially. I just see this working within the application.
      I am glad you at least think the idea has potential, though. It is certainly something that I would like to pursue somehow, as there are several point and click adventures I would absolutely love to play. I will say one last thing, though. I agree with you that ideally, games should be built from the ground up with accessibility in mind. However, that dream doesn’t stop me from wanting to play games that already exist, and that I know are good. And so, I am foolishly hoping for both. Can’t help it. I want to play games, and that’s that. Thanks again!

  2. Ian Hamilton says:

    Along similar lines to Daniel, I don’t think it would be feasible for an external bit of software to be able to interact with a game in that way without the game having been designed to support it in the first place.


    You may not have to rely on an external bit of software. The main way people consume old adventures like that isn’t in the form of the original software, it’s through ScummVM, which is essentially an emulator.

    So what you have with that is a replacement for the executable files for all of those games.. one single executable that runs the data files for all of them. So changes made to that executable would affect all ScummVM compatible games (there are over 100, all the classics like Monkey Island, King’s Quest etc).

    And best of all, there’s a team in place regularly working on the core code, and other developers regularly working on implementations for different platforms.

    So I’d really strongly recommend getting in touch with them. Along with AGS, they’re I think your most likely possible avenue of making this happen.

    1. Brandon Cole says:

      Hi there,
      Thanks for the recommendation. Actually they have been mentioned to me by another as well, the writer of that article in fact, and I do indeed intend to contact them this weekend. Thanks again!

  3. I’ve spent the better part of the weekend mulling these ideas over and, while I still think it would be possible to create an overlay using standard Window’s controls that could be read by a screenreader (or self voiced by the program itself), it’s still not immediately obvious to me how it could have any interaction with the game.

    I’ve played around with the free NVDA screenreader and it would seem, as suspected, that unless the game author has taken steps to make their game accessible, it can only announce the window name and buttons (minimise/maximise etc). Nothing of the game itself is read.

    Presumably because it isn’t using Window’s controls, it’s rendering its content to some sort of ‘canvas’. Admittedly i’m not sure how, for example, Direct3D, works but you get what I mean.

    At a minimum, you would need to know what room you were in in order to present the appropriate labels/lists etc for voicing. I thought of reading pixels here as last resort, but I think there’d still probably have to be some logic involved if nothing else can be read from the game – knowing what’s in your inventory etc.

    In short, you’d effectively end up writing a text-adventure form of the whole game, rather than a utility to pull that information out on the fly.

    I’ll admit this is all speculation though. I’ve read around but i’m still no expert. Maybe there is another way entirely that I don’t know about.

    That’s a promising suggestion from Ian instead though and definitely one worth exploring I feel. I’m going to take a look myself!

  4. Craig says:

    I know Jaws for example includes the ability to run OCR on the current window or screen but this is highly unreliable in games I’ve found. You could try including graphical room recognition but there may be issues with differing window sizes and resolutions here, as well as the fact that in many of the games rooms would scroll as the character moves from one side to the other. Adaptation of ScummVM is probably the best bet and is something I’d dearly love to see, I’ve many fond memories of playing the Curse of Monkey Island or Simon the Sorceror 2 before my sight loss made it impossible.

    I personally have a dream that mainstream devs could add in support for a plug in to their games which we could develop ourselves, so we have one plugin to create an audioscape of the environment and then point all the games at this one universal plugin, but I’ve no real expectation of this. Especially not with MMOs since giving a plugin access to the kind of information we’d need to navigate and interact with the world and monsters would require them to expose this functionality to plugins in general and that’s exactly the kind of feedback botting software would need.

Leave a Reply

Your email address will not be published. Required fields are marked *




This site uses Akismet to reduce spam. Learn how your comment data is processed.