What are replays?

In a skill-based competitive game like SpyParty (or Starcraft or Counter-Strike or Go or Poker…), watching and studying games is an important part of learning and improving your skills. Right now, the only way to watch a game of SpyParty that you’re not participating in as a player is to watch a stream or a video. I stream on the SpyParty twitch.tv channel, and I post videos to the SpyParty YouTube channel, and lots of other beta testers stream and post videos too. You can catch a lot of streams by signing up for the SpyParty Streams Notifier, and you can check out all the Let’s Plays by searching YouTube, but if somebody didn’t capture their game, there’s no way to go back and study it and it’s gone forever. Even if somebody did capture the game to video, if their camera angle wasn’t right, you might miss the thing you want to see.

Replays are the solution to these problems. A replay is a recording of the game, but it’s a recording of the stream of animation commands and events and movements instead of just a stream of images like a video, so you can move around in the replay while it’s playing, freeze it and look at the layout of the party, where the Sniper’s laser is relative to the Spy, and even rewind and study a section from different camera angles. Once replays are in, they’re going to revolutionize the study of elite SpyParty games; the plan is to capture a replay of every game ever played, and add them to a database that can be queried by any beta tester to study any game. About to play in a tournament against kcmmmmm? Study the last 100 of his games against other high level players and try to get a feel for his play style. Heck, you can even sort of play the games from the Sniper’s point-of-view, trying to find the Spy, although since the Spy won’t repond to the laser sight it’s not going to be a real test. I don’t know whether this is going to benefit Spies or Snipers more,¹ but it’s definitely going to raise the level of play across the board.

After I get replays working, the same technology will be used to implement “spectation”, which will allow you to log onto a game in-progress and watch it live, which is like watching a stream, but you can move the camera and see the action from either side, or even from a different camera position. At that point, if you join the lobby and everybody else is playing, you can just go spectate until somebody else joins to play. This will be huge for streaming, since it will allow people to cast other games, and commentate on the play! I’ll even implement mini-tournaments and simple betting within a spectation match, like some of the Starcraft mods do.

First, though, will come raw replays saved locally. You’ll be able to review your games, but to see somebody else’s you’ll have to get the replay file from them. The files should be pretty small, like hopefully one or two megabytes. After the bugs are worked out for that, I’ll get the replay database server up and running, and then spectation.

I really excited about replays, and I think they’ll increase the depth of the meta-game, and help the community share and discuss strategies.

Video Previews

Here’s a video I recorded from last night’s stream for drawnonward‘s 10,000th game (!). I gave a short preview of the current state of the replay system, which is still buggy but the hard part (rewind) is mostly working:

Click here to view the embedded video.

If you’d like to see videos of the terrible things I did to the partygoers as I was getting rewind working, you can check out these two videos:

Click here to view the embedded video.

Click here to view the embedded video.

1. my hunch is Snipers will benefit a bit more, but I don’t know

These builds were a long time coming because I had to re-do the entire inside of the game in preparation for spectation and replays, which aren’t quite done yet, but are getting closer.  However, I put in a bunch of really important bug fixes, the cool looking “menu party”, the oft-requested “back button” on the game setup screens, and Practice Mode is now handled properly so it can be used for bug repros much more reliably.  Also, you can see some nascent spectation action developing with the addition of hitting <tab> to switch between Spy and Sniper views in Practice Mode.

Oh, and of course:

Santalanterns! Inspiration courtesy of zerotka, as per usual with our holiday Easter Eggs!

Click here to view the embedded video.

1. If you really want to see that, here is your link.

What is This Bug?

The bug that was discovered allows your laser to pass through certain sections of certain characters. This makes it seem like you miss some of your shots even though you really shouldn’t. This bug is easy to reproduce as well. All you have to do is aim at these imaginary holes and that’s it. For example, you can aim your laser at the stomach region of Alphonse “Snaps” Mcgee, or at his hat: 

Since there was a good repro, checker was able to find and fix it pretty quickly. I asked him if he could give us a more technical side to this bug and he agreed. Here is his response:

Okay, so the characters are made out of triangles. For the old characters, there are about 1.5k tris, and for the new ones there are about 11k tris.  It’s slow to do a raycast against all these triangles, so you do check on the overall bounding box of the character first, since that’s much faster.  If your bounding box is conservative, then that means all the triangles are inside it, so if the ray doesn’t hit the bounding box, you don’t have to test all the triangles.  But, since that test assumes all the triangles are inside the bounding box (hence the word “bounding”), if that’s not actually true, you won’t test the triangles at all. 

This is what the bounding box looked like before the fix:

This is what it looks like after the fix:

I looked into this bug so I could gain my own understanding of it. I noticed the holes seemed to disappear during animations and while holding a drink. I asked checker if he could give some insight into what’s happening. Here is what he had to say:

With animated characters, you really don’t want to build the bounding box from the triangles as they move around because that’s slow, and you often don’t have access to the posed triangles because that happens on the video card, so you build the bounding box from the bones themselves, and then expand it a bit (20% in this case) to get all the triangles. Well, with the old skeleton, it’s almost flat on the xz plane in the rest pose, so the expansion doesn’t do much.  When you’re holding a drink, your arm is forward, and that makes the bounding box actually bound the whole body! 

These are images with the old bounding box built from the bones:

So what was the fix?

I looked at the code, and with the current (read: slow) animation code I decided it wouldn’t actually be any slower if I just computed the triangle-accurate bounding box.  Once I optimize the animation code, I’ll have to come up with a better solution, potentially having bounding boxes for each bone authored in Maya, or something smarter, I’m not sure yet.

How was it Discovered?

krazycaley took on challenger virifaux in Spy on the Hill week 7. It was during one of these games the Sniper appeared to shoot through one of the characters to both the stream viewers and to himself.  Interestingly enough, he did!

You can hear  krazycaley exclaim “Noooo! Miss!” right after the shot. He knew who virifaux was and virifaux knew he was dead to rights. Yet somehow he was still alive. This was enough to pique the interest of virifaux and he chose to pursue it more deeply a day later. Once he discovered a reliable way to reproduce the bug, he made a quick video showcasing it in action, marking the beginning of the end for this bug.

Streaming was a big help in squashing this bug.  SpyParty doesn’t support replays yet, which makes bug hunting difficult sometimes. Streaming is probably the next best thing. If pictures are worth a thousand words then videos are probably worth a couple million. In this case, the video gave clues to virifaux for where to start looking. He knew exactly what it looked like when the bug occured. He could watch it over and over until he formulated his hypothesis. Then he could go into practice mode and test it out. 

Streaming and videos give us great insight into reproducing pesky bugs. Streaming may not always be the main tool in bug hunting, but streams and videos are almost always helpful for gathering more evidence and figuring out a clean repro.   This was the case with another bug’s story. I don’t think streaming was the main reason this bug was caught and squashed but it certainly played a role. 

Why is This Interesting?

I find it very interesting the bug was discovered only now. When first looking over the new bug report, checker made a post stating he remembers this happened a long time ago and that it was super rare, so we know this bug has been in the game for quite some time.

I am going to throw some numbers your way to help give an idea of the scope of this thing. There are currently over 11,000 people who have registered with the beta. The top 100 players have a combined total of 185,520 games played. It’s safe to say there have been a lot of games played across the entire community. This isn’t even counting the number of people who have played at conventions. In all of the games that have been played, this bug is just now being talked about and reproduced.

That’s not the only thing about this particular bug that makes it so interesting to me. Not only have there been tons of games in which it could have been discovered, the steps needed to reproduce the bug are really simple. When you combine these two characteristics together, it’s crazy how long this bug has lasted! However, thinking about it more deeply, maybe it’s not so crazy after all…

There are a few things that need to align correctly in order for this to be spotted. The main thing is looking at the character from the side. Next the character can’t be doing the vast majority of animations nor can he or she have a drink in their hands. Finally, the sniper has to aim in the correct spots. Perhaps you also need to be streaming for Spy on the Hill week 7.

Whatever the case, the bug is nearing the end of its lifetime since it’s fixed on checker’s local copy, but until the patch goes live you might be able to utilize this bug in a game mode. As beta tester virifaux said, “Now that we have a reliable reproduction, we’ll have skill shots where we shoot through McGee to hit the spy.” Bugs can be fun to play with so get in your skill shots before it’s too late!

This bug also has an interesting story, because it turned out to have a very subtle cause, one that manifested itself intermittently in ways that looked almost random, and for a long time there was no “repro case”.  Getting a repro on a bug is the key to fixing it, as I discuss in the How to Report Bugs the SpyParty Way post.  It can make the difference between a 10 minute fix and a 10 day fix…or never managing to find and fix it. 

*shudder*

But let’s start at the beginning…

I first noticed the bug long before I’d invited people into the beta, but it was so rare that I didn’t prioritize finding it, and in fact I would forget about it for stretches of time.  Yes, I make notes of bugs I see, but I’ve got so much high priority stuff to do right now that I don’t go back and look at that list very often…the important bugs get fixed immediately, but a bug like this can stay in for a long time.

So, what’s the bug?  Well, let’s go to the videotape:

Click here to view the embedded video.

This video was from a Spy Commentary game played between buxx and dieffenbachj, two early beta testers who were some of the first to upload gameplay videos.

It turns out, in addition to just being plain awesome for games overall, the rise of videos and streams is also an amazing resource for bug finding and fixing!  The more people record their games, the more they’ll be able to point to a video of exactly what went wrong so the developers can see it almost first-hand.  The bane of a developer’s life is a bug report that says, “the game broke” with no other description.  You know something’s probably wrong, but it’s basically useless for finding a problem.  With a video, you can usually see exactly what’s going on, so that problem is eliminated, or at least massively reduced.

So, as you can see in that video, the Spy is facing the wrong way on the floor pad.  The Spy should be facing the pedestal with the statue on it in that position, but in this case the Spy is turned 90 degrees.

I watched buxx‘s videos when he posted them¹ and I noticed this, and so I posted a bug in the Bugs Forum on the private beta website myself on May 6th, 2012.

Obviously, trying to repro it by replaying the steps in that video was fruitless.

Next up, bishop caught it with a screenshot and posted on May 13th, 2012:

It’s a perfect shot, but his next post is “I haven’t had much luck on the repro.”

ardonite chimes in the next day:

I got the rotation bug once at a statue.

I think I was rriiiiight in the bounding box. So maybe if it’s on a border pixel of the box then it glitches out?

Edit: no, hypothesis incorrect.

That’s how it goes: make a hypothesis, check it, repeat.

More shots every couple months over the summer from bishop and r7stuart:

Still no repro.

This entire time—in fact since I first saw it myself pre-beta—I’ve been trying to resist thinking it’s some kind of “numerical issue” with the code that handles the facing angle.  Yes, angles are finicky to deal with due to wrapping, but I’ve found programmers, including myself, tend to immediately go to vague concepts like “floating point error” for anything like this.  To fight this tendency back when we were working on physical simulation code together, Casey Muratori and I developed a mantra:  “Assume it’s a bug!”  It means that instead of assuming it’s some subtle floating point error creeping in, or anything mysterious like that, it’s almost certainly just some dumb programming bug.  That mantra has never failed me.  It’s always just a plain old bug.

Onward…

In the fall of 2012, streaming SpyParty took off bigtime, and so people were recording their games more often, and we started to get more videos, this one from r7stuart in October:

Click here to view the embedded video.

And tytalus in November:

Click here to view the embedded video.

I saw this last one live on tytalus’s stream, and grimaced when it happened, but also was happy to have more data to some day find a repro, or just have a random brainwave and fix it by intuition.²

At this point, people were reporting NPCs doing it, which at least made me happy, because it meant it wasn’t a tell.  Tells and anti-tells³ are the most serious SpyParty bugs, because they undermine the delicate balance of the game, so I proritize them highest, even above crash bugs sometimes!

A couple on New Year’s Eve from jorjon:

Then two clips from streams, the first from slappydavis, who’s Seduction Target appears to do it at the bookshelf on January 6th:

Click here to view the embedded video.

And then from james1221 on January 12th during the SpyParty New Years Cup Tournament:

Click here to view the embedded video.

Both of these are different, however.  In both cases, the NPC is being blocked by another character, and instead of repathing to a new place, they just wait until the blocking character leaves.  This is both good news and bad news.  It means it’s easy to project this bug onto other bugs, it means there are other bugs,⁴ and it means all the real examples of this bug so far have involved the Spy.  I don’t mention this last part in the hopes that nobody notices.  Luckly it’s rare enough that it’s not going to be a game balance changer even if it is a tell.

Finally, kcmmmmm finds a reliable repro on February 7th, two days ago, and 8 months after the first post in the Bugs thread!  These pictures are beautiful to me:

You can stand at that position, with that camera angle, and repro the bug most tries.  He also figures out that it’s very camera angle dependent, which is another clue, but once I could repro it locally, its remaining time on this earth was measured in minutes.

I had some trouble reproing it reliably here, including a couple wild-goose chases where I thought it wouldn’t repro with the debugger running⁵ or in my debugging modes,⁶ but in the end I got a case where I could catch it in the debugger, and I looked at the source, and there it was, suspiciously rotten code.

It was an old check from when I used to support click-to-move, as opposed to direct-control of the spy.  There was a case in the code that would check if you were clicking on the bookshelf itself, rather than the floor pad in front of the bookshelf, and it would helpfully direct you to the floor pad.  The position part of this got taken out long ago (I think), but the angle part remained in, so when the Spy stopped moving, if the mouse was over the bookshelf that code would return the angle for facing the bookshelf.

Wait, you say, there’s no mouse cursor in Spy mode?  Ah, yes there is, it’s just hidden and forced into the middle of the screen.  So, most of the time it hits your back, but sometimes, if you’re turning or leaning down or whatever when you stop, it’ll miss you and hit what’s behind you, and if it hits a bookshelf on the frame when you stop, you get the wrong facing angle.

Now, with this knowledge, go back and watch the videos and look a the pictures above.  Always a nearby guilty bookcase, isn’t there? 

But wait, you say again, what about the very first video, the green bookshelf is nowhere near the middle of the screen!  Ah, but buxx uses a controller,⁷ and the mouse gets hidden but doesn’t get centered if you’re using a controller!  It probably should, but it doesn’t.  So, buxx‘s hidden mouse pointer is probably off to the right of the window, over the green bookshelf, until he moves one pedestal pad to the left, and then the mouse pointer is no longer on the bookshelf, and he faces the right way!

Awesome, all the cases explained, and the bug was trivial to fix!

Okay, there’s actually one more case in the Bugs thread I didn’t post here, because it’s funny enough that I’m going to make an entire post about it soon.

So, just remember, always, Assume it’s a bug!

1. …because they’re great for learning, the only thing better than video is commentated video!
2. This happens in programming a lot, but you don’t want to count on it if you don’t have to.
3. Where the NPCs can do something the Spy can’t.
4. duh
5. Sometimes this happens with an uninitialized variable, since the debugger initializes most memory to zero for you.
6. Could mean some debug code was correcting for the bug?
7. You can see the action UI in the video uses the controller icons! No detail is too small to matter in a bug!