It’s been a while since the last update in this series, sorry about that!  At the end of Part 3, I mentioned the SimCity launch giving me pause about my goal of testing the SpyParty lobbyserver to 1000 simultaneous robots.  Well, I got scared enough after their launch that I increased my optimization target to 2000 simultaneous robots on my old and slow server, and then I also decided to bite the bullet and upgrade the server hardware after I hit 2000 to give myself some extra headroom.  I really don’t think I’m going to hit these numbers at Open Beta launch or even for a long time after that, but I’d rather err on the conservative side and have it purr along nicely.

Since I waited so long to post this Part 4, I can’t really give a play-by-play of all the optimizations I did as they happened, so I’m going to give the general arc I followed, and then talk about some of the interesting stops along the way.

iprof, atop, oprofile, et al.

As I mentioned at the end of the last post, I’d fixed some of the huge and obvious things with the network bandwidth usage, so it was time to start profiling the CPU usage.  There are lots of different kinds of profilers, but the one I use the most is based on Sean Barrett’s iprof.  I’ve modified it a fair bit over the years,¹ but the core of the system is still the same.  It’s a runtime profiler that requires instrumenting your code into blocks, it’s efficient enough that you can leave it on all the time as long as you don’t stick a “prof block” in an inner loop, and you can generally see where you’re spending your time hierarchically.  It can draw to the screen, but I also have it output to a string, and so on the lobbyserver I can have it output to the log after a spike, and also catch a signal I send and it’ll force a prof dump.  Here’s an example:

2013/04/17-16:12:10: 85.156 ms/frame (fps: 11.74)  sort self - current frame
2013/04/17-16:12:10: zone                                                     self     hier    count
2013/04/17-16:12:10:  ProcessMessages                                      59.2910  59.2910     1.00
2013/04/17-16:12:10: +Send                                                 17.8164  18.4493  1120.69
2013/04/17-16:12:10: +ClientsUpkeepAndCloseLoop                             2.3034   2.6989   793.15
2013/04/17-16:12:10:  Log                                                   1.3559   1.3559    25.97
2013/04/17-16:12:10:  unpack_bytes                                          0.7311   0.7311    26.56
2013/04/17-16:12:10:  unpack                                                0.6674   0.6674  3551.71
2013/04/17-16:12:10: +ClientsPacketLoop                                     0.5492   3.7160   792.58
2013/04/17-16:12:10: +ClientsUpdated                                        0.4023  10.9843     0.56
2013/04/17-16:12:10: +SendQueuedClientRoomMessages                          0.2524   7.3086     1.00
2013/04/17-16:12:10:  FindClientByID                                        0.2494   0.2494   267.05
2013/04/17-16:12:10: +JournalQueuedSave                                     0.2374   0.3882     1.43
2013/04/17-16:12:10: +Tick                                                  0.2112  25.6313     1.00
2013/04/17-16:12:10:  iprof_update                                          0.2051   0.2051     1.00
2013/04/17-16:12:10: +JournalSavePrep                                       0.1560   0.5442     1.43

As you can see, it’s pretty easy to read, and you can drill down on individual blocks and see who calls them and who they call:

2013/04/17-16:12:10: 85.156 ms/frame (fps: 11.74)  sort graf - current frame
2013/04/17-16:12:10: zone                                                     self     hier    count
2013/04/17-16:12:10:     LoginReply                                         0.0006   0.0006     0.01
2013/04/17-16:12:10:     JOINING                                            0.0007   0.0008     0.03
2013/04/17-16:12:10:    +TYPE_CLIENT_GAME_ID_REQUEST_PACKET                 0.0011   0.0014     0.44
2013/04/17-16:12:10:    +NewWaitingForJoinClients                           0.0019   0.0019     0.01
2013/04/17-16:12:10:    +TYPE_CLIENT_PLAY_PACKET                            0.0086   0.0087     0.12
2013/04/17-16:12:10:    +TYPE_CLIENT_INVITE_PACKET                          0.0213   0.0225     2.37
2013/04/17-16:12:10:    +TYPE_CLIENT_IN_MATCH_PACKET                        0.0280   0.0282     0.49
2013/04/17-16:12:10:    +JournalQueuedSave                                  0.0563   0.0571     1.18
2013/04/17-16:12:10:    +RoomsChanged                                       0.0852   0.0939    15.38
2013/04/17-16:12:10:    +NewInLobbyClients                                  0.1155   0.1342    14.55
2013/04/17-16:12:10:    +ClientsUpkeepAndCloseLoop                          0.1362   0.2093   150.55
2013/04/17-16:12:10:    +TYPE_CLIENT_MESSAGE_PACKET                         0.3122   0.3170     8.87
2013/04/17-16:12:10:    +SendQueuedClientRoomMessages                       6.7122   7.0013   501.99
2013/04/17-16:12:10:    +ClientsUpdated                                    10.3361  10.5720   424.67
2013/04/17-16:12:10: -Send                                                 17.8164  18.4493  1120.69
2013/04/17-16:12:10:     unpack                                             0.6329   0.6329  3362.06

This is super useful.  The biggest downside to it is that it’s not thread-aware, but I’ve made it thread-safe via the brute force method of having it ignore all threads that aren’t the “main”.  My code is mostly single-threaded, but the threadedness increased a fair bit during these optimizations, so I hope to eventually modify iprof to be thread-aware without losing too much simplicity and performance.  However, until I make those modifications, any background thread activity will show up attributed to one of these main-thread blocks.  You can still get useful data, you just have to be aware of this.  For example, ProcessMessages in the loop above is hiding a WaitForMultipleObjectsEx call on Windows, or a call to select/epoll on POSIX, so it’s not actually taking that much active time on the main thread.

I also used oprofile, which is a nice sampling profiler on Linux that can profile per-thread using just the debug information in an application, and atop for keeping track of things happening on the machine as a whole.

Here’s a list of the stuff I ended up optimizing:

• I was originally sending out the chat messages to all clients as they came in, but I started queuing them up and sending them all out at once to reduce send calls.  Of course, once you do this, you have to make sure you don’t overflow the network packet if you have queued a lot of messages that tick, so that makes the code more complicated and harder to modify, which is a tradeoff one often has to make while optimizing, and it’s why you want to put off most optimization until you need it…although you should have a rough plan for how you’ll optimize a piece of code in the future even if you write it the dumb way first.
• I made more threads, including putting network sending and receiving on separate threads, making a separate thread for logging, and a thread for saving files to the disk.  There were already threads for talking to the database and Kerberos, for receiving network packets, and for checking for new client builds.  These are all relatively simple threads to add, because they’re all just throwing data into a queue on one thread and taking it out on another, although multithreading a program always makes it harder to understand.  I discovered a fair number of deadlock bugs in libjingle, the library I’m using for NAT traversal and some cross platform threading stuff, and I’ve fixed some of them.  I’ve veered far enough from the original libjingle code that I’m probably just going to have to put my version up as a fork, sadly.
• I timesliced the login phase for the clients.  Previously, when a client would log in, I’d process a bunch of stuff immediately, including some authentication stuff which can be somewhat time consuming.  In a load test where hundreds of clients log in to the server at the same time, this would bog down, so I now process a maximum of 20ms worth of clients each tick.  This makes some clients wait a bit longer before they’re logged in, but doesn’t result in a positive feedback loop where there’s a really long tick, so a lot of packets will have arrived while it was happening, so the next tick is really long too, etc.
• Like the player list packets, I also made the room list packets incremental, and able to span multiple network packets.  This way all the lists of players and rooms that the lobby sends to the clients can be differential and arbitrarily long, so there’s no more hard limit on the number of clients that can join the lobby.  I think there’s actually a bug in this code, but I’ve only ever seen it once, even after tens of thousands of robot sessions, so I just hope it shows up more at some point.
• I switched the POSIX networking inner loop in libjingle from select to epoll.  This was not so much an optimization as it was simply to allow more than 1024 sockets to work at all.  epoll is also a lot faster, but I’m currently kinda using it in a dumb way, so I’m not benefiting from that speed boost much yet.
• There were also a bunch of smaller traditional code optimizations, like using maps to cache lookups, using free lists to avoid some allocations, and whatnot.  Oh, and don’t forget to change the ulimit -n settings in limits.conf on Linux, so your process can actually accept a lot of connections!

As I was doing these optimizations, I would run a loadtest with a bunch of robots and profile the lobby.  I was at 500 robots at the end of Part 3, and I slowly raised the ceiling as I improved the code over the weeks:  569 robots…741 robots…789 robots, 833, 923, 942, 990, 997, 1008, 1076, 1122, 1158, 1199, 1330, 1372, 1399, 1404, 1445, 1503, 1614, 1635, 1653, 1658, 1659…

When I hit 1659 it was late one night, and so I stopped for the day.  When I resumed work and did the next couple of optimizations, I figured I’d get it to 1800 or something.  I always launched 20% or so more robots than I was hoping to support in a given test to account for internet and EC2 variation, and for plain old bugs in the clients that would sometimes manifest themselves, so this time I must have launched 2500 robots, because when I looked up from the profiles running in ssh terminals and over to my SpyParty client logged into the test lobby, I saw this:

This wasn’t supposed to happen yet.

Uh, I guess I was done optimizing?

I was actually kind of disappointed, to be honest.  I had all sorts of cool ideas for optimizations I was planning to do that I’d come up with while testing and profiling the code, and now, if I was going to follow my own plan and stop when I hit 2000 simultaneous robots, I would have to just take a bunch of notes for next time I optimized so I could pick up where I left off, and move on.  The good news is I’m pretty sure I can make the lobby almost twice as efficient if and when the time comes to do that!

Room at the Inn

If you look closely at that screenshot, you’ll see the thumb on the scrollbar for the player list is pretty small.  That’s because all 2010 players are in a single room, which is not going to work very well for a lobby full of real people.  In fact, the only reason there were 2010 players in that room was because at the time I’d limited the room size to 2010 because I didn’t want to bother teaching the robots how to use rooms.  There were actually a few hundred more robots knocking on the door but they couldn’t get in.  But, now that I’d hit my 2k target, it was time to fix that.

I immediately realized I had a problem.  Currently, when you connect to the lobby, it sends you a list of rooms, and you have to pick one to log in.  But, what if the rooms are full?  Oops, you couldn’t log in.  So, as soon as I set the room size down to something more reasonable, like 100, then the first 100 robots got in and the rest just sat there failing to join.

It seemed like there were a number of solutions to this problem, including allowing players to create new rooms before logging in, but in the end I went with the simplest and most robust solution, which is to have the lobby create a new empty room if all the current rooms are full.  The initial room is always called Headquarters, so I named these new dynamic rooms Headquarters 2 and onward.  Very creative, I know.  Somebody suggested using spy movie titles for these room names, but I figured that wouldn’t scale very well, ignoring the potential copyright issues.  If the lobby ever finds one of these dynamic rooms empty, it kills it, unless all the other rooms are full.  I also have the lobby automatically put you in a now-guaranteed-to-exist-non-full-room if you log in and try to join a full room, even if it wasn’t full when you clicked on it, so this eliminated a login race condition too, which is always a good sign.

This last bit also made it so I didn’t need to make the loadtesting robots know very much about rooms:  they always try to join Headquarters and if they don’t end up there, oh well.  As they join, they kind of spill over into the latest dynamic room until it fills up, and then they continue to the next, kind of like filling up an ice tray with water from one end.  I should probably make them test the actual room features by creating and changing rooms and whatnot, but the single giant 2010 player room was a way more intense loadtest than having 20 rooms with 100 players in each due to the chat broadcasting.

I don’t know if 100 is the right limit for room populations.  100 would still be way too many people to have in a single reasonable conversation, but I didn’t want to put too low of a limit on the size before I have tested things with humans instead of just robots.

The Client

There’s this annoying thing that happens when you’re testing computer code, and it’s that you encounter problems and bugs not only in the code you’re trying to test, but also in your test code.  This was no different.  I was constantly fixing various bugs in the robots that would keep them from all connecting correctly, and I even made sure some of the optimizations helped the client side so I could run more robots on a given EC2 server.  Plus, just making sure the robots keep trying to connect and login was important, because if there was a timeout due to an initial burst, you want them to try again automatically after it dies down, rather than just sitting there not doing anything.

As I said in Part 1, I started out running about 50 robots on each m1.small EC2 instance.  That didn’t scale, for some reason I’m still trying to figure out.  That worked okay with a low number of instances, but as I increased the number of instances, I had to lower the number of robots on each instance, eventually to around 20 per m1.small.  An AWS account starts with only being able to start 20 instances, so I did a total of two instance limit requests to Amazon, first to 100 and then to 300.  It’s scary to have 300 instances running…even though m1.small instances are only 6¢ an hour each, that’s still $18 an hour when there are 300 of them running, and Amazon rounds up to the hour, so if you miss shutting them down by a minute you just lost a large pizza!  It looks like Google’s new Compute Engine thing is about twice as expensive for their somewhat similar low end machine (ignoring performance differences), but charges in 1 minute increments after the first 10, which might be cheaper for this very transient use-case.

I seem to remember reading somewhere that Amazon allocates instances for the same account to the same physical machine if possible, which might explain this scaling problem, since it means I was probably maxing out a given piece of server hardware with too many instances bursting at the same time.  It’s hard to tell if this is the case, and I need to do more testing before saying for sure.  A commenter said there might be a packets-per-second limitation in EC2, as well, but I haven’t verified that.  Once I’ve tried a few different things, I’ll do a long technical post on chrishecker.com about EC2, linode, and my dedicated host machine, comparing the different results I got.

Finally, I had to do some optimization on the SpyParty game client when the numbers started getting high.  I went a little nuts with the chat system early on and it has completion on all commands, room names, and player names, but the code that builds the completion tree was calling the memory allocator 35k times per update when the numbers of players got high, so I had to remove some of the stupid in that code as well.

The New Server

With all that done, and 2010 robots running on the old server, I haggled with my hosting provider and started renting a newer and much faster server.  I use SoftLayer for dedicated hosting, and have for years.² My old server was a Pentium 4 with a single hyperthreaded core, 1GB ram, and a 100Mbps uplink, and the new server is a Xeon 3460 with four hyperthreaded cores, 4GB ram, and 1Gbps uplink, so it’s slightly more expensive but a lot faster.  That said, everybody seems to be using VPS hosts these days.  I talked to some other indie game developers, but I didn’t have time to do a full evaluation of the tradeoffs, so went with the devil I knew, so to speak.  It seems like VPS is going to be a bit slower but also a bit cheaper, but the big advantage of VPS to me is that you can move the virtual machine image to faster hardware and have it up and running again in minutes.  That’s a pretty great scaling sweetspot between having a single physical server and praying it doesn’t melt, and a scalable system that elastically uses cloud computing The Right Way™, but it’s also hundreds of times easier to get a VPS image working and then move it to a faster machine than it is to scale elastically.  So, I dunno, it’s definitely something worth looking into more during the year as I see how things are scaling.

The new server ate the robots for lunch:

The new server works pretty well.

For reference, 4850 simultaneous players is pretty far up the top 100 Steam games by player count, so I don’t think I have to worry about those numbers for a while.  Here’s atop’s view of things:

Well within parameters.

What’s Next?

So, that’s it for the lobbyserver loadtesting.  Now I need to move the website and registration system over to the new server, test them a bit, and start inviting everybody in in big batches.  Soon I’ll send out email to the beta testers to set up some scheduled human loadtests as well.  The robots will be jealous, left out in the cold, looking in at all the humans actually playing the game. 

Open Beta is fast approaching.

1. I’ll  release my changes at some point.
2. Well, they were servermatrix when I started, and then The Planet, and now SoftLayer.

When we last left our hero, our differential state update change was a resounding success and reduced the network bandwidth utilization from 98% to 3%, and it looked like we could move on to optimizing the lobbyserver code itself to get to our goal of 1000 simultaneous loadtesting robots, until we noticed some of our robots were missing!  This led me on a wild and wooly chase through the code, which I will recount for you now…

Where’d the robots go?

The first order of business was to figure out why some robots were dying when they weren’t supposed to, and some weren’t dying when they were supposed to.  Robots: they never do what you tell them.

If you look at this graph of the number of running robots from last time, you can see that right off the bat, a bunch of them die on all the machines, and then they keep dying for about 30 seconds, and then it stabilizes.  Each of these machines should have 50 robots running solidly during the test period.

The number of loadtest robots running on each EC2 instance.

Then, to make matters worse, some of them don’t die when they’re supposed to at the end of the test.  In the graph above, they only all finally die when I kill them manually from a separate script at 03:16:30.  This points towards two different problems I’m going to have to debug on the test machines…that only manifest themselves intermittently…with gdb…in the cloud. Good times!

Okay, first things first, let’s see if the robots will tell me where they’re going.  The lobbyclient robots can output verbose log files, but I had them turned off because I was worried about bogging down the client machines.  It turns out this isn’t much of a problem as I’ll discuss below, so I turned on logging and re-ran a test.  Then I ssh’d into one of the servers, and looked at the log files.  Well, before I looked the files themselves, I just did an ls of the directory:

-rw-r--r-- 1 root root 258577 Mar  5 03:02 out59
-rw-r--r-- 1 root root 332320 Mar  5 03:02 out60
-rw-r--r-- 1 root root 177743 Mar  5 03:02 out61
-rw-r--r-- 1 root root 181639 Mar  5 03:02 out62
-rw-r--r-- 1 root root 264535 Mar  5 03:02 out63
-rw-r--r-- 1 root root 333515 Mar  5 03:02 out64
-rw-r--r-- 1 root root 282875 Mar  5 03:02 out65
-rw-r--r-- 1 root root 271040 Mar  5 03:02 out66
-rw-r--r-- 1 root root    264 Mar  5 03:01 out67
-rw-r--r-- 1 root root    264 Mar  5 03:01 out68
-rw-r--r-- 1 root root 284838 Mar  5 03:02 out69
-rw-r--r-- 1 root root 332967 Mar  5 03:02 out70
-rw-r--r-- 1 root root 303352 Mar  5 03:02 out71
-rw-r--r-- 1 root root 310596 Mar  5 03:02 out72
-rw-r--r-- 1 root root 194669 Mar  5 03:02 out73
-rw-r--r-- 1 root root 313193 Mar  5 03:02 out74
-rw-r--r-- 1 root root 238246 Mar  5 03:02 out75
-rw-r--r-- 1 root root 264190 Mar  5 03:02 out76
-rw-r--r-- 1 root root 198096 Mar  5 03:02 out77
-rw-r--r-- 1 root root 233980 Mar  5 03:02 out78
-rw-r--r-- 1 root root    264 Mar  5 03:01 out79
-rw-r--r-- 1 root root    264 Mar  5 03:01 out80
-rw-r--r-- 1 root root 301029 Mar  5 03:02 out81
-rw-r--r-- 1 root root 299694 Mar  5 03:02 out82
-rw-r--r-- 1 root root    264 Mar  5 03:01 out83
-rw-r--r-- 1 root root 351158 Mar  5 03:02 out84
-rw-r--r-- 1 root root 188071 Mar  5 03:02 out85
-rw-r--r-- 1 root root 242228 Mar  5 03:02 out86

Well, there’s a clue, at least for the early-dyers.  The contents of those 264 byte log files look like this:

Lobby Standalone Client: 1000.0.0.5
init genrand w/0, first val is 1178568022
Running for 61 seconds.
LobbyClient started, v1000.0.0.5 / v12
LobbyClient UDP bound to port 32921
lobbyclient: sendto_kdc.c:617: cm_get_ssflags: Assertion `i < selstate->nfds' failed.

A-ha!  sendto_kdc.c is a file in the Kerberos libraries, which I use for login authentication.

I really love Kerberos, the architecture just feels right to me, the API is simple, clean, and flexible, it’s cross-platform and open source, so I’ve been able to contribute features and bug fixes as I’ve used it and trace into the code when I was confused about something, and the folks at MIT that develop it are smart, knowledgeable, open-minded, and don’t mind some crazy indie game developer asking dumb questions about the best way to do things that were pretty clearly not part of the original university and enterprise use-cases.  Most importantly, it’s battle-tested; it’s used by tons of different applications, and it’s the foundation of the modern Windows domain and Xbox authentication systems, so I know it works.  The last thing you ever want to do is roll your own authentication system.

So, that assert’s the first place to look for the early-dying robots.

Next, I looked into the never-dying robots.  I logged into one of the machines that still had zombie robots¹ running, ran pidof lobbyclient to figure out the process ID of one of them, and attached gdb to the robot.  A quick thread apply all backtrace full and I found the thread that was hanging while the main thread was trying to join them and exit cleanly.  It looked like the bad code was in a call to poll, and it just so happened it was in sendto_kdc.c as well! I realized I was going to need some debug symbols, but this was easy since I build the Kerberos libraries myself,² so a quick scp of the debuginfo rpm and reattaching gdb and I could dig down a bit deeper.

The Kerberos libraries are built with optimizations on, which always makes debugging interesting, but I think it builds programming character to debug optimized code, so I don’t mind.³  Here’s the code in question:

    if (in->end_time.tv_sec == 0)
        timeout = -1;
    else {
        e = k5_getcurtime(&now);
        if (e)
            return e;
        timeout = (in->end_time.tv_sec - now.tv_sec) * 1000 +
            (in->end_time.tv_usec - now.tv_usec) / 1000;
    }
    /* We don't need a separate copy of the selstate for poll, but use one
     * anyone for consistency with the select wrapper. */
    *out = *in;
    *sret = poll(out->fds, out->nfds, timeout);

Well, these loadtesting machines are under some load themselves so they can be a bit sluggish, and there’s a problem with this code in that scenario if the call to k5_getcurtime() happens later than the in->end_time passed in by the caller.  As it says on the poll manpage, “Specifying a negative value in timeout means an infinite timeout.”  Digging around on the stack verified the timeout was negative.

Okay, so now we have a pretty good clue for each of the problems.  The second problem with the poll timeout seemed easy to fix, but the first one was pretty mysterious and might take some real debugging.  I decided to check with the krbdev mailing list to see if they had any ideas while I looked into the problems more deeply.  While doing so, I looked at the main Kerberos source repository and found a commit for the timeout problem, so it had already been fixed in a later version.  I was hoping maybe this was true of the assert as well.  True to form, the most excellent Greg Hudson replied with three more commits he thought might help.  Meanwhile, I hacked the code to loop on a call to sleep() instead of asserting to convert the early-dyers into never-dying zombies so I could attach the debugger, since that’d worked so well on the second problem.

Sadly, although the negative-timeout-check fixed the original zombies, none of the fixes prevented the assert problem.  It wasn’t asserting anymore because the asserters were now looping, so now I had more zombies to deal with.

Lots of zombie robots!

Time to get down and dirty and debug it for real.

As an aside, it’s a weird feeling when you’re debugging something on an EC2 instance, since you’re paying for it hourly.  I felt a definite pressure to hurry up and debug faster…oh no, there went another $0.06 * 5 instances!

Too deep we delved there, and woke the nameless fear!

Like I said, debugging optimized code builds character, and I built a lot of character with this bug.  The assert was in a function that was inlined by the optimizer, which was in a function that was inlined by the optimizer, which was in a loop, which looked like it had been unrolled.  It was slow going, with lots of restarts and stuffing values into memory and registers so the code would execute again.  At one point, I thought I’d narrowed it down to a compiler bug in gcc, because it seemed like a variable wasn’t getting reloaded from the stack correctly sometimes, but it was really hard to tell with all the inlining.  Even thinking it was a compiler bug was pretty silly and that thought always violates Assume it’s a Bug, so I should have known better, but it happens. 

Finally, a combination of stepping through the code, and looking at the code, and modifying the code revealed the problem. Here’s the source file at the version I was debugging, linked to the area of the code where the bug lurked.  If you search for “host+1″, you will see that it occurs twice, once inside the loop, and once outside the loop.  This is what threw me when I was debugging…initially I didn’t notice there were two separate calls to service_fds(), so in the debugger I thought it was looping again but loading weird values.  I can only assume the second call almost never occurred in the wild for anybody but me after the inner loop on hosts completed, because in that case host+1 is n_conns+1, which is out-of-bounds for the connections.⁴  This bug was easy for me to fix locally, and it looks like it was (inadvertently?) fixed in this commit in the main Kerberos code.

Thank goodness for open source code, where you can modify it and debug it when you run into troubles!

No more zombies!

Moar Robots!

Now that I (thought I) was done debugging the robots, and I still had 5 EC2 instances running, I decided to see how well the instances did with 100 robots on each.  My original tests indicated I could only run about 50 per m1.small instance, but the client also got a lot more efficient with the differential state update change described last time, and it turns out 100 per instance is no problem, as you can see here:

Top on an m1.small instance running 100 robots at only 20% CPU.

 The lobby was a little more grim with 501 clients:

500 robots and me.

 Here’s how the CPU looks with all these robots in the lobby, chatting at each other:

atop in CPU mode with 500 robots in the lobby jabbering.

There are two cores in this machine, which is why the lobbyserver is at 115% CPU.  It’s mostly single-threaded for simplicity, but it uses threads for servicing network connections.

However, once the robots start playing each other, the CPU usage drops a bunch:

Stop talking, start playing!

This is pretty good news.  I think it means the chat system needs some work, because when everybody’s in the lobby all the chats go to all the players, but when people in are a match, chats only go between those two players, and they don’t get any of the lobby chats.  We’ll find out soon as I describe below.  Memory looks pretty good with 501 clients, staying at about 256kb per client:

2013/03/16-04:53:11: MEMORY_POSIX 501/993/492: resident 25540/25540, virtual 198000/198000
2013/03/16-04:53:11: MEMORY_NEW 501/993/492: bytes 132098963, news 69166, deletes 55478

One last atop screenshot…this one is while the robots are starting up and connecting, but before they’re in the lobby:

Loadtest startup performance.

This one shows Kerberos and OpenLDAP taking a fair amount of time at the start of a new loadtest.  I use LDAP as the database backend for Kerberos, among other things, and when all of these robots are trying to get login tickets at the same time, it bogs down a bit.  I’m not too worried about this profile, since this scenario of 500 people all needing tickets at the same time is going to be rare (the tickets last a while, so this doesn’t happen every time), and there are well-known ways of scaling Kerberos and OpenLDAP if I need them.

Finally, here’s a shot of the 100 robots per instance:

Wait a second…

Oh no!  Who the hell is that single zombie robot at the end on instance 4!?!  Sigh.  I find that machine, log in, attach the debugger, and check it out.  It looks like I have a pretty rare deadlock between two threads during shutdown.  I’m just going to ignore it for now and deal with it later.  All the bugs above were preventing robots from doing a good job at loadtesting, while this one is just preventing 1 out of 500 from shutting down completely…it can wait.  Here’s a shot of this guy, still in the lobby, mocking me:

At least I have one more Sniper win on this debug server than this troll!

There’s actually another bug I found in the new differential state update code while I was testing this, where the server will send a duplicate client sometimes, but I had a comment in the code that I thought it might be possible, and now I know it is.  It turns out when you have 500 clients pounding on a server, you find bugs.

Coming Up Next Time

Okay, so now we’ve got things where I can easily run a predictable number of loadtesting robots against the debug lobbyserver, and I’ve got some high level profiles telling me that I’m now CPU bound inside the server itself.  That points to a clear next step:  profile the code.  I use an old hacked up version of Sean Barrett’s iprof for all my client runtime profiling, so my next task is to integrate that into the server code, and get it running on Linux.  That shouldn’t be too hard, and then I’ll be able to tell what’s actually taking the time⁵ when a lot of clients are in the lobby.

My prediction, based on the above, is that the chat message handling is going to be the main culprit.  If so, it’ll be easy to queue up the chats and send them out in bunches, but I need to be careful here, because the robots chat a lot more than real humans would right now, so I don’t want to spend too much time optimizing this.  I think I’ll keep the robots as they are for the initial profiles, and then dial back their chattiness to more realistic levels after I’ve plucked the low-hanging chat fruit.  I also need to teach the robots how to use lobby rooms for a more realistic test.

Finally, I’m wondering if my usage of select() is going to be an issue as I get close to 1000 robots.  I may need to port to epoll().  We shall see!

“Assume Nothing!”

And finally, the SimCity launch has given me pause…I’m still forging ahead with my 1000 simultaneous goal, but I really hope it’s enough and things go smoothly.  I would much rather have a slow buildup of players over the next year as I roll out more cool stuff than a giant spike that melts everything and makes players grumpy.

On to Part 4…

1. ZOMBIE ROBOTS!!!
2. I have some local patches I haven’t cleaned up enough to contribute yet
3. gdb is not the best for assembly language debugging, but I did learn about “layout asm”, which helps a bit.
4. It never crashed because conns has a preallocated number of connections that was always bigger than n_conns+1
5. This is only partially true, because iprof is single-threaded…I really wish there was a good cross-platform light-weight way to get per-thread timings.

Specifically, it failed by saturating the server’s 100Mbps network link, which turned out to be a great way to fail because it meant there were some pretty simple things I could do to optimize the bandwidth utilization.  I had done the initial game↔lobby protocol in the simplest way possible, so every time any player state changed, like a new connection, or switching from chatting in the lobby to playing, it sent out the entire list of player states to everybody.  This doesn’t scale at all, since as you add more players, most aren’t changing state, but you’re sending all of their states out to everybody even if only one changes.  This doesn’t mean it was the wrong way to program it initially; it’s really important when you’re writing complicated software¹ to do things the simplest way possible, as long as you have a vague plan for what you’ll do if it turns into a problem later.  In this case, I knew what I was doing was probably not going to work in the long run, but it got things up and running more quickly than overengineering some fancy solution I might not have needed, and I waited until it actually was a problem before fixing it.

Tell Me Something I Don’t Know

The solution to this problem is pretty obvious: differential state updates.  Or, in English, only send the stuff that’s changed to the people who care about it.  Doing differential updates is significantly more complicated than just spamming everybody with everything, however.  You still have to send the initial state of all the curent players when new players log in, you have to be able to add and remove players in the protocol, which you didn’t have to before because you were just sending the complete new state every time, etc.

This was going to be a fairly large change, so I took it by steps.  I knew that I’d have to send out the complete state of everybody to new logins, so it made sense to start by optimizing that initial packet using normal data size optimization techniques.  I pretty easily got it from about 88 bytes per player down to 42 bytes per player, which is nice, because my goal for these optimizations is 1000 simultaneous players, and at 88 bytes they wouldn’t all fit in my 64kb maximum packet size, where at 42 bytes they should fit, no problem, so I don’t have to add any kind of break-up-the-list-across-packets thing.  However, it turns out I actually got the ability to send the entire list across multiple packets while I was doing this, because I had to program the ability to add players as part of the differential updates, so now I could just use that packet type to send any clients in a really large player list that didn’t fit in a single packet.  But, like I said in the last episode, although I don’t think I’ll hit 1000 simultaneous outside of load testing for a while, it’s always nice to know you have that sort of thing in your back pocket for the future.

Once I’d tested the new optimized player list, I started making the updates differential.  New players get the initial list, and then they’re considered up-to-date and just get updates along with everybody else.  The list of new players is sent as additions to players already in the lobby.  For each player, I track some simple flags about what’s been updated in their state, so if they set or clear their /away message for example, that flag is set, and I only send that information.

In programming, usually when you’ve got the right design, you get some unintentional upside, and this case was no different.  Previously, I was not sending live updates to player stats (wins, game time, etc.) to the players in the lobby until the player was done playing the match, or some other state changed that caused everybody’s state to be re-sent.  Now, since the differential updates are efficient, I’m updating player stats in real time as well, so people in the lobby can see wins as they accumulate for players in matches, which is nice and how you’d expect it to work.

Results

It basically worked exactly as planned.  After lots of debugging, of course.  Here you can see the profiles for one of the loadtests, which got to 340 simultaneous players in the lobby:

I really need to have the robot Sniper win sometimes.

atop in memory mode

atop in cpu mode

Look ma, 3% network utilization!  That’s whats so awesome about a really spiky profile…when you pound one of the spikes down, things just get better!

Here’s the new table of packet sizes for this run.  If you compare this with the previous results, you can see the PLAYER_LIST packets are way way way smaller, and this table was accumulated from two longer test runs, so it’s not even a fair comparison!  It’s interesting, because the TYPE_LOBBY_MESSAGE_PACKET is smaller as well, and I think that’s because now the robots can actually start games since the network isn’t saturated, and this means they don’t broadcast chats to the entire lobby while they’re playing, so that’s a nice side effect of optimizing the bandwidth.

Hmm, I just noticed as I’m writing this that the resident memory utilization in the atop screenshot is way lower now than before…I wonder why… On the application side I take about 250kb per player right now, which at 340 players should be about 85MB.  Looking at the lobbyserver logs, right about when the screenshot was taken, the lobby self-reported this data:

2013/03/03-02:13:15: MEMORY_POSIX 348/757/409: resident 12808/12808, virtual 160276/160276
2013/03/03-02:13:15: MEMORY_NEW 348/757/409: bytes 91766974, news 45707, deletes 36155

The MEMORY_NEW stats looks about right for this load and my quick math, but the MEMORY_POSIX stats—which are read from /proc/pid/status—match the atop results: expected virtual but low resident.   Maybe it was just paged out for a second, or maybe I’m not touching much of that 250kb and so it doesn’t stay resident.  A lot of it is network buffers, so it makes some sense with this lower bandwidth protocol that it wouldn’t be resident compared to last profile because less buffering is having to be done.  I’ll have to investigate this more.

Up Next, The Case of the Missing Robots

So, the bandwidth optimizations were a resounding success!  Plus, both the CPU and memory utilization of the lobbyserver are really reasonable and haven’t been optimized at all, so we’re sitting pretty for getting to 1000 simulataneous robots…

Except, where are the remaining 160 robots?  In the test above, I ran 10 EC2 instances, each with 50 robots, thinking the optimizations might let me get to 500 simultaneous and find the next performance issue…but it never got above 340 in the lobby.  I updated my perl loadtesting framework and had each instance output how many lobbyclients were running every two seconds with this shell command over ssh:

'while true; do echo `date +%T`,`pidof lobbyclient | wc -w`; sleep 2; done'

And then I loaded that into gnuplot,² and graphed the number of robots on each instance:

The number of loadtest robots running on each EC2 instance.

You can see that they all started up with 50, but then a bunch of them lost clients until they found a steady state.   Something is killing my robots, and I need to figure out what it is…

Turn the page to Part 3…

1. especially by yourself!
2. …which I hate, but I forgot to install excel on my new laptop, and Google’s spreadsheet sucks at pivottables, and the Office for Web excel doesn’t even have them as far as I could tell!

Like with any kind of optimization, Assume Nothing, so you should always write the loadtester first, and run it to get a baseline performance profile, and continue running it as you optimize the hotspots. When I started, the signup submission could only handle 2 or 3 submits per second. When I was done, it could handle 400 submissions per second. I figured that was enough.³ If more than 400 people were signing up for the SpyParty beta every second, well, let’s file that under “good problem to have”.

After all the loadtesting and optimizing, the signups went off without a hitch.

Loadtesting and optimizing the beta signup process was important, because the entire reason I took signups instead of just letting people play immediately was “fear of the unknown”. I couldn’t know in advance how many people would be interested in the game, and getting a couple web forms scalable in case that number was “a lot” was much easier than getting the full game and its server scalable, and that’s ignoring the very real need to exert some control over the growth of the community, to make sure the game wasn’t incredibly buggy on different hardware configurations or that there wasn’t some glaring balance issue, etc. Overall, starting with signups and a closed beta was great for the game, even if it’s meant frustrating people who signed up and want to play.

But it’s been long enough, and I’m now finally actively loadtesting and optimizing for opening the beta!

Lobby Loadtesting Framework

Like with the signup form, I’m loadtesting first. This will tell me where I need to optimize, and allow me to test my progress against the baseline. However, loadtesting a game lobby server is a lot more complicated than loadtesting a web form, so it’s a bit slower-going. I’ve had to create a robot version of the game client that logs into the lobby, chats, invites other robots to play, and then reports on the results of the fake games played. I build this on top of the game’s client interface, so it looks just like a real game to the lobby.

As with all testing, you need to make sure you aren’t Heisenberg-ing⁴ your results, so I wanted to get fairly close to the same load that would happen with multiple real game clients hitting the server. This means I had to have a good number of machines running these robots hitting the test lobby at the same time, and that means using cloud computing. I was inspired by the bees with machine guns article about using Amazon Web Services’s Elastic Compute Cloud (EC2) to launch a bunch of cheap http load testers. I use AWS for SpyParty already, distributing updates and uploading crashdumps using S3, so this seemed like a good fit. At first I tried modifying the bees code to do what I want, but I found the Python threading technique they used for controlling multiple instances didn’t scale well running on Windows, and since I wanted more control over the instances anyway and the core idea was not terribly difficult to implement, I wrote my own version in Perl, which I’m much more familiar with. The code uses Net::Amazon::EC2 to talk to AWS to start, list, and stop EC2 instances, and Net::SSH2 to talk to the instances themselves, executing commands and waiting for exit codes, downloading logs, and whatnot. I just use an existing CentOS EC2 AMI⁵ and then have the scripts download and install my robots onto it from S3 every time I start one up; I didn’t want to bother with creating a custom AMI when my files are pretty small. I’m going to post all the loadtest framework code once I’ve got it completely working so others can use it.

How Much is Enough?

In loadtesting the loadtesters, I found that an m1.small instance could run about 50 loadtest bots simultaneously with my current client code. I can switch to larger and more expensive EC2 instance types if I need to run more robots per instance, and as I optimize the server I’m pretty sure the client code will get optimized as well, which will allow more concurrency. Amazon limits accounts to 20 simultaneous EC2 instances until you apply for an exception, so I’ve done that,⁶ but even with that limitation, I can loadtest to about 1000 concurrent clients, which seems like more than enough for now.

I still don’t know exactly what to expect when I open up the beta, but I don’t think I’ll hit 1000 simultaneous SpyParty players outside of loadtesting anytime soon. If you look at the Steam Stats page, 1000 simultaneous players is right in the middle of the top 100 games on the entire service, including some pretty popular mainstream games with mature player communities. In the current closed beta, I think our maximum number of simultaneous players has been around 25, and it’s usually between 10 and 15 on any given night at peak times, assuming there’s no event happening and I haven’t just sent out a big batch of invites. I still have about 6000 people left to invite for the first time from the signup list, and 9000 who didn’t register on their first invite to re-invite, all of whom I’ll use for live player loadtesting after the 1000 robots are happily playing without complaints. I think the spike from those last closed invites will be bigger than the open beta release spike, unless there are a ton of people who didn’t want to sign up with their email address, but who will buy the game once the beta is open. I guess that’s possible, but who knows? Again, if we go over 1000 simultaneous, I guess I will scramble to move the lobby to a bigger server, and keep repeating the “good problem to have” mantra over and over again, but I’m betting it’s not going to happen and things will go smoothly.

After open beta there will be a long list of awesome stuff coming into the game, including new maps and missions, spectation and replays, the new art, and lots more, but once things are open it’ll be easier to predict the size of those spikes and plan accordingly. Eventually I’ll probably (hopefully?) have to move the lobby off my current server, but I’m pretty sure based on my initial testing that the old girl can keep things going smoothly a bit longer.

Initial Loadtesting Baseline

Okay, so what happens when I unleash the robots? Well, I haven’t let 1000 of them loose yet, but I’ve tried 500, and things fall over, as you might expect. It looks like around 250 is the maximum that can even connect right now, which is actually more than I thought I’d start out with.

The loadtesting robots are not very good conversationalists.

Things don’t work very well even with 250 clients, though, with connections failing, and match invites not going through.⁷ However, when I looked at atop while the robots were pounding on the lobby, a wonderful thing was apparent:

atop in CPU mode

atop in memory mode

Neither the CPU utilization nor the memory utilization was too terrible, but the lobbyserver was saturating the 100 Mbps ethernet link! That’s awesome, because that’s going to be easy to fix!

Before I explain, let me say that the best kind of profile is one with a single giant spike, one thing that’s obviously completely slow and working poorly. The worse kind of profile is a flat line, where everything is taking 3% of the time and there’s no single thing you can optimize. This is a great profile, because it points right towards the first thing I need to fix, which is the network bandwidth.

My protocol between the game clients and the lobby server is really pretty dumb in a lot of ways, but the biggest way it’s dumb is that on any state change of any client, it sends the entire list of clients and their current state to every client. This is the simplest thing to do and means there’s no need to track which clients have received which information, and this in turn means it’s the right thing to do first when you’re getting things going, but it’s also terribly wasteful performance-wise compared to just sending out the clients who changed each tick. So, I was delighted to see that bandwidth was my first problem, because it’s easy to see that I have to fix the protocol. I’m guessing switching to a differential player state update will cut the bandwidth by 50x, which will then reveal the next performance spike.

I can’t wait to find out what it will be!⁸

Oh, and the total EC2 bill for my loadtesting over the past few days: $5.86

So…Open Beta?

Within weeks! Weeks, I tell you!

Oh, and as I’ve said before, everybody who is signed up will get invited in before open beta. I will then probably have a short “quiet period” where I let things settle down before really opening it up, so if you want in before open beta, sign up now.

Update: Assuming More Nothing…Er, Less Nothing?

After posting this article, I was about to start optimizing the client list packets, when it occurred to me I wasn’t assuming enough nothing, because I was assuming it was the client list taking all the bandwidth. This made me a bit nervous, which is the right feeling to have when you’re not following your own advice,⁹ so I implemented a really simple bit of code that accumulated the per-packet send and recieve sizes, and printed them on exit, and then threw another 250 robots at the server for 60 seconds. The results validated the client list assumption, it’s by far the biggest bandwidth consumer, sending 1.6GB in 60 seconds.¹⁰ However, it did show that the lobby sending chat and status messages to the clients is also maybe going to be a problem, so yet again: measuring things is crucial.

It’s interesting that the clients are only sending 300KB worth of chat messages to the lobby, but it’s sending 66MB back to them, but 66MB is around 250 * 300KB, so it makes back-of-the-envelope sense. I’m probably going to need to investigate that more once I’ve hammered the player list traffic down. Maybe I’ll have to accumulate them every tick, compress them all, and send them out.

This way to Part 2…

1. See this thread for how I wrote the dynamic loadtesting form submission in a way that would saturate the network link.
2. I use CF for images and other static stuff, with W3 Total Cache to keep them synced to S3, but I only use W3TC for this CDN sync, since Varnish blows it out of the water for actual caching.
3. Let me be clear, I think 400 submissions per second is really pretty slow for raw performance on a modern computer, but web apps these days have so many layers that you lose a ton of performance relative to what would happen if you wrote the whole thing in C. For an interesting example of this, there’s a wacky high performance web server called G-WAN that gets rid of all the layers and lets you write the pages directly in compiled C.
4. I just read on wikipedia that the uncertainty principle is often confused with the observer effect, and so on the surface this verbing of Heisenberg’s name isn’t correct, except he apparently also confused the two, so I’m going to keep on verbing.
5. ami-c9846da0
6. although they haven’t gotten back to me so I guess I’ll apply again…sigh, customer service “in the cloud”  Update: Woot!  My limit has been increased, now I can DDOS myself to my heart’s content!
7. Let’s ignore the lobby UI also drawing all over itself for now.
8. You can see the CPU usage is pretty high relative to the memory usage, and seeing slapd and krb5kdc in there is a bit worrying, since that’s kerberos and ldap, which are used for the login and client authentication and are going to be a bit harder to optimize if they start poking their heads up too high, but both of them have very battle-tested enterprise-scale optimization solutions via replication, so worst-case is I’ll have to get another machine for them, I think. If the lobbyserver itself is still CPU-bound after fixing the bandwidth issue, then I’ll start normal code optimization for it, including profiling, of course. I’ll basically recurse on the lobbyserver executable!
9. …let alone Mike Abrash’s advice!
10. Or actually trying to send, since 1.6GB in 60 seconds is 200Mbps, which is not happening on a 100Mbps link!

Courtyard

In the old days, we walked to the only map, the Ballroom, uphill, both ways, and we liked it.  Eventually, after a couple years of playtesting on that single map, I decided to test my working theory that making new and interesting maps for SpyParty wouldn’t be too terribly hard, at least relative to something like doing the AI and animations for a new mission.  In other words, I was working under the assumption that because most of the gameplay in SpyParty is based on the character behaviors and the perception and deception mechanics, doing new maps would be a relatively easy way to add new gameplay.  The maps would obviously be important and add a lot of variety to the game, but their layouts would not be as crucial to the feel of the game as they are in a shooter or a platformer.  I mean, Ballroom itself is just a box and everybody seemed to like it!

So, I tested this theory by adding the Balcony and Veranda maps, and I’d say the theory was proven to be true, since both of those maps worked well and people like playing them.   The Ballroom is still the go-to map for serious play because it’s the most tuned and balanced, but the other maps add some spice and hint at how the game will feel with lots of different settings and map topologies.

On the runup to PAX, I had an idea for a new map where the statues would be centrally located, but the Sniper couldn’t see them all at the same time.  I was wondering how that would change the Inspect Statue and Swap Statue missions, which have turned out to be two of the riskiest missions for Spies to attempt.  My original thought for this level was a single statue right in the center, but I decided that would be too hard, so I settled on three statues embedded in a central pillar, where the Sniper can only see one at a time.

I hacked this map together one night at PAX, and we playtested it the next day.  The initial tuning was way too hard for the Sniper, not to mention how hard it was to get your bearings.  After some tuning, and Wessel’s suggestion of using a statue of Alan Turing to give the Sniper an orientation reference point, the Courtyard seems to be working out well.   I think I overcorrected a bit and the statue missions are now a bit hard for the Spy again, and it needs more playtesting and tuning, but it’s definitely already interesting and fun to play for both roles.

Current stats for Courtyard look like this:

You’ll recall there there are currently three game types in SpyParty, “Known n” where the Sniper knows the n missions the Spy will try to accomplish, “Pick n of m”, where the Sniper knows the m missions available, but not which n ones the Spy picked, and “Any n of m”, where the Spy can choose which n to accomplish opportunistically during play, rather than having to pick them ahead of time.  We’ve started abbreviating these k4, p4/5, a3/4, and so on in the beta chat and forums.

As you can see from this table, the different modes have different levels of handicapping between the players.  I’ve just gathered these numbers from all the games on the Courtyard map, so this is totally ignoring skill differences, early games where players were just testing out the map, and the like.  But, it’s still interesting to see the balance.

Unfortunately, as you can see, the Any game types are missing their n parameter.  As I started gathering this data, I realized I’d screwed up and left that number out of the database for Any game types, so I can’t tell if the a?/5 games are a3/5 or a4/5, which is a bummer, because it makes a big difference in difficulty for the Spy and Sniper.  I’ll probably have to ignore old Any games when I do real ranking with this data.  I’ve since fixed it,¹ and although there aren’t enough games with the new correct data yet to be statistically significant, here they are:

Chat Rooms

Soon I will be opening up the beta, and this (hopefully) means a lot of people will be playing SpyParty simultaneously, at least towards the end of the closed beta when I send out the rest of the invitations and into the beginning of the open beta when anybody can sign up and play immediately.  The old lobby had a single room, and when 10 or 12 people were in there chatting, it got a bit hard to keep track of who was talking to who and what they were saying.

So, I did the simplest chat room design possible, with the goal of load-balancing the lobby if there were 30 or 100 simultaneous users chatting.  I basically decided to put all the room creation and multiple-join/leave commands on the chat console, and have the GUI only allow you to select a single existing chat room at a time.  This way advanced users can hang out in multiple chat rooms and make new rooms, but newbies can just join one and chat and play a match without thinking too hard about it.

The jury is still out on whether the current chat rooms work very well, and we’ll have to wait until I invite more people in to see if they solve the load balancing problem, but they’re certainly slightly confusing at the current density.  I tried to make sure joining “Headquarters” (the default room that’s always there) was easy enough that you didn’t have to think about it and it acts just like a single room lobby if there aren’t many people online, but I didn’t want to dump people automatically into a chat room for fear of causing problems when the game was scaled.  We’ll see how it works out.  I foresee a lot of changes in this code and UI as the number of simultaneous users goes up.

One change I made shortly after releasing the feature was adding a “Created by <username>” line in the chat room description.  Since users can create any rooms they like, this will at least keep people honest, or at least culpable!  Before I added this players were creating rooms that sounded official, which was funny but probably fraught with peril.  I also don’t support hidden rooms or password protected rooms or anything yet.  I’d like to keep the community from segregating as much as possible for the time being.

On the subject of chat, I really need to do something soon about my text rendering code, it is incredibly slow right now:

checker: /fps
SpyParty version v0.1.2202.0
19.869 ms/frame (fps: 50.33)  sort self - current frame
zone                                                     self     hier    count
 checkerlib::simple_font::DrawStringOrthoPixels       18.6377  18.6377    54.78
+chat_display                                          0.3964  16.6585     1.00
 SwapBuffers                                           0.2445   0.2445     1.00
+IdleFinal                                             0.2400   0.4846     1.00
+_global                                               0.0582  19.8692     0.00

Logging Performance, etc.

There were a bunch of other smaller fixes, one of which other programmers might find interesting:  I use the spypartyhelper.exe to write the log files now, rather than having the main spyparty.exe write them out.  The helper has always done the crash dump management (using a modified version of Google’s Breakpad) and the auto-updating privilege escalation, but now I also have it handle logging, using a named pipe from spyparty.exe.  The reason I made this change was some machines (like my main development machine) would happily write out the logs in the main process without affecting performance at all, even though I had commit mode turned on for the log file.  I used commit mode so if the game crashed, the log file would contain the most up-to-date log entries, so when the helper grabbed it and put it in the crash dump report and uploaded it to my server I could use it for debugging.  However, on some machines, the commit mode logging would take hundreds of times longer, and I couldn’t figure out what the difference was between fast and slow machines.  What’s worse, it became clear that a lot of beta testers were assuming the game just loaded slowly, which subtly (or not-so-subtly) affects how you feel about the game, even though all the time was being spent writing log statements.  Once I realized people were suffering with slow load times, I knew I had to fix it.

The weirdest bug I fixed involved the Spy playing the cough animation as soon as control was taken from the AI.  The cough animation is supposed to be played when the Spy bails out of a conversation too soon after saying “banana bread” to contact the Double Agent.  I added the cough because some people were cheesing out and running away from the circle before the contact animation had finished, and it was too hard for the Sniper to spot this glitch.  So, the cheesers got a big surprise a few builds back!  But, it shouldn’t be happening in a new game if you haven’t even done anything yet, and beta testers couldn’t get a good repro.  Well, after a bunch of debugging, it turned out I wasn’t clearing a timer in the Contact Double Agent situation on new game, so if somebody was shot within 2 seconds of a “banana bread”, then since they were moving at the start of the next game but the timer hadn’t expired, the situation decided they’d cheesed out of the conversation and made them cough.  Yay for cross-game bugs!  I guess it was the ghost of the previous victim making the Spy cough!

Mail Deliverability

Man, sending mail reliably on the internet today is fraught with peril.  I’ve been trying to figure out how often a beta invitation or confirmation mail simply doesn’t get delivered, or gets dumped in a spam folder.  So far, I’ve struggled with and been victorious getting SenderID and SPF working, and I’m still wrestling with DKIM, but I am confident I’ll prevail…eventually.  Of course, all the major mail clients use different standards.  I guess I could just pay a mail sending service to handle this for me, but it just seems crazy if I’m forced to do that to get mail delivered.  Then again, I’ve wasted a ton of time dealing with this, so maybe it would have been worth it.  I tend to think the learning and understanding I get from working through these problems myself is worth the time investment, but that could easily be rationalization, who knows?!

Update:  I think I have prevailed in the email war.  I’ve got all the major email providers² verifying my DKIM and SPF records.  I had to battle php messing up newlines which confused the hash, SELinux disallowing postfix from talking to opendkim, and whole bunch of other randomness.  My only problem now is with AOL, and their postmaster just admitted the problem was on their side and they’re fixing it.  Hopefully that is the end of any spyparty.com email going into the spam folder, woohoo!  I sent out 500 invites tonight to celebrate, which brings us to  2011-05-11 06:35:21 US Central Time on the in-order invites.³

1. I’ve tested the results with beta member “tytalus”, who was very patient!
2. Out of 20k beta signups: gmail 56%, hotmail+uk+live 14%, yahoo 7% aol 1%
3. I do 60% in-order and 40% random in each batch.

I’ve been a bit quiet because I’ve been working on (hopefully) completely invisible stuff involving backend server scalability.  What does that mean, you ask?  In practical player-facing terms, it means I’m trying to get the lobby and registration system robust enough to first invite the rest of the waiting list in,¹ and then to open the beta up completely, allowing people to hear about the game, visit the site, pay their money, and get into the beta instantly.  Between 10 and 500 people sign up a day, depending on how much press the game is getting at the time, and I’m very curious to find out how many of those will join the beta if it’s available immediately, and they aren’t told they have to wait.

I’m currently working on the text-mode robot client.  I’ve got it logging into the lobby and pretending it’s a full SpyParty client, and it can chat, and invite people to games.  I posted on facebook and twitter asking for suggestions of what the robots should say to each other as they’re hanging out in the lobby.

Next up, I’m going to get the robots playing fake matches and games and reporting the game results to the lobby, and logging in and out a bunch.  Then, I’m going to hack up the bees with machineguns load testing app to launch a bunch of EC2 micro instances with multiple robots running on each, and aim them all at my test server.  I expect chaos.

Once I’ve got the obvious stuff fixed, and have figured out how many clients my system can host at a given machine size, I’ll do some tests with beefier machines to make sure it scales linearly.  There are four basic machine resources: CPU, memory, network, and disk…I assume I’ll run out of memory first, but I don’t know for sure.  I was originally going to try to get the entire backend infrastructure running in the cloud, but I think for the near term I’m going to just make sure I get a machine that can accept way more clients signing up and playing than I think I’m going to get when I open up the beta, and hope it holds up.  If it dies due to too much traffic, well, I guess that’s a good problem to have, at least!  I did this same kind of thing with when I started taking beta signups, and I stress tested and optimized it well enough that it didn’t even break a sweat, so hopefully the beta launch itself will go as smoothly. If not, then if Blizzard can do it, so can I! Yikes.

Then, once I’ve got the backend scalable for the robots, I’ll start inviting much larger groups into the beta to loadtest with live players.  There are some features I’m going to need to add to make this work, like chat rooms and colored chat text, and it’s going to be pretty raw to start with, but it should work okay.  The beta testers have been very forgiving of my incredibly primitive lobby so far, so hopefully that attitude will continue!

After everybody’s in, I’ll shut off the signups and let it stew for a few days, and if it’s working, I’ll open it up!  I hesitate to give time estimates on this stuff, because I’ve never hit a date, but most of this will be happening over the next few weeks.

Asserts!

Programming is complicated, and handling errors makes it even more complicated.  Oftentimes, it’s good programming practice to not handle some types of errors gracefully, but simply assert that you aren’t in a state you shouldn’t be in, and if the assertion fails, you exit with an error.

assert(1 + 1 == 2);  // integer arithmetic better work or we're hosed

Of course, during development, the impossible is not only possible, but likely, and so your asserts fire.  And, often in complicated code, you’ll assert things that should be true, but aren’t catastrophic if they’re not true, and so you usually pop up a dialog that gives you the choice to break into the debugger, abort the program, or ignore the assert (just this once, or forever).  The problem is, once you let yourself have that ignore option, you can get lazy, and start using asserts as popup reminders of things to fix.  This tends to be really bad on a large team, because the game is asserting every 2 seconds, and you’re just hitting ignore all the time to other people’s asserts, and it becomes a habit.  However, on an indie sized team, which in my case means exactly one programmer, one can use asserts this way and they can still be useful.  I can get a feel for how often certain types of things are going wrong while I’m testing, and I can remember what I was doing when a specific assert fired most of the time…it goes from a purely quantitative tool to a qualitative tool.

I also can leave the asserts on in beta builds, but in a way that fires silently once and then auto-ignores-forever, and then have them shipped off over the network to the server, so I can see what kinds of things are going wrong on player machines, and how frequently they go wrong.

I tend to be pretty liberal with asserts in my code, and so they fire a lot, and in turn the server logs a lot of them.  About 30000 of them so far in the beta.  Here they are, sorted by frequency:

1. currently at 16681 people as of this post

aaboyles aaerox abettega ablemonkey aca15 acbgardner aces acron actionpotential adam adman aerox2 aethyr agent agentmidnight agradula ahalay ahnimal aisugirl akuda alessandro ali alkapwn almostpeaceful74 amagineer amar amassingham amonjura ancillas andrew andy angellion01 angryamish angryjaffa angrywombat ankhwatcher annoyedguy anotheruselesspwn antoine antonio applemint apreche arbaard arbiter ardonite aresbece arthol ashenke astroblack astrognat attackslug atticusfinch austin authoralden avian avianflame awakewise awayman awesome awesomex awkwardhero azazoth azeltir bambu85 bassguitarbill bater bbrocker bc5389 bclose bdgolfer81 beanopolis beavens beckett007 beffy belak beljerd ben benny bepedos berbank beric berndie90 berryalice bespectacledgent betabetamax billiechar binarychimp biodork bishop bizzkaj bjorngylling bkyoops bl00dw0lf blabber blackheart blacklamb blaine blargnblarg blimeycow blind blindcicero blindprophet blinkity bluenator bluewins bnetter12 bobbesmcgee bobo bogger bom351 boner bracken brackhar brawnyfanta braytek brettbot bronn browncoat768 bslinger bubonulus bucksexington bukit1 bull bungeeman bunnyfoofoo bureaucromancer burgly burguois buxx cafebagels caincc calitar callguinness camgoeswild camperbob canadianbacon candafilm caphector capnlee captainduh captainquincy captgadianton carrington casey castor cathlin causeofdeath15 caustic cba123 chan charliechan charliehuge checker chris chris2c2 chrisbell chrisfranklin christianwins chuckgriff chvyvele cian cigumo ciminator ckarinja clarkson101 click cliff cloversquirrel clovest clownbaby clownzilla clubley2 codarl codyme1 cogbyrn cokelogic colemip coleopteran coleslawesome colinnorthway collinc colourmeindigo commandercup commandersven communistbloc compositeredfox congodave cornpalace corvon coubs38 cpsabotage crackilator cramill creasy cru31ty cskilbeck cube cukydoh curieous cyberpirate cyclpsrock d1a2r3k d4rkfr4g dacool561 daddykingkong daeke daenrr daethwing188 dagstur daheadhunter dak722 damatman dan danblanchard dandyshlonglegs danger dangerskew danicia danielben danigold dariusk darkcephas darkernukes darkswordmaster darktrooper darrentorpey darthbutternutz dash daveycraney david2crazy dbfclark dbltnk dead deadbolt deadjoe deep9x defendership deftmunky dekuplatformer desdewd develin devp dexan dfan diamondae dieffenbachj dienekes00 diffusion digikrynary dinozzo dios dippyt dirk dizzyduke dj2scoop dla26 doc docace doctorwhy docyahoo doddzy39 doenan domofizz doneval dookie dorkus1218 dotreptile doug drdos dredgman drmuncie drqqq dukrous dumbitdown duncank dungeonduck dutchsanta duwilli dvorak eddie edgeblend eggs eight einhandr eirabben eklyptz ekombokom elcarlito elendira eljenso ellbent elpicante emanon emil emmick4 emuspawn emwurst engagequadlaser enragedviol ento ephargy erariga escapecharacter esperento esuark06 etran1 etsd eversoslightly evolved exayden ezithau fain faithbleed falcon9xr falconred fancyfrog fapsauss faryn fatheroctopus favocado feenox fenir fetthesten finners fischl flamingarms flanjygo foo forspareparts fortninety foxblock fredthedeadhead french fringe frondo furay fuzzyspo0n fweez fyre gabriel gabrielsp gabumon2 gahd gamersdg garythellama geekgrrl geekynerd geibys geiko generalchaos gerardi getaway geveeso gimmefiction ging girricane gkyl gmandm goat6star6of6astaroth goblineat2 godfatherbrak godfire godzchozenson goinbananas golds golzernurf goodboy goofus gordon gordonfremen gouda graywolf greger grempkin griffinmcelroy grifta67 grimwtf grinspork gryphus1 grythean gummikana guncannon gunner1905 gunslingernz guylum haganehylian hajav halcyon hansoneg harrisonthefan harrylee hasmarth hatebobbarker havochq hawk haze hdfisise headbuddy hehateme henningdc hermituk hernismall herrdoctor hinges hobbes2424 hollygolightly holodoc homeless hop3less hothmonster hulkmaster hunter4hire hutty211 iamcarneiro iax ibis idrisguitar iiatlas ikubiakius ilzxc imaginarydinosaur immortalwombat imthedude ineffable ininja132 inker intero iocat iou1username ironhive isamu itsatrap itsmyyard itt0 j0be jacksonaces jake james1221 jaqenhghar jason jasonbourne jcmoore jedexodus jedimasta jeffatrad jeffnosanov jello jennylynn jentaro jeremiahbritt jetlizard jguzikowski jjb273 jleet82 joakim joeaverage joeprunz johnnemann johnnyr jonmm1776 jonoabbo jorispz joset305 jrmtz85 ju34u julian justin justinliew jvnane kai kalei50 kami kapit karabinerkarl karaidon karibou kate katya kb0m kbd keith kelemvor kellywallick kennyal kevin keysosaurus kidko kifil killercow kkriller klinefelter0 kmonarq7 kmuncie knifeninja kobolt kramff krow007 krpiper kujiun kunstbanause kwhiggs8 kyarourin kyoun lackey lagarathan lala lameduck lamzor lancelake larakin launchbay07 lavinashfield lawiin lawlercaust lazz leaveittoreaver leichelle lemming liamh101 lightishred linkz lioninacoma liquidindian liquience ln3000 loddy logomorph loki lopert lordking loshon lovick lrrr ltfragbox ludusvan lusomai m0r7if3r maarch macintoth mackinder maddragon maedas malbios malfunktionv2 maniacalv mantis marcham93 marciantobay marcnotmark markfnd markrestart markus marlowe mathewcrichton matt mattseligman maverick21xx maxemoe mazazon mbrewski mcc meanbrian megadestructo megapixel23 mehall merc mercuryn mercy metaregress metronome49 mevica michael michibuenkosai micole miggins mike mikecruz mikegbrown mikejmika mikella mikey minecraftboy3437 mistadad misterfix misterpanky mizzrie mmcdhoward mmorier mmustapic mogri moo moof moomanibe morfran morganic mosse motorcrash mrbanana519 mrcompletely mrhoten mrmember mrpants mrphatjared mrprice33 mrpropellerhead mrssaurbaum msarge mtb mukow murray mvandevander mylene namelesssniper nara nateventure nayon7 nayrev naysayer nazzerac nefiorim nelsormensch nemesiscw nemo nerdmancer neuman nexai nicemissmayonnaise nickmac863 nicktheflanders nicta nille niteowl noc noclip noethis nogon noise noisia noonan noonchild norskov northernlion nosferatastic notbengilbert notch nuhuhnoway nukednova nunix nyo odeyin odie ohsickkbro okibi olninyo omar omgmyface omnomonsouls onik onyxmoron opticalshadow or1g1nal1ty orahpajo ossy otakushi overlordror ozzyj88 paarre palm partario pastorpeej patrick patyrick pauzle pawng pax217 pbobzebuilder penumbra peter phaazoid phase phatjax pheranova phobon phokal pickard12 pigo pikes piratepwnsninja pixelfenix pixeltitan pixistick pnomoth pobblepop poison5htp pokeadot polycube popabawa porousnapkin postgoodism powerlanguage pparks76 prall prelone prios proverbialbones psponge psytog pubtongue puppet purplevanes pylons qq qualityjeverage quanimal queenkid1 quickfics quirken rachel rachet20 radchek rage ragegoblin rager ragtag21 rainault raphaelo rapscallion ravi ravie77 raypaik rebeleleven redbeardedowl redivy redrocket redrum remy remzo rennix resdog reusablebox revdancatt reverendanthony rex rick rift rinzai ripply rj rnw159 robinaire robotpants rockliao rocky rodomont roel1976 rogermoore rook rorrim ross rowdynun ruhmann rullerr rune ryan rym sabones sachiarias sagemichael samuel samuelcole sarahnorthway sardiax sarge sargonas sarrick saucepain saurbaum sawchuk1 sb00ya scnigey scopesofter scottish scotto6uk screwjack scud seansf seeking segolectrick seiromem senisan sentry sepherimorth sequel seregrail7 sgtfury shakewell shatter1313 sheepofheight shurp sigmatheta simon2776 simonbob sirblastalot sirlaitier sk82712 skapig skrymir skuchachii skuzeeii slack slats sleepyendymion slyvr smacktubby snakeaes sneakypeaky89 snoken snowmaneater soda solidplasma someguy sonicblastoise sonicboom soolseem spencerrohan spikerama spinnermaster spinyeti spliter splodeybaloney splodn spouwny spy spychao999 squirtle srjenkins starkiller2000 stdizzie steakeatsbabies stents sterling steve stockdogfactory storm stu stuart superdraw swenson sylver symlink syrinx t3kk taco taker talensis tam tantfarbror taquelli tautology teddydief telephis tembac tenebrous tenjouutena tenochtitlan terrytate testing texanfromin thealmightydan thebogdan thechicken thecze thedirt thedoctor52 thedoomsmith thegeckoj thegerf theharribokid theinfamousnolan thejoshfox thelaughingman themantis themightyyeti themosb thename thenew110 theotheralex ther thesocialsolipsist thief thirdordeal thornsten thoseposers thsandman tiefooz tigerseye85 tigershark tilton time5 tlabresh tobybot tokion tomeng1997 tomqbui toomuchpete tpwt tralan trauts travis treadd tree treythalomew trickery triplelex trouserdemon tubybuny tulse twie twinbolt twistamafoo twoflower tylerthedesigner undeadnub undisclosed uneddy vadid37 valir valuedrug vanlo varanas venom veturi vidvandre vincentdpc vinnyboombatz vorrik vortexmortinghan voxhumano vs vutdevuk wantedbob warlock watertower webjr weebee886 weeble wesley weteor whatthegeek whispous whitewolf whizkid wiggy wikzo willdude william wilson2k4 wiltwin wirapuru wish wkerslake woodensplint woof woomstick woseseltops wubang3r wubby wyrmling x13ucketheadx x1pyvr84 xav xemu xenoacid xlom3000 xspjigo xtreyu xzalander yanndos yazzflute yegg yobani yokelbloke yorudy yourdad zacaj zachwlewis zafo999 zaktilu zapdsl zen zerooneinfinity zerosyphon zerotka zerozshadow zgilly zhiwiller zoink2000 zoneghost zynux

1. Updated numbers to be clear: There are 13788 people signed up as of right now, 1081 registered (I invited more in after this post), 2430 invites total sent, for a 44% registration rate…I’m still inviting people in slowly until I get the lobby scalable.

I was killed.

Wow, I have been busy, and the Early-Access Beta todo items are piling up! I will just get right to talking about stuff, no introduction necessary!

The Depth Jam and Deduction

Recently, a small group of us did an intensive game design retreat we called the Depth Jam, during which I dug into the deductivity of the Sniper gameplay.  I want there to be some deductivity on the Sniper side, so the player has some method or process he or she can step through to eliminate suspects and not just completely drown in information, but I don’t want you to be able to completely grind out who the Spy is by a series of rote operations, like in a game of Clue.  Since one of the aesthetic themes for SpyParty is about “making consequential decisions with partial information”, it’s very important that the Sniper not be able be 100% sure of the Spy’s identity, unless the Spy screws up and the Sniper sees a hard tell.  There are a lot of places where deductivity comes into play for the Sniper, but an important one is the highlight/lowlight mechanic.

If you’ve read the documentation, you know that the Sniper can highlight and lowlight people at the party to manage levels of suspicion.  This is purely a Sniper-side mechanic: the Spy doesn’t know for sure which characters are being tagged in this way or how, except that the Sniper’s laser has to hit the character to tag them, and this is visible to the Spy.  The current version of this mechanic has two levels of highlight above neutral, and two levels of lowlight below neutral.  The original intent was that you’d want to vary the amount of suspicion or lack thereof based on how the party is progressing, and for the most part, this worked.  However, gamers are very good at optimizing things, and soon one of the elite players realized he could use these five levels as a 5-counter, using both the highlights and lowlights to count total number of visits to the statues and bookcases.  Depending on the missions selected for the game, this Sniper would have a pretty good idea of who had to be the Spy.

From a game design standoint, this kind of player behavior is a delicate thing to direct in a player-skill game.  Should the game help the Sniper do this bookkeeping because the player is going to want to do it anyway?  Or, does the existence of multiple levels actually encourage this kind of bookkeeping?  Should I nerf the mechanic, or just make the NPCs go to the statues more often so the bookkeeping is less relevant?  It’s very hard to know the answers to these questions, but after thinking about it for a while, and talking to the elite players, I decided I needed to deal with this in a few different ways, and it will be an ongoing iterative design challenge:

• I proposed nerfing the Sniper’s mechanic to a single level of highlight and a single level of lowlight.  Snipers could still use this as a 3-counter, but that’s less valuable, and each individual highlight or lowlight becomes more consequential.  On the flip side, it removes the cognitive load associated with trying to figure out if you’re going to single- or double-highlight, which is subtle but significant.  It’s also a lot cleaner controls-wise.  This was the first real nerf, and the beta community responded well to the idea.  There was some grumbling, but players posted thoughtful critiques and analyses of how they thought they’d be affected.  I realized I was going to have to test this thoroughly, but it was easy to isolate and I chose it as my question for the Depth Jam.
• I decided against changing the NPC behavior in the short term, but in the long term, instead of simply increasing the probability that NPCs will visit the statues (for example), I might make it so some NPCs end up going to the statues a minimum number of times.  However, either of these is a very complex change, because if an NPC is at a statue, it means they’re not at a bookshelf or in a conversation, so the entire flow of the party will change.   This is a very nonlinear change, and it’s hard to predict what will happen.
• I can also change the missions that require the Spy to be in “deductivity-susceptible situations”.  Keeping with the statue example, at the Depth Jam we ended up modifying the Inspect Statues mission so that instead of requiring three visits to the statues, the Spy could also inspect the neighboring statues if they’re not being held by and NPC.  This allows the Spy to trade off duration and number of visits, and requires Snipers to have a feeling for both these quantities.  There’s also a discount on the amount of time it takes to inspect each additional statue in a single visit, so you’re encouraged to inspect more than one.  Finally, it makes the middle pedestal the most valuable statue to visit, because it allows the Spy to inspect either side, so characters going to the middle statue are a bit more suspicious.  I love these tradeoffs, and these changes felt great when we tested them.

To give you an idea of how we went about testing this stuff at the Depth Jam, here is a screenshot of a special build we played.  In it, I had the computer automatically increment two counters over the characters’s heads, one for statue visits, and one for bookcase visits, so the Sniper didn’t even need to click.  This was never something I’d release to players, but I wanted to see how the game felt if the deductivity was “turned to 11″.  Especially before the Inspect Statues changes, you could definitely just shoot (or at least watch) the person with the highest numbers.

Shoot Danger P. Johnson, he's got the highest numbers! Except he's the Ambassador.

I’m very happy with both the single level of highlight/lowlight and the Inspect Statues changes, and they’re going to roll out in the next build.

Beta Balance

I haven’t run detailed metrics yet, since I’m scrambling just to keep things humming along for players, but I did get curious about whether the game is even roughly balanced right now.  Qualitively, I’ve been surprised by the seeming consensus in the private beta forums that the Sniper is the harder side to play.  In the private testing I did with Ian and Paul, it seemed like at elite levels the Sniper had the advantage, but I wasn’t seeing people complaining about this, which I found interesting.  Even more surprising was that players had settled on Pick 4 of 5 Missions on Ballroom as the balanced game mode.  Back in the day, Pick 3 of 4 Missions on Ballroom was considered balanced, and even then we were worried about a Sniper advantage at elite levels. 

Because SpyParty is so intensely player-skill oriented, I’ve implemented a number of game types so players can handicap matches to make up for skill differences.  You can tune the Spy’s difficulty by choosing modes with more or less missions to accomplish, and giving the Sniper more or less information about which missions will be enabled.  Some modes allow the Spy to complete any small subset of the missions chosen opportunistically while playing.  Other modes require the Spy to divulge exactly which missions he or she will attempt.  A large skill gap in favor of the Spy can be handicapped by choosing “6 Known Missions”, where choosing “Any 3 of 6″ makes up for a lot of skill on the Sniper side. 

Pick 4 of 5 is significantly harder for the Spy than Pick 3 of 4,¹ yet people were playing it as the default mode once they graduated from Beginner Ballroom, so I decided to run some quick numbers.

The first thing I did was looked at the results of all games ever played in the beta.  There are four possible outcomes of a game of SpyParty:  the Spy can accomplish the missions, the Spy can run out of time, the Sniper can shoot the Spy, or the Sniper can shoot a civilian.  We call the first and the last a Spy Win, and the middle two a Sniper Win.  The results were even more balanced than I thought they’d be!

Total Games: 13238, SpyWins: 6494 (49.1%), SniperWins: 6744 (50.9%)

Whoa.

Okay, so this is great, but it’s pretty silly, since this includes every newbie game, every game where somebody said “shoot me because I’m stuck on the briefcase”, and the like.  So, next I decided to look just at the elite games.  I defined “elite game” as one played between the people on the leaderboard with more than 20 hours of game time.²  That part of the leaderboard looks like this:

The top 8 players on the leaderboard by time.

I did a few different queries here.  First, all games these guys played in the last three weeks:

Total Games: 1794, SpyWins: 826 (46%), SniperWins: 968 (54%)

This includes teaching games, which these guys do a lot because they’re awesome, and everything else, so then I narrowed it to all games they played versus each other in the last three weeks, so these are the most advanced games going on in SpyParty right now…and these guys are very good…I know this because they beat me routinely:

Total Games: 266, SpyWins: 128 (48.1%), SniperWins: 138 (51.9%)

Wow, again, that’s just incredibly balanced!?  I couldn’t believe it when I ran these numbers, but I’m super happy with them, and it goes a long way towards explaining why buxx said this to me in chat:

checker: "so, you've played 43 hours or whatever, do you feel like there's a
ceiling for you yet, or still going strong?"

buxx: "Skillwise, not even close, I still feel like I have a ton of room
for improvement"

I just feel incredibly lucky to have a game this deep, this early in its development.  I’m so excited to take it even farther…people in competitive game design talk about “300 hour” games, and games like Go and Poker can be played for a lifetime.  I hope to take SpyParty to those levels, or as close to them as I can get.

Finally, I looked at these players versus each other at a finer grain.  I’ll just post some rough numbers for buxx and ardonite, because they’re an interesting contrast:

buxx is the top player right now, and he wins most of the time even against the other elite players, but especially as Spy.  ardonite also wins over half the time against the elites, but moreso as Sniper.

I can’t wait to delve into the metrics more in the future.

Todo

Due to the Depth Jam and other distractions, the todo list has been building for weeks.  I’m finally cranking through it, but I’ve dubbed this next build the “megabuild” because of all the stuff I’m shoving into it.  This is not the right way to develop software.  You should do more small releases, so you can test and isolate bugs, but I’m kind of on a roll and don’t want to slow down to make a build until I get some more stuff fixed.  I won’t do this entire giant list before releasing, but I will do a couple more days worth of work.  I’m afraid there are going to be bugs due to the number of changes, so it’s really the build after the megabuild that is going to be totally awesome!

• misc
• added matches and total match time to leaderboards
• alpha punchthrough for binocs and timed text…draw after
• kill “sleep too long” log message
• put graphics and cpuid info in log, at least glext
• pack cpuid into string in journal too
• don’t make duplicate logins delete cache and crash, stupid and annoying
• hittest laser to world for marking, except for window muntin bars
• mission text in practice sniper mode buggy
• terrible lowercase in username entry on login page
• s/affordance/action/g in code
• highlight/lowlight nerf
• ctrl/shoulder slam all the way to highlight/lowlight
• hysteresis on actions
• if it’s present in the current batch, set the index (unless it’s been manually set)
• if it’s not present, grey it out and fade it away
• this needs a lot of testing, it was a complex change
• clean up timeouts, wins, blah text everywhere in game
• remove debug_state_transitions
• check for pdf file association in setup program
• need network test during greeting
• invite beep in practice mode?
• don’t hide mouse during sniper menus
• put stats on match menu
• mouse sensitivity config prop and readme.txt
• do in-game update with a popup window instead of modal!
• sell subliminal advertising in the crowd walla
• endgame and results
• not top suspects, just high/lowlights, filtering down cast
• escape confirmation on results screen with settings
• can take out the eat stuff and timer
• make Last Results on the match menu? possible?
• test invite cancel
• OT confirmation, OT happens on sniper machine for a second
• output game result into log
• shoot after timer runout still records shot person
• Shot person on sniper screen but not on Spy screen
• escape from invite to game needs confirmation?
• if sniper escapes while spy is waiting for results, game doesn’t end
• need to record actions for spy when ai before control
• briefcase
• make briefcase a heavy pathing weight, not infinite
• briefcase making characters float off ground
• moving while putting down briefcase
• stuck in COURIER_WAITING_FOR_SCIENTIST_HOLDING when ambassador has a book…not moving
• need to put down the briefcase when the timer runs out!
• dispose is broken, courier won’t put it down!
• add pauses
• conversations
• fix interruption stuff to occasionally finish talk animation
• people in convo circles by themselves can read book until somebody else comes
• hmm, one frame of Read Book action, and immune to pause?
• ahh, the listen stomps it, need some kind of “interruptable” flag on anim, or priority
• can interrupt while taking drink…need to block
• people need to leave during listen idles?
• people going through middle of convo circles?
• double agent
• fake bbread not available when selected but not enabled
• banana bread double agent signal…if da leaves while action testing converts to fake
• need to have DA check to see if spy is action testing, and not leave convo if so
• how to do override in double_agent.cpp?
• double agent won’t go into circle during last minute – guarantee in last 30 seconds?
• fake banana bread is default option over flirt
• drink over conversation
• need to have cancel behavior on bbread
• okay – what to do here? person starting to talk is enough? short cough?
• bad – cough as walk away
• statue
• new inspect statues design, can inspect statues next to you if not held
• finish drink at or take briefcase to pedestal like bookcases
• play sound when teleporting statue back, sorry ardonite!
• swap wrong statue when between pedestals – /tmp/fraps
• two people pick up same statue
• bad statue action test is fastest, make it delay, or flicker longer
• statue swap cancel makes it disappear
• statue swap bad action test needs timer before swap happens
• maybe bad swap doesn’t actually swap, it toggles it and back?
• or, statue swap is too hard already so don’t make it worse?
• good swap statue action test
• need to prevent run out of time between good action test and swap?
• its obvious to elite snipers, worse than hiding body…
• make it swap when somebody else picks it up? too powerful?
• make it instantly warp, not alpha
• more than 5 seconds? cheesy way of buying time then…
• bug ambassador
• while watch check is happening, walking bug amb is not blocked
• ability to bug holding book in one hand
• bookcase
• fix bookcase pileup – do vertical expanded shape
• AIs wait a bit more randomly before leaving bookshelves
• marking book on shelf doesn’t record character because not attached yet
• do I want to support this even? have all books be grey on sniper?
• change put book and get book to have start/end events too
• idles in book watch animations, mix it up a bit
• second book page turn? or just use the idle for a time?
• make cycling books unified, right – blue, left – green, add red as double click, then grey
• bookcases vary in color per map…need to standardize and make maya prop type
• seduction at paintings
• chat
• window size, pos, borderless, maximized through chat commands
• add @ with username completion in chat
• color code names in chat
• kill “bogger says:” in single line chat copy
• history when typing chat line
• add crlf to chat copy so notepad works
• crash on non-text data in clipboard
• hitting escape kills the chat message you’re typing
• make it clear the first whisper went through even if got the away message
• packet loss test on console
• can’t run /fps command even without connection…but can do practice in lobby
• manual & readme
• put the latest manual images up, readme, etc
• s/noob/newb/ in manual…ugh
• documentation change to security agent
• remove y button
• inspect statues
• highlight/lowlight levels and controls
• add note about SpyPartyHelper.exe to the readme
• note lag is only on sniper side
• buxx feels watch check is underutilized…
• add more time?
• at paintings? in conversations while by yourself?
• watch check with book
• cancelling watch check needs failure or is animation tell enough?
• zerotka list of helpful things for research
• unlimited game time endless practice mode
• ability to see all floor pads in sniper view
• books come out same color as shelf
• Seduction Target in sniper view
• inability to complete missions or way to clear them
• no back walls in Ballroom as sniper ( or transparent if that’s easier?) ( aka, i want to be able to see in from all angles)
• camera track be a complete circle all the way around the ballroom.
• bugs
• intel problems
• lobby message box bad z
• key and mouse button drops due to shit framerate
• fix the rare junked game bug
• put more debugging code into the drinks.cpp coreanim crash
• store things to locals to see, do memory readable check on coreanimation
• phatjax crash in stun code, assert in log first
• 0.1.1983.0-phatjax-7cs4v-e9SBihV1znwyvuFw.7z
• lobbyserver leak – use backtrace
• inviting phatjax after cancelled invite gives assert
• assert !”what do to here?”, File: .\examples\lobby\lobbyclient.cpp, Line: 1798
• fix DTScale behaviors, briefcase, waiter, have game clock

1. Pick modes require the Spy to choose which missions to accomplish before the game starts, while Any modes let the Spy opportunistically choose during play.
2. Important note: this is actual game playing time, not time spent in the matches overall, or time in the lobby, both of which are way higher than this.  A lot of people like to hang out and chat in the lobby!

That said, I thought you’d like a taste of this server voyeurism yourself, so I asked arbaard and dbfclark if I could repost their chats from a series of games a few nights ago.  These chats are great, because they show two very good players (both were at PAX 2010 when I first showed the game, arbaard was in the tournament, and dbfclark is the person who noticed that the line watching the games was getting implicit Sniper practice).  You can see them helping each other after each round, discussing what went right and what went wrong, changing the game types as they handicap the games, and sharing information so they both can improve.  Plus, they’re funny.

I’ll do more posts from the beta soon!  I’m slowly inviting folks from the beta sign-up in now, and it’s working out great so far.  Lots of bugs reported, lots of bugs fixed, and people are playing a ton and seem to be having a great time!  Here’s the leaderboard as of right now, look at some of the play times and remember a lot of these people have only been in there a few days!

Seventeen and a half hours played in four days!? So. Awesome.

But, I have fixed most of the bugs that Jonathan Blow—the first official SpyParty Early-Access Beta invitee—found, so I guess I’m not going to wait for PayPal¹ and hope things are working now.

There have been a lot of questions here on the blog, on Twitter, and on Facebook about how the beta rollout is going to work, and a few misunderstandings based on ambiguous statements I’ve made, so I figured I’d try to clear all that up in this post.

First, and most importantly, everybody who signed up will get invited into the beta, it just might take a while.  Just getting to the point where I could send a single invitation email was way more work than I thought it would be, so it is going to take some time to get things to where I can invite in all 11,147 people who have currently signed up (as of this blog post).

I can, however, say how the very near future of the next few weeks is going to go…

Near-Term

Today, I need to finish writing up some basic documentation, including the Welcome Message in the private beta forums, the How to Report Bugs post in the forums, the Beta Test FAQ, and I need to update the game’s README.txt.  Hopefully I will grind these out in the next few hours.

Then, this evening, I’m going to send out a small number of invitations to my hardest-core and longest-term playtesters, folks like Paul and Ian.  This will be sent to only about 8 friends, all of whom are game developers and who have played a lot of SpyParty already, and who are very comfortable playing with very early software.  This group’s job is to play a lot, to find more subtle bugs—especially if I’ve broken any tuning since I haven’t playtested since PAX West 2011—and to make sure all the new score ranking and player database stuff works.  I’ll be fixing the bugs they report and implementing the features they request on a daily basis, in addition to playing a lot myself so I can stay even remotely competitive in my own game.

While these folks are testing, I’m going to finish the Invite-a-Friend feature, so they can test that as well, which will mix a few more people into the beta.

At this point it’s probably around March 1st, and I’m hoping the extremely-low-hanging bug/feature fruit will be mostly picked and everybody will be having a fun time playing and I’ll be ready to invite some more people in, including people I don’t know personally.  Unfortunately, this is also when GDC hits, which means I lose a week of productivity, so I’m going to hold off on new invites until after that’s over.  I’ll be doing a little SpyParty press at GDC, but I’m going to keep it pretty mellow.  I also might set up the laptops at some parties again, not sure.

After GDC, or maybe just before GDC if I’m feeling really confident and things are going swimmingly, I’ll invite in a small group of the earliest and hardest-core fans from the invite list, the blog, Twitter, Facebook, GDC, and PAX, and a small hand-picked batch of press who’ve been covering the game closely since the beginning.

Then the real bulk invites will happen shortly after that.  I’ll start with 20 people from the beta sign up list, mostly in order from the beginning, but also probably randomizing 25% of them from the whole list so at least a few people won’t have to wait quite as long to get invited.  Then, once the dust settles from those 20, I’ll invite 50, and then 100, and so on.

Less-Near-Term

At some point, my server will start melting from too many simultaneous players, but I don’t expect that to happen until I’ve got 1000 or more people invited in, so this first group should get in pretty quickly.  I will look at the bottlenecks and optimize the server, and that might get me to 2000 invited, or around 200 simultaneous players.²  By that point I want to have my server infrastructure scalable and running in the cloud, but that’s going to be a significant chunk of programming, so there might be a delay at this point in getting more invites out.

Finally, once the backend is scalable, I can start inviting larger groups of people in at the same time.  When I can send out 2000 invitations and everything runs smoothly as those people sign up, download the game, and start playing, it will be time to invite the rest of the beta list in, shut down the list, and let people join the beta directly without an invite. I think I’ll probably have a couple week period where I invite all the confirmed signed up people in, but don’t allow new signups, as a way of saying Thank You for signing up.

But, that’s all months and months away.  The Near-Term section above should be reasonably accurate, but this part is mostly me making stuff up right now.

1. Give me my $14.26, PayPal!  And yes, I am going to provide other payment provider options soon.
2. That is, if 10% is a reasonable estimate for peak concurrent users relative to registered users, which it seems to be.