This is the second of a three-part interview. You can read the first part here.
***
My understanding of agile development is fairly basic. I’ve never worked under the methodology, but have read a little about it here and there. What exactly is a technical debt backlog?
A backlog is a task list; but it is a prioritised task list that may get re-prioritised every two weeks (on sprint boundaries) and the teams only commit for a two week window (one sprint). A technical debt backlog is a subsection of the overall backlog and stories (tasks) which are interleaved with the general backlog.
Well, that doesn’t tell me a tonne, but I did a quick google, a bit more reading, and have determined that “Technical Debt is what makes code hard to work with. It is an invisible killer of software, and must be aggressively managed.” Based on that, I believe I understand one aspect of your job much better. Modernizing, bringing up to standards, some of the older code in the EVE Online codebase, such as what happened to Crimewatch last year.
I know any revamp of the old corporate and POS code is not on the development slate anytime soon, but how excited would you be if someone said “Let’s rewrite this and make it right!”
You may recall the discussions that happened around POSes recently; CCP Seagull handles communication on that subject. I could discuss the subject of technical debt but not in the context of POSes.
Fair enough. Let’s tackle this from a different direction. Crimewatch. By all accounts an old, very fragile piece of code. It was very difficult to work with, and most projects avoided interacting with it, because it could cause unforeseen problems. When CCP made the decision to rewrite this code, how involved were you in the process that focused on the new design? How much oversight do you give to projects such as Crimewatch to ensure that they are to your standards and that they don’t add to technical debt down the road? How happy were you when the greenlight was given to rewrite Crimewatch?
In terms of the actual technical design, not a lot, and not involved in the game design. The technical lead for the game play teams (CCP Atlas) and primarily the senior server programmer (CCP Masterplan) in the team that implemented the new system were the people in the trenches for the actual design work. My role was to highlight the fact that the old Crimewatch code was brittle, caution programmers and teams that ventured into that code and directly monitor their work, promote the idea that it should be refactored by demonstrating the cost the current system/code was causing us, and set the standards for implementation and performance testing (the QA Director is responsible for feature testing and general testing practices).
I was very happy when this project was finally greenlighted; it’s always good to be able to cross these things off the list, and then move onto the next system.
I’m finding the whole technical debt backlog part of your job fascinating, especially since it revolves around a lot of old, core EVE systems that the players find difficult to work with and/or would like to see refactored with better, more modern features. CCP has been careful on tackling these areas of old, brittle code.
Would the corporate role system fall into the Technical Debt Backlog?
To a certain extent, but mostly that system is a question of what it’s supposed to accomplish and from there possibly derive an overhauled game design. The code for that system is not in particularly bad shape.
“Not in bad shape,” in what respect? From a player perspective, the role system is difficult to work with, and things that people would expect it do, often have to be performed with various odd workarounds. (Kelduum has documented a few of these workarounds in his struggles get the corporate roles to behave in some basic ways.) I suppose that the code could be in “good shape” given what it actually was and was not designed to do. Most players would agree that it is in need of an overhaul. Is it in good enough shape for such an overhaul, were it given development priority?
I’m using “not in a bad shape” in the context of the Technical Debt Backlog from a purely technical aspect. What you are describing are usability issues in the system, what I referred to as “a question of what it’s supposed to accomplish and from there possibly derive an overhauled game design”. From a technical perspective then the code itself is not that bad, comparatively readable in the grand scheme of things and not badly structured.
What are some of the systems that fall into the Technical Debt Backlog?
The POS system, the in-game browser, performance improvements to client startup, performance improvements to dispatching physics simulation events to clients, performance improvements and refactoring of the attribute system; to name a few. There are other systems but they are either low-level or internal tools or pipeline. Some of these systems above fall into multiple other categories; such as the POS system has usability and design aspects, some of which we are addressing in Odyssey with Quality of Life Improvements.
Who makes the final decision on what Technical Debt Backlog items will be tackled?
Ultimately it’s the Senior Producer that makes a call on what the backlog for each release contains. She seeks input from various parties on the priorities and tries to balance the various technical and business needs. Items on the Technical Debt Backlog are of various sizes and therefore a smaller task may get done earlier (because it fits into the schedule) even if has less technical priority than a larger task. Where there will be significant changes in the game mechanics, such as with Crimewatch, this falls under the purview of the lead game designer.
Even so, you must still have a fair bit of input on those priorities. I would imagine the Senior Producer must rely on your expertise and experience with the Technical Debt Backlog?
Knowing how the Senior Producer needs to balance the different needs then I don’t send her a single prioritised list; rather I discuss the backlog with her and the relative importance and possible size of each project along with how doing certain Technical Debt Backlog tasks may enable other things for her and how not doing other certain Technical Debt Backlog tasks may possibly “paint us into a corner”.
Are Technical Debt Backlog items handled by a particular team? Or are they handed out to teams based on which can best deal with them (i.e. team expertise)
They are handled by all the teams, although Team Gridlock has been involved in only Technical Debt Backlog tasks, as fits the rest of their backlog and expertise.
Are Technical Debt Backlog items tackled on an expansion-by-expansion basis, or are they simply ongoing, and not generally tied to a specific expansion cycle?
Both.
What Technical Debt Backlog items were tackled for the Odyssey expansion?
To name a few: We are improving patching (there has been a low number of failures when using HTTP/1.0 proxies), rewriting the Image Export Collection generation process, and revamping error handling and logging in the EVE API as well as the deployment method of the API and updating its internal caching mechanism (local and distributed.)
