When we started brain storming a project for hack week, one of the floated ideas was to remake the 1983 film WarGames, and for lack of available space, a local lot with storage units was proposed. Over the course of the following years, while we planned, we realized that this whole idea would not be the most feasible, but it still felt like we were onto something.

Eventually, we settled with keeping the name and changing the scope.

Instead, we are proposing to create a virtual environment on which war games will be played, with a focus on storage -- in particular, with a focus on Ceph and its ecosystem.

What are war games?

Wikipedia has a few entries talking about the concept of war games, pertaining to military exercices and simulations. We are not going to go into much detail about what the whole concept is about for two reasons: 1) getting definitions 100% right is a deep rabbit hole that would consume our whole week, and 2) because "war game" is chosen mostly because it's a cool name.

In essence though, the concept relies on simulating adverse conditions, to develop, trial and refine possible solutions, without actual exposing the participants to real-life scenarios where failure would (potentially) be catastrophic.

In the context of software-defined storage, Ceph in particular, we are looking to leverage this concept to allow participants to develop their capabilities in recovering from cluster failures, as well understanding how failures are caused and how to prevent them.

General Overview

The ten thousand feet view involves two teams: Red, and Blue.

The Red team's takes the adversarial position, meant to cause as much trouble for the Blue Team as possible, while observing whatever constraints are established for the duration of the exercise. The Blue team's objective will be to keep a healthy, functioning cluster, thwarting Red's attempts at mayhem.

Each exercise will be bound by a set of constraints, defined before the exercise begins, and to be observed by both teams. For instance, if a constraint is "no data shall be deleted", then the Red team shall not delete the data on the disks. Remember, this is meant for people to learn, may it be by causing the problems or by fixing them.

The exercises will take place on virtual machines: a healthy cluster will be set up, with all the services that are meant to be running for a given exercise; there will be a login node, that shall be accessible by both teams. Teams will log into this node, and will issue their actions into the cluster from it.

Each team will have a predefined, non-overlapping time window to perform their actions on the cluster. Once the window closes, teams will no longer be able to login, connections will be booted off the node.

All actions, commands, shall be logged to a remote node, alongside with cluster health and other relevant information, for further analysis, postmortem, etc.