A Complete Guide to Designing a Tabletop Exercise – Evaluate (4/4)
A Complete Guide to Tabletop Exercise Design
I’ve distilled what I’ve learned from designing exercises for some of the most elite organizations in the world into this comprehensive tabletop exercise design guide. Join me as I plan, design, execute, and evaluate. I’ll even simplify your process with free pre-built document templates.
For an exercise to have any impact, there needs to be a change of course. The After-Action Review (AAR) will guide those course corrections through data and observation.
Note: The formal AAR process is different from the immediate hotwash you did with the participants in the after execution.
Your final document will list your purpose and objectives and detail the path of the discussion. It will also identify areas to improve and steps to do that.
STAKEHOLDER REPRESENTATION
The evaluation phase doesn’t need to (and probably should not) include all exercise participants. Include the key decision-makers from the departments that participated. They will be responsible for their team’s improvement plans, so involve them in this step.
Data Analysis and Synthesis
Coding Feedback
I start with the participant feedback. I usually have a standard form with set questions. I list all the feedback under each question and combine similar comments. I then move to observations made by facilitators, controllers, and evaluators.
Synthesizing Observations
Once I have a reduced list compiled, I organize the feedback so it is helpful to the process. You don’t want to miss any key points. There should be two ‘categories’ of feedback.
One category is for you as you develop future exercises. These notes don’t necessarily need to make it into the formal AAR, though you could add an apendix if you’d like. They discuss things like the venue and the technical delivery of the scenario.
The other feedback is relevant to the outcome of the training. Things like collaboration challenges, team dynamics, decision-making processes, and the overall training experience.
After-Action Review Document
I have an AAR Google Doc template that you are more than welcome to copy and use.
This document provides a comprehensive overview of the execution’s outcomes. You will distribute this final AAR document to stakeholders to implement mitigation steps and improvement plans.
Objective Grading
Your evaluation should have a dedicated section that lists each objective with a box to mark the completion status. Note if the objective was 1) completed without challenges, 2) completed despite challenges, or 3) not completed due to limitations or challenges.
I find it easiest to insert a four-column table into your document. One column is ‘Objective,’ and the other three are simple checkboxes on the ‘status’ of the objective (listed above). See page 2 of my AAR template to see what I mean.
This should set the tone for everything that follows. If an objective was not fully met, you will need to outline the remedial steps to fill the gap in the future. The grading block is a quick reference that provides a high-level assessment of the organization’s readiness.
Observation Data
Go objective-by-objective. List the objective at the top of the page and place all the details below it.
Strengths
Take all the constructive feedback and reduce it to a paragraph overview, highlighting the most important aspects. Even if the participants failed to achieve the objective, there should be helpful takeaways from the discussion.
After the summary of strengths, take time to outline specific strengths in a list. Again, highlight key details here. (Don’t make a list of 30 strengths to go through.)
Areas for Improvement
Now, write a similar summary paragraph for areas of improvement. Below your summary, list the specific areas and provide details on what went wrong. You’ll analyze the outcome in just a moment.
Clear documentation explaining why is especially important if the objective was unmet (e.g. What challenges caused the failure?).
The tabletop AAR document will likely alter the crisis response plan to adjust for lessons learned.
Analysis
When a crisis occurs, each stakeholder has a specific role to play. If conflicts are not resolved and plans between the departments are unclear, you will struggle during a crisis.
Beneath each Area of Improvement note, record an analysis of your observations. Detail blockers that prevented the participants from meeting the objective or hindered them from doing so. Be detailed here because you’ll use this to craft the improvement planning section that comes next.
The more detailed you are here, the easier the improvement plan will be.
Improvement Plan
As stated previously, an exercise is ineffective if improvements are not implemented as a result. The section of your AAR will outline–in detail–what steps the organization must take to improve.
You have already listed the areas you need to improve. Now, you need to determine how the organization will adapt to fix the discrepancies.
Corrective Action
This is where close collaboration with key stakeholders is essential. You may need to lean on them to detail specific mitigation steps for your report.
The important part of this section is that each improvement step is actionable. There needs to be a clearly defined ‘end state’.
You won’t see significant organizational improvement if your corrective action is ambiguous.
Organization and Point of Contact
With each corrective action, identify the organization responsible for the implementation. Again, ambiguity in who is responsible will lead to finger-pointing and blank stares.
Additionally, listing a specific point of contact will instill greater accountability. If someone is confident they will receive a phone call with a request for an update, they are more likely to follow through with the action steps.
Deadline
Robert Harjavec once said, “A goal without a deadline is just a dream.” That rings true for your improvement plan. Put a calendar date on each improvement step and hold your teams accountable.
Depending on the complexity of your organization and the depth of your improvement step, you may want to set checkpoints. If your deadline is six months away, a bi-monthly check-in may help keep things on track.
Conclusion
Much more could be said about designing a tabletop exercise. As I said at the beginning, this is not THIE way to design an exercise, but A way.
Take what works and leave what doesn’t.
The most important part of the exercise is that you do it. You will forget things in your planning, miss an inject in the design, fumble during execution, and your AAR will not be specific enough. All those things will happen.
Let them. Once they happen, you can improve.
The danger of exercise design is making sure it’s perfect. Chasing perfection moves the priority from preparedness to a great exercise. Don’t sacrifice your organization’s preparedness for your ego.
Can I Help?
My goal in this series is to make tabletop exercises achievable by all organizations. Resiliency doesn’t happen by accident, and every organization has the opportunity to exercise.
However, I realize designing and facilitating exercises still requires time and effort. I would love to partner with you! I have been doing exercise design for more than a decade. Sometimes, it helps to have a pro in your corner.
