A Complete Guide to Designing a Tabletop Exercise – Evaluate (4/4)

For an exercise to have any impact, there needs to be a change of course. The After-Action Review (AAR) will guide those course corrections through data and observation.

Note: The formal AAR process is different from the immediate hotwash you did with the participants in the after execution.

Your final document will list your purpose and objectives and detail the path of the discussion. It will also identify areas to improve and steps to do that.

Data Analysis and Synthesis

Coding Feedback

I start with the participant feedback. I usually have a standard form with set questions. I list all the feedback under each question and combine similar comments. I then move to observations made by facilitators, controllers, and evaluators.

Synthesizing Observations

Once I have a reduced list compiled, I organize the feedback so it is helpful to the process. You don’t want to miss any key points. There should be two ‘categories’ of feedback.

One category is for you as you develop future exercises. These notes don’t necessarily need to make it into the formal AAR, though you could add an apendix if you’d like. They discuss things like the venue and the technical delivery of the scenario.

The other feedback is relevant to the outcome of the training. Things like collaboration challenges, team dynamics, decision-making processes, and the overall training experience.

After-Action Review Document

I have an AAR Google Doc template that you are more than welcome to copy and use.

This document provides a comprehensive overview of the execution’s outcomes. You will distribute this final AAR document to stakeholders to implement mitigation steps and improvement plans.

Objective Grading

Your evaluation should have a dedicated section that lists each objective with a box to mark the completion status. Note if the objective was 1) completed without challenges, 2) completed despite challenges, or 3) not completed due to limitations or challenges.

I find it easiest to insert a four-column table into your document. One column is ‘Objective,’ and the other three are simple checkboxes on the ‘status’ of the objective (listed above). See page 2 of my AAR template to see what I mean.

This should set the tone for everything that follows. If an objective was not fully met, you will need to outline the remedial steps to fill the gap in the future. The grading block is a quick reference that provides a high-level assessment of the organization’s readiness.

Observation Data

Go objective-by-objective. List the objective at the top of the page and place all the details below it.

Strengths

Take all the constructive feedback and reduce it to a paragraph overview, highlighting the most important aspects. Even if the participants failed to achieve the objective, there should be helpful takeaways from the discussion.

After the summary of strengths, take time to outline specific strengths in a list. Again, highlight key details here. (Don’t make a list of 30 strengths to go through.)

Areas for Improvement

Now, write a similar summary paragraph for areas of improvement. Below your summary, list the specific areas and provide details on what went wrong. You’ll analyze the outcome in just a moment.

Clear documentation explaining why is especially important if the objective was unmet (e.g. What challenges caused the failure?).

The tabletop AAR document will likely alter the crisis response plan to adjust for lessons learned.

Analysis

When a crisis occurs, each stakeholder has a specific role to play. If conflicts are not resolved and plans between the departments are unclear, you will struggle during a crisis.

Beneath each Area of Improvement note, record an analysis of your observations. Detail blockers that prevented the participants from meeting the objective or hindered them from doing so. Be detailed here because you’ll use this to craft the improvement planning section that comes next.

The more detailed you are here, the easier the improvement plan will be.

Improvement Plan

As stated previously, an exercise is ineffective if improvements are not implemented as a result. The section of your AAR will outline–in detail–what steps the organization must take to improve.

You have already listed the areas you need to improve. Now, you need to determine how the organization will adapt to fix the discrepancies.

Corrective Action

This is where close collaboration with key stakeholders is essential. You may need to lean on them to detail specific mitigation steps for your report.

The important part of this section is that each improvement step is actionable. There needs to be a clearly defined ‘end state’.

You won’t see significant organizational improvement if your corrective action is ambiguous.

Organization and Point of Contact

With each corrective action, identify the organization responsible for the implementation. Again, ambiguity in who is responsible will lead to finger-pointing and blank stares.

Additionally, listing a specific point of contact will instill greater accountability. If someone is confident they will receive a phone call with a request for an update, they are more likely to follow through with the action steps.

Deadline

Robert Harjavec once said, “A goal without a deadline is just a dream.” That rings true for your improvement plan. Put a calendar date on each improvement step and hold your teams accountable.

Depending on the complexity of your organization and the depth of your improvement step, you may want to set checkpoints. If your deadline is six months away, a bi-monthly check-in may help keep things on track.

Conclusion

Much more could be said about designing a tabletop exercise. As I said at the beginning, this is not THIE way to design an exercise, but A way.

Take what works and leave what doesn’t.

The most important part of the exercise is that you do it. You will forget things in your planning, miss an inject in the design, fumble during execution, and your AAR will not be specific enough. All those things will happen.

Let them. Once they happen, you can improve.

The danger of exercise design is making sure it’s perfect. Chasing perfection moves the priority from preparedness to a great exercise. Don’t sacrifice your organization’s preparedness for your ego.