Measuring Training

We security boffins love metrics.   We measure compliance, performance, response times, and roll them all up into pretty dashboards so we can document the efficacy of our programs.  Makes a lot of sense - why don't we do it for training?

Most organizations require all their users to undergo some form of cyber security awareness training, and most organizations squander their users' time and attention by trying to boil the ocean.

Rather than share practical skills for avoiding common online threat, the vast majority of security shops use their awareness training to test their colleagues on how well they can regurgitate the company security policy or jargon.  Under this model, there is no good way to quantify the effectiveness of the training, or to see which parts worked and which could use some improvement. More important, most of the training I've seen doesn't address real-world security issues that the organization is grappling with.  Instead, it deals mainly with policy and HR issues. Not only is it no fun to take, but it's impossible to know whether it does any good.

Instead of using a shotgun approach, what if you had your incident response team work with management to determine the issues behind the three or four most common real security incidents from the past year?  Things that can be counted, and are quantifiable in terms of cost, labor, and damage to reputation. Things like phishing, NSFW web surfing, and overly permissive file shares.

Now build a training module around each issue you selected. Explain the risks, what the problem looks like, and how to avoid and report it. Use anonymized, real-world case studies from your organization to illustrate the issue. Rather than bore them with acronyms and policy jargon, engage your students by talking about things they care about - like downtime, loss of privacy data and revenue.  This resonates: The reason the holiday bonus was smaller than usual was because we had to purchase Credit Monitoring for 10,000 customers whose data was stolen because of a phishing attack. 

At the end of the year, compare the number of incidents between the last two years.  If your training module is effective, the numbers around that incident category should have gone down.  If so, work with your incident response team to identify a new issue to target.  If you see the numbers for a particular incident type start to creep up again, you can rotate the corresponding module back into your training.

If, on the other hand, the numbers for a particular incident category aren't going down, or at least remaining static, that module may not be effective.  At this point, you have a couple of options:
Solicit feedback from your users as to how the module could have been more effective.
Try attacking a different issue. It could be that training just doesn't prevent that type of incident.

Over time, you will develop a library of training modules for all of your most painful security issues.  You can continue to expand and update them based on emerging threats.  And even if you don't like the results, you can still quantify the business value and ROI of cyber security awareness training, enabling your management to make informed business decisions about the program.