Phishing Simulation Difficulty: Match Scenarios to Learning Goals

A higher click rate than in the previous phishing simulation does not, by itself, show that employees have become less capable. Check separately whether the email was harder to recognize, the audience changed, or the measurement method differed.

Start with the behavior you want people to practice: noticing suspicious details, reporting to the correct channel, or asking for help promptly after interacting with a message. The right scenario and interpretation of the results depend on that objective.

This guide helps training owners consider the purpose, the recipients' work, observable clues, and the handling of questions when choosing a scenario.

The same email can have different difficulty for different jobs

An invoice email may feel routine to someone who handles invoices and unusual to someone who does not. Difficulty depends on how closely a message fits everyday work, as well as whether its writing looks polished.

The NIST Phish Scale assesses how difficult a phishing email is for a person to recognize by considering its observable cues and how well its premise aligns with the recipient's work or circumstances. It also emphasizes the importance of assessors understanding the intended audience. NIST Phish Scale User Guide.

It is not a single score for ranking employees' abilities. Understanding message difficulty helps interpret click and reporting rates.

When selecting a scenario, ask someone familiar with the target department whether similar messages normally arrive and how employees would usually verify them. This makes the difficulty assessment more specific.

Choose the behavior to practice in this exercise

In Yagura's implementation support, one plan used the first exercise to help people learn the report button. Another combined more challenging exercises with easier exercises designed to encourage reporting.

These plans gave individual exercises distinct roles based on the behavior to check. Making every scenario difficult was not the goal.

An exercise for learning the reporting process

If people are learning the reporting channel or button for the first time, extreme realism is not necessary. Let them experience where to report, what happens after submitting, and how to ask for help even if they have already interacted with the message.

Successfully reporting in this exercise and recognizing a highly convincing message are different outcomes. Stating the objective explains why a simpler scenario may be appropriate.

An exercise for handling an ambiguous message

Once the reporting process is familiar, you can test what employees check when a message resembles their everyday work.

Provide learnable clues and a useful explanation afterwards. Go beyond telling people they were caught: explain what to check and whom to ask when uncertain. Choose difficulty based on the audience's experience and the exercise's purpose, rather than fixing it solely by whether this is the first or a later round.

Assess difficulty and business impact separately

A work-related scenario can be challenging without disrupting real work. Attempts to verify a training email may lead employees to contact an actual supplier or public institution.

IPA explains that using real or similar organization names in exercises can generate third-party inquiries or reputational effects. Training designers should plan the verification route and potential impact while supporting the act of checking a suspicious message. IPA's explanation (Japanese).

The following checklist is proposed by Yagura. It organizes objectives and delivery conditions; it does not replace the NIST scoring method.

As a hypothetical design, people unfamiliar with the reporting process could practice using the channel, while people who know it could practice verifying work-related messages. This distinction should reflect experience and learning needs, rather than age or seniority.

Check busy periods and overlap with important real announcements. Adjusting the audience or timing while preserving the learning objective may be more appropriate than increasing business disruption to obtain a result.

Separate difficulty from measurement conditions when comparing results

Comparing click rates or similar metrics requires two checks.

First, are you measuring the same behavior in the same way? Opening a message, following a link, entering information, and reporting are different actions. Message format, image-loading settings, and the treatment of automated access can change what a recorded number means.

Microsoft explains that security products inspecting its simulation messages can sometimes produce recorded clicks without a user taking that action. This is a measurement issue separate from scenario difficulty. When a metric changes unexpectedly, check the records and measurement conditions before evaluating employees. Microsoft's official FAQ.

Yagura's implementation discussions have also included choosing other metrics where message opens are difficult to measure, and requests to distinguish opening an attachment from subsequent actions. Missing records should not automatically be treated as proof that an action did not occur.

Second, once measurement conditions are comparable, how did the scenario or audience change? A message closely aligned with people's work and a message containing more obvious warning signs can produce the same click rate with different implications.

Keep separate records for the following:

A lower click rate after making a scenario easier does not by itself demonstrate better learning. A higher rate with a harder scenario does not necessarily show a decline in capability. When conditions differ, explain the behavior verified in each exercise instead of ranking the rounds on a single scale.

Choose the scenario together with the follow-up explanation

A scenario that looks useful before delivery may offer little guidance afterwards if the explanation is unclear. Prepare the explanation when selecting the message.

Show the suspicious cues, how to verify a legitimate communication, and where to report uncertainty. Move beyond spotting spelling errors and explain what could be checked in the actual work context.

In Yagura's support work, organizations have raised concerns about wording sent to people who did not open a message and about assigning learning materials after a false click classification. Even when everyone receives the same guidance, avoid asserting an action that the records cannot reliably establish.

For example, explain the purpose and verification points instead of telling every recipient “you performed a dangerous action.” Provide a contact for questions about the measurement. Describe the value of the learning materials without presenting them as punishment.

Check five points before sending

Before approving the next scenario, summarize these points on one page:

• What behavior should people practice?

• How closely does the scenario relate to the audience's usual work?

• What clues can recipients check, and what will the follow-up explanation teach?

• Have the contact route and business impact been reviewed?

• What will be measured, how, and which conditions differ from the previous exercise?

Adjust difficulty to practice and verify the behaviors people need. Use the results to decide what to teach next, and include that decision in the exercise design.

For the full delivery process, see our guide to targeted email simulations. For training and education support, explore Yagura Awareness.

For ownership and completion criteria after a report arrives, see our suspicious email reporting workflow.

References

• NIST Phish Scale User Guide, NIST TN 2276, November 2023 / PDF.

• IPA, Clarify the purpose of targeted email exercises in your organization, July 31, 2017 (Japanese).

• Microsoft Learn, Attack simulation training deployment considerations and FAQ.