Email Security

Do Phishing Tests Work? Effectiveness and Best Practices

What is a phishing test and how effective are they in improving cybersecurity?

Phishing Tests Effectiveness

Phishing is one of the biggest threats that organizations of all sizes need to address. This is where a message (often email, though we are seeing SMS and other forms) is sent to an individual, encouraging them to click on a link or share sensitive information. To make the request more convincing, phishing messages will often masquerade as a trusted brand or individual. To combat this, organizations will invest in phishing training, teaching employees on what to look out for. This training can be followed up by phishing tests which involve a message being delivered to users that may display some of the hallmarks of a suspicious email, to test how a user reacts in their everyday life.

By running these tests organizations can effectively assess their level of vulnerability to phishing attacks, identify knowledge gaps, evaluate the effectiveness of training programs, and promote vigilance among employees. If a user flags the email as suspicious, they have taken the correct action. If they click on a link or even if they do not report the email, they could be endangering themselves as well as other users.

The effectiveness of these tests depends on whether they are providing realistic test designs, regular execution, and integration with broader security awareness initiatives. Poorly implemented tests may lead to mistrust or fail to produce meaningful improvements in behavior.

How Effective Are Phishing Tests?

According to KnowBe4’s 2024 Phishing by Industry Benchmarking Report, the average organization had an average of 34% employees that were prone to phishing during baseline assessments. This figure fell to 19% when the employees received 90 days of security awareness training, and after a year of training the number decreased further to just 5%. This illustrates the substantial improvements to phishing awareness and readiness that comes from dedicated training.

A 2021 study from the Department Of Computer Science At ETH Zurich found that:

  • Adding warnings on top of suspicious emails helps users detect phishing. No significant difference was found between shorter warnings and more detailed ones.
  • Even with repeated training, there will be a small number of employees that will click or fall for phishing emails multiple times.
  • If reporting is made easy for end users, then they will continue to report suspicious emails even after training has ended. Positive feedback encourages employees to report more.
  • This study encourages organizations to adopt phishing prevention tools like warnings, exercise caution around deploying embedded phishing exercises, and consider crowd-sourced phishing detection as a complement to other forms of protection.

2022 study following phishing tests conducted in a large hospital setting found the following:

Key advantages of running phishing simulations that researchers identified include:

  • Allow a broad workforce to be tested simultaneously without scheduling time away from work to take part
  • Ensure that the skills and awareness of the users are tested in a naturalistic setting 

Some of the drawbacks included: 

  • Users receiving a simulated phishing email are not typically made aware of the test in advance. The lack of informed consent can cause feelings of anger and distrust, which can negatively affect morale and productivity.
  • Employees may disengage from official future correspondence due to fear of being caught by a phishing test and / or fear of being sacked if they fail.

The results show that customization of phishing emails makes them much more likely to be acted on. In the first campaign, 64% of staff did not open the general phish, significantly more than the 38% that did not open the custom phish.

There was also a significant positive correlation between employee workload and phishing vulnerability. The higher their workload and fatigue, the less likely they were to detect the attacks. 

What Factors Can Make Phishing Tests An Attractive Option For Organizations?

  • Being able to collect tangible data on employees’ phishing performance helps highlight potential security risks and vulnerable users 
  • Cyber insurance providers typically require organizations to implement some kind of Security Awareness Training (SAT) to be covered
  • Phishing simulations that offer gamification elements can enable more engagement from end users

What Factors Can Drive Organizations Or End Users Away From Phishing Tests?

  • Inconvenience for end users – if tests are too time consuming, people will resist the task of completing them (Google’s security blog compares phishing simulations to fire drills).
  • Potential for end users to be unnecessarily upset or confused. Intentionally going out of the way to trick end users and punish them for failure can create distrust between users and their IT teams 
    • The Harvard Business Review cites one example of a bad phishing test conducted by GoDaddy. They sent an email to 500 employees a $650 holiday bonus back in December of 2020, but the surprise of receiving security training instead of a much-needed bonus resulted in backlash.

What Are The Best Practices For Conducting Phishing Tests?

Phishing testing is a key part of cybersecurity and security awareness, but it is important to ensure that any phishing test you include in your training programs are following ethical frameworks to ensure they don’t do more harm than good.

Some factors to keep in mind include:

  1. Keep ethical considerations in mind – the test should not cause as much disruption or distress as a real phishing incident.
  2. Understand the goals of conducting a phishing test in the first place and know that KPIs to look for. According to Hook Security, some KPIs that can be helpful for gauging phishing simulation success include:
    • Number of email clicks and opens, both per test and over time 
    • How many users complete their assigned phishing training 
    • Number of users that successfully report suspicious emails 
    • Trend lines for subsequent test results, to see how much employees improve on average 
  3. Instead of punishing end users for failure, consider positive reinforcement and turning clicks into learning opportunities.
  4. Ensure that the appropriate teams (like help desks) are notified that a phishing test is taking place; this can prevent some confusion and chaos later.
  5. Reward end users for taking positive actions such as reporting phishing.
  6. Making cybersecurity into a collaborative team effort is ultimately more productive than pitting end users and IT teams against one another.

Conclusion

Phishing tests are a popular tool because they offer organizations a proactive and cost-effective way to enhance cybersecurity awareness, assess vulnerabilities, and improve training programs.

By simulating real-world phishing scenarios, these tests help employees to better recognize and respond to emerging threats, thereby reducing the risk of a successful attacks. They also provide valuable insights into weaknesses in employee behavior and security protocols, allowing organizations to tailor their defenses and meet compliance requirements.


Overall, phishing tests serve as a practical tool for strengthening an organization’s security posture. To learn more about phishing and what solutions are best suited to protect your organization, read these relevant resources from Expert Insights: