3 Tips and a Trick for Building Awesome Assessments

Bill Sherman

Whenever I check Facebook, my feed seems cluttered with surveys asking questions such as “Which Game of Thrones character are you?” or “What house of Westeros do you belong in?” This clickbait tempts us. If we simply answer a few questions, then in a just a few moments we’ll learn whether we’re Arya Stark, Sandor Clegane, or Cersei Lannister. And whatever the result, we’re likely to smile and say “that’s me!”

These surveys are the latest iteration of ancient diversions, including horoscopes, fortunes and even the Magic 8-Ball. As long as the results contain a few vague favorable elements, we’re likely to see ourselves within the description—no matter how the results are generated. It’s called the Barnum Effect.

Assessments aren’t just popular on Facebook, they’re frequently used in organizations for critical tasks such as selection and talent development. However, a lot of developmental assessments organizations use simply aren’t effective and deserve an FDA black-box warning label that says “For Entertainment Purposes Only.”

Many thought leaders want to offer some form of developmental assessment within their program. While it’s easy to fool individual consumers with an assessment based on the Barnum effect, organizational customers have become a lot more sophisticated. Here are four questions you should be prepared to answer when organizational buyers ask about your program’s developmental assessment:

What was this assessment designed to measure?

An effective assessment should be built around a clear model.

If you can’t recite your assessment’s dimensions on cue, you probably have built an assessment that’s based on intellectual ambiguity. If you don’t know what target you’re trying to hit, then you’re almost certain to miss the mark.
However, score bonus points if your model’s dimensions are orthogonal and have clear operational definitions.

Is your assessment accurate?

An effective developmental assessment should measure what it claims to measure. You can’t use a tape-measure to take someone’s temperature.

Feel-good insights are nice, but an organizational buyer wants to know that your assessment isn’t based on guesswork and hunches.
If you haven’t validated your assessment’s accuracy, you may be claiming to measure a person’s temperature with a tape-measure.

Is your assessment reliable?

A reliable assessment produces consistent results, regardless of whether they’re close to the mark or not.

If you measure something under similar conditions, it should have similar results.
If your assessment has wide variance, it’s not very reliable.

Do you have a technical report?

If you get asked this question, you’re being interrogated by a sophisticated organizational buyer who wants to really understand your assessment’s validity.

Technical reports are written by trained professionals (such as organizational psychologists, psychometricians, and statisticians). They conduct formal statistical tests to validate your assessment’s accuracy and reliability.

If your developmental assessment lacks a technical report (or has weak findings), then you’re forced into a sales strategy where you hope your buyers are either gullible or unsophisticated. That’s just a poor business plan when today’s economic world increasingly relies on data science and hypothesis testing.

However, if your technical report shows that you have a strong assessment—one which is both accurate and reliable—then you can take pride in the asset. When you talk to prospective buyers about your program’s developmental assessment, be prepared to answer their toughest questions with a confident smile.

A Study in Psychology

In his classic 1948 study, psychologist Bertam Forer asked his students to take a personality test, but he provided the same “report” to every student. Each student was asked to rate how well the analysis described them. On average, students rated their analysis as 85% accurate. Likewise, it’s easy to read a horoscope and learn fortune will be coming your way, or that you undoubtedly belong in House Stark. Telling people what they want to hear might make for easy meetings, but it won’t solve any organizational issues in the long run, and believe me, Winter is coming.

Bill Sherman

Bill Sherman works with thought leaders to launch big ideas within well-known brands. He is the COO of Thought Leadership Leverage. Visit Bill on Twitter