Gamification – Ben Williams, Sten10 Event Report

15th May 2019 in Manchester, we saw ABP Chair Ben Williams, Sten10 present Gamification: benefits, progress and challenges

The costs of hiring have escalated so much in recent years that it is inevitable that recruiters are looking to increase the efficiency of hiring.  Increased computing power and the ability to process ever larger amounts of data has meant that in many circumstances there is a greater correlation between data gathered objectively during assessments and job performance, and that this trend will gather pace, predicted Ben Williams at the May meeting of the Manchester section of the ABP,

The concept of Gamification arose from some work done by TMP in the area of web based assessments, where they were trying to personalise online activity to supplement information on candidates obtained elsewhere.   Since then a number of providers have entered the market trying to offer superior and more highly targeted products which are “Games Based Assessment” tools.  The traditional games based assessment products have been supplement by standard “off the shelf” materials along with custom built products with a diversification into Virtual Reality (VR) which use “meta-data” to determine behavioural choice.

So what are we trying to assess?  GBA tools are useful at addressing:

Abilities:

• • • Cognitive processing speed
    • Attention span
    • Working memory
    • V, N, A reasoning

Personality traits:

• • • Persistence
    • Risk propensity
    • Emotional Intelligence

‘Role-Fit’ – A.I. % match

Most Millennials and younger people have had experience of computer games so they are used to the electronic playing environment.  It is not the game which is important, rather, it is providing an environment where they are relaxed and feel that they are not actually being tested.   The concept of VR is currently being introduced into assessments and has huge potential because it takes the idea of “immersion” much further.

From a cost and efficiency perspective, a whole new opportunity has arisen whereby personality measurement can potentially be undertaken without human involvement.

Games publishers however are not finding it easy to become established in a mass market and to find their niche.  There a number of reasons for this:

• A New Market – GBA Test publishers are quite young meaning evidence of predictive power is limited by necessity
• Generalisations about the evidence base are difficult compared to ‘traditional’ psychometrics due to the variety of design
• Objectivity – Investigating GBAs objectively is problematic as commercial IP is tied up in the algorithms used. Also, most research being funded and facilitated by the publishers themselves
• Common method variance – using GBAs changes the way constructs are measured (construct validity). This is a real challenge when clients want to link the information acquired to a Big 5 assessment and achieve an acceptable level of correlation.
• Complex – not only raw score but thousands of meta-data points are measured

And to add to this, there is little in the form of clear academic review, looking at the challenges of and the properties of games based assessment.

There is also much discussion about what a “reliable assessment” is.  Correlations have to consider where the sources of error measurement might lurk, and whether there is internal consistency and/or consistency over time.

Sten10 has arrived at aggregated data from a number of games publishers and concluded that internal consistency mirrors results from traditional assessment but some work still has to be done in the area of consistency over time (0.57) and specific results for subtests.    Areas which need addressing are the length of assessment which can be too long for some people, hand eye coordination in visually complex assessments and distortion resulting from direct observation in the assessment situation (this can preclude accurate assessment on levels of extraversion and agreeableness).

Those reviewing the results of GBA need to be mindful of anxiety, perception of fairness, enjoyment, the familiarisation with technology and gaming expertise, and whether GBA will encourage or discourage job applicants from accepting offers.

Allied to this is of course the need to stay on the right side of the law and four studies were undertaken to review adverse impact in sensitive areas, including Age, Gender, Ethnicity, Disability (after WM adjustment for dyslexia), Gaming experience, Handedness, Screen size.  These reviewed little or no adverse impact but it has to said that the sample sizes were small (no more than 5000 for the maximum sample).  It is clear that as one of the tools for improving the overall reliability of assessment, GBA can make a considerable contribution.

In conclusion, much work has been done to progress the reliability of GBA as a contributor to assessment, and of course there are huge potential cost savings.  Construct validity and parallel form reliability is largely positive.  However, there is a lack of peer viewed and academic research, much evidence is based on marketing material from the developers, and the practice of gamification has far outpaced researcher understanding of processes and methods.  The advent of new VR technology is an unknown quantity but will no doubt enhance the quality of results and prediction of performance.

Ben’s slides are available to download here.

www.sten10.com

Report by Richard Taylor.