Presentation by Dr Stewart Desson on research undertaken by Dr Stewart Desson of Lumina Learning, Dr. Joana Suta, and Dr. Tatiana Schifferle Rowson. 4414 people completed a 20-30 minute questionnaire from all over the world, with the objective of a comprehensive analysis of how Covid-19…
Chris Small – Vice President PSI
Any consideration of the evolution of Assessment and its current state requires a more joined up way of thinking, argued Chris Small of PSI at The ABP event at the Linnean Society.
PSI is a client focused assessment business with two core business areas:
* Talent Assessment, career assessment and guidance
* Certification and qualification in areas where skills are used most days, such as professional and digital skills.
PSI is a technology solutions business, an assessment “supermarket” providing a single gateway to 550 testing tools. Using innovative technology, PSI offers leading Assessment content and specialised Expertise and Support in talent acquisition and HR management. Now the official technology partner of The ABP, PSI provides access to its assessment platform to all ABP members.
The changing landscape of assessment and technology
The relationship between assessment and technology has certainly changed emphasis over the last 15 years. In 2000, assessment was about simple comparability to highlight differences between candidates and although the theme remains unchanged, the means of delivery has changed fundamentally, with the focus shifting from desktop to mobile. A more subtle yet fundamental shift has been the gradual acceptance of unsupervised assessment: back in 2000, most organisations were only beginning to accept the benefits that could be realised through large scale remote testing. Today unsupervised testing in the talent assessment space accounts for 99% of all delivery.
The complexity of assessment delivery has dramatically evolved given increased scale and technology demands. Chris highlighted one aviation client who delivers over 2 million assessments per year, uses 12 separate assessment vendors and delivers across 5 technology integrations. Working at this scale and with sophisticated technology expectations 4 key client themes emerge-
* Use of technology for scalable solutions
* Tailoring of individual solutions
* Use of Data
* Candidate experience and feedback
These challenges can be met by streamlining gateways and infrastructure simplification, with a key driver being centralisation of data. Having data in one place is a core need, although the use of historically different systems preclude this at the moment.
Candidate experience is king – and has commercial impact
Improving the candidate experience is a core concern for most recruiters. In these days of near instant sharing of information, the commercial impact of alienating individual customers can be considerable
The case of Virgin Media illustrates the point: 130,000 candidates were assessed over a 12 month period, with18% of these being existing customers. Of these rejected applicant customers, 35% cancelled contracts. This is equivalent to 7,500 lost customers and, critically, £4.4 million in lost revenue. Put another way, for every one person they hired they lost 2 existing customers. The challenge is to turn the candidate experience into a revenue generator.
Aligning technology and brand values
Technology should and does resonate with candidates. Having a quirky brand is a key attraction for candidates, but organisations struggle with reconciling the conflict between values and technology. Any outputs from the recruitment process need to resonate in a positive way with the organisation – and clearly in an unacceptably high number of cases they don’t. A solid example of aligning technology and brand comes from Amazon where they cherry pick service content in order to provide personalisation of the assessment experience and have as many as 130 platforms to tailor the candidate experience.
New ways of boosting candidate experience
One common bugbear amongst candidates is the lack of formal candidate feedback – this often manifests itself informally in a negative fashion on social networks. With 37% of candidates still believing that they are more likely to win the lottery than get any meaningful feedback on their applications, the solution is to make technology “sweat” to work in this area.
One solution is some kind of “self development” platform and linking assessment results to a feedback platform.. In practice this means speedier feedback and improvement in opportunities for career development. More 18 year olds are applying for more jobs and an assessment platform is their first touch point with the world of work. Solutions such as “Sokanu” can integrate career guidance with the applications process and are having a mitigating effect on the inevitable candidate rejection process.
Another way to enhance candidate experience is through “immersion” or a “simulation” experience of what the job entails. For example, Lloyds Bank are experimenting with the use of Virtual Reality. As with any new practice, there is a balance to be found and it’s important to consider the ‘degree of Immersion’ versus the ‘acceptability of the Assessment Method’. Clearly a paper-based assessment provides no real-world practical experience whilst a virtual reality or immersive process increases and improves the candidate experience. However, care must be taken to ensure some technologies don’t create unacceptable and unethical processes and result in negative feedback.
And finally: embrace technology but make it acceptable and ethical
So, in summary:
* Technology has a critical role to play in assessment and development delivery and evolution
* Opportunities are always there for technology to disrupt tradition assessment delivery
* Candidate experience and acceptability are critical drivers, but we should not lose sight of the fundamental aim of assessment- identifying high performance and potential
Alan Bourne- Founder, Sova Assessment
Amidst all the concern about markets and companies competing to offer ever better candidate experiences, the one area which is often forgotten is the candidates themselves. Companies have resources to update their profiles: candidates are served up with whatever companies offer, argued Alan Bourne of Sova Assessment.
Young candidates with little experience of the job market are served up with the following:
* Clunky and jarring experience little realistic preview of the role
* Disjointed and fragmented experiences of the world of work. They are assessed on their behavioural and technical skills in an unstructured and mesmerising way.
Companies are becoming increasingly cynical about face to face assessment which they see as expensive and standard assessment centres a weapon of mass rejection! But they ignore this “expense” at their peril. It is estimated that for many jobs in large companies candidates have as many as 300 touchpoints with this single organisation. Rather than seeing this as a problem which they can wish away, they should be investing in making the experience frictionless. By applying more imagination there are several dimensions to how the candidate experience could be improved. Some ideas could be:
1. Convergence : many people apply for the wrong job and some candidates would benefit from the application of other data about themselves to help them help themselves
2. Personalisation: when a candidate clicks applies they might see “If you like this role, you might also like…..”
3. Technology enablement: the experience of Uber could be extended to recruitment by offering a geo location of job opportunities
4. Consumer grade candidate experience: the operators of Tinder could offer some advice on how to interface with candidates effectively.
Companies also owe it to the next generation to be more honest and be aware of the digital impact of work and rise to meet the challenge of the digital impact on the world of work. We need to all face the facts:
* 30% of all existing UK jobs could be automated by 2030
* new jobs will likely arise to take their place
* 47% of jobs will be lost in Transport, Manufacturing, Wholesale and Retail
* The lowest impact will be felt in personal contact occupations such as education, health and social work
What skills and capabilities will candidates need to demonstrate in the future?
* An increasingly complex world will demand more complex problem solving skills and creativity to apply these skills to new situations
* Emotional intelligence and resilience to cope mentally with an increasingly challenging and fast paced working environment.
* Cognitive and behavioural flexibility, to accommodate new patterns of working.
There will be a decrease in the need for routine customer service skills and management, routine activity and provision and processing of information which is currently regarded as lower level professional activity.
The other big change under way can be summed up by “Internet is Mobile”. By 2030 80% of the internet will be mobile. Already 20% of millennials do not own a computer and use only mobile to access internet. The implications are huge and not only technological: expectations of efficiency and speed will continue to increase.
In China already over two thirds of internet connections are via mobile as it leapfrogs the technology which provides fixed line access. The technology is already available for us to make our own choices – we need to adapt to make it work for us. It’s here and we have to respond.
Scope of Digital Assessments
Costs of hiring will continue to increase as will pressures to “Get it Right”. Those conducting assessments will have to pay as much attention to candidate communication as to content, and use technology to minimise costs:
* Realistic Job Preview
* Online assessment
* Video interview will become more frequent, especially in early selection processes
* Digital Assessment Centres and simulations
* Onboard experiences rather than pen and paper exercises
* Capacity for development
* Predictive analytic, using finely tuned different criteria for different jobs
Pressures will on reducing assessment time for each candidate and costs resulting in
* Blending and convergence of assessment techniques
* Assessing and processing on mobile devices the evidence
* Expectation of mobile option for applications. The application process should be a maximum of 15-30 minutes, not one hour.
* Finding solutions to complex on screen tasks on mobile. Cognitive questions need to be designed to work on the smaller screen
Some complex simulations are not suitable to mobile at present but mobile functionality and capability will continue to increase.
What are the key objectives of Digital Assessment?
* Increasing precision in using candidate experience to predict job performance
* Reducing costs by using proven off the shelf tests and mixing these with relevant bespoke content.
* Increasing reliability in the context of the job and activity : the assessment should be adaptable
* Assessing a mix of targeted and generic norms
* It should address the diversity agenda, and assess social mobility and inclusion
* It needs to look at localised validity and impact as well as linking to a braoder context
* It should measure the breadth of capabilities while taking a focused approach to cognitive skills
Different tests will be needed to assess blended content as needed and new skills will be needed to make decisions on the use of bespoke content. Already users are developing and using their own platforms and ACs are having to adapt to using different platforms. The danger is loss of precision and ACs will have to become increasingly adaptable in working across different platforms.
Further challenges however lie ahead. Technology can not manage issues such as:
* Valid predictions and equivalence in assessment with related activities
* Validation of content
* Rater reliability
* Compensating for unconscious bias
But technology can be used to work to the following guidelines:
1. Solutions must have the necessary precision
2. Solutions must be responsive, device independent and easily accessible
3. Content should have technological equivalence
4. Readabilty on mobile is vital; scrolling should be avoided if possible
5. Provision of Cognitive Questions to fit small screens
6. Appropriate use of video – interpretive use of video expressions
7. Local validation is vital, especially in relation to being clear as to what is being measured, and giving candidates a fair and reasonable experience. It is useless to devise some tests where for example 51% didn’t complete the assessment part of questions.
So, in conclusion, where are we heading? In short, “Mass Customisation”: using increasingly sophisticated platforms it is increasingly possible to devise sophisticated bespoke assessments, with some being provided in house and others externally on assessment platforms. As in other areas of technology application it is therefore essential from the point of view of candidate experience and efficiency we need
* A new set of Digital Assessment Standards
* Standards should cover all types of digital assessment
* A review process to provide a reality check
* More localised validation techniques to ensure at least acceptable levels of accuracy.
Professor John Rust – Director of Psychometric Testing centre: Cambridge University.
The advent of the online digital footprint has thrown up huge challenges, both technologically and ethically, argued Professor John Rust on the occasion of the Digital Assessment debate at the Linnean Society on May 4th. We need to either communicate and collaborate, or submit to a George Orwell type “Big Brother” takeover by powerful minorities who control vast databanks of information about each one of us.
Psychometrics is not a new subject. It dates back to 1890 when an interest in the predictions of behaviour took root. The link between behaviour and performance at work soon became apparent.
A defining moment came in 1948 when two novels were published which took entirely opposite viewpoints on using information to modify people’s behaviour. One, Walden Two, by US author BF Skinner, took a utopian view of how positive rewards could be used as a means to modify behaviour so that people existed harmoniously and collaborated. The other, George Orwell’s 1984, predicted a sinister future in which the past and the external world existed only in the mind: behaviour could then be manipulated to further powerful interests, resulting in permanent conflict.
Summing up 20th Century Psychometrics, it has been used and abused by everyone from the Nazis to those practising discrimination through Eugenics. This was summed up by Samuel Taylor Coleridge, who, writing in 1831, lamented that “if only we could learn from history, what lessons it might teach us”, going on to postulate that “Passion and Party blind our Eyes, and the light which experiences gives us….is a lantern on the stern which shines only as a light on the waves behind us”. There was much innovation but there were also many intended and unintended consequences.
There have been four main psychometric revolutions, all of which have resulted in unintended consequences. Examinations used to establish meritocracy have been used in discrimination, mental tests used in diagnosis have resulted in Eugenics, Digital Footprints used in communication resulted in Big Brother, and Adaptive Testing, designed for prediction, has given us Artificial Intelligence.
From the first paper and pencil tests in 1960 to online psychmetrics in 2000 and a digital footprint in 2015 technology has provided us with a form of one dimensional progress. However, what had been lacking in the debate has been some honesty in the accuracy and application of the acquired information.
For the development of psychometrics, the most important advance since the Big 5 was in 2007 when David Stillwell, today at the University of Cambridge Psychometrics Centre, democratised the process through the development of his Facebook MyPersonality App. With its development people for the first time had immediate access to their own personality profiles and it went viral with over six million users, over half of whom agreed to share their Facebook data for research purposes. The system now uses 55 psychometric tests accessible via opt-in, and the data has been shared with more than eighty universities worldwide, contributing to understanding while at the same time building trust and confidence with the public.
The system was extended in 2012 with the development of Apply Magic Sauce, an API that relates personality and demographic profiles to Facebook Likes. It can predict up to 34 traits from open text, tweets and LikeIDs, providing feedback to users on what can be learned about them as well as being an important source of research, development, understanding and debate on wider issues in society by public, government and business. It is regularly accessed by on average 10,000 users per day, with a peak of 145,000 users a day. The digital footprint can make predictions about preferences, lifestyle, intelligence, psychological predispositions and aspirations. Even simple use of information collated via our “Likes” and text provide information and recent research has shown that, with over 160 Likes, computers know us better that our spouses.
So, what can we predict from our digital footprint? The list is intimidating, and includes relationship status, sexual orientation and ethnic origin.
The result is that we are being subjected to ever increasing amounts of tailored advertising. Tailored advertising can increase by 1000% the effectiveness of marketing efforts.
What these tests have shown is that tailored advertising makes us more impulsive and more likely to walk into situations where we should have paused and thought first. And who controls the data gathered? Although there is increasingly tightening data protection legislation, there are no laws around the gathering, structure and use of data for private purposes provided simple declarations are made. Our digital footprint reveals our hopes, needs and expectations: should this be for public consumption?
Your online Digital Footprint
• Demonstrates your habits, lifestyle & desires
• Shows where you and your friends are
• Assesses your needs, hopes and expectations
• Predicts what you will do, or not do,
• and can control your behaviour,
• with or without your knowledge,
• or consent.
There is plenty of scope for legislation in this area. However, as in many areas of life, technology is ahead of the regulation and probably always will be so.
The online age is upon us but no framework is in place to enable society to use information in a productive and collaborative way. Our biggest challenge is not the technology but agreeing on a framework so that technology and information can be used to maximise benefits which it brings.