October London Event Report

Roy Childs, Team Focus

Self-report questionnaires – how can they be made more effective?

University of Westminster, 15th October 2019

Those of us in the business of psychometrics need to take a long hard look at what it is we are selling and trying to deliver – in this age of ‘Fake News’ we must ensure that what we promote is fair and accurate.  At the October event held at Westminster University, Roy Childs argued that one of the success areas for Business Psychology was Psychometrics but that this area needed more scrutiny and challenge – that some of the ‘models and methods’ were not as rigorous or appropriate as was often claimed and that, to gain greater client confidence, these claims should be more consistent and realistic.  His presentation picked out aspects from the following strands in psychometrics:

*           Personality

*           Capability

*           Motivation

*           Values

The focus of the talk was on the nature of self-report questionnaires and how ‘hard stats’ were being used on ‘soft numbers’ to draw inappropriate conclusions – which led to his conclusion that many claims for Rigour were in fact evidence for Rigor Mortis!  An example of what he meant by ‘soft numbers’ was illustrated from Will Schutz’s work with NASA where, after a year of workshops designed to increase Openness the pre and post organisational survey results showed a marked drop in Openness.  A ‘hard number’ interpretation would be that the workshops had been counter-productive.  However, it was clear to participants that they were now more open.  What had changed was their understanding of what Openness really looked like.  It was the workshops that had made them realise their initial concept of Openness was quite superficial.  In other words the bar had moved and the decrease in scores reflected the reality that they were more open but that there was still work to be done.  Roy argued that all self-report results are ‘soft numbers’ and yet, by using a ‘hard stats lens,’ the interpretation can be at best misleading and at worst totally inaccurate.  Another example was of a person who completed an Emotional Intelligence questionnaire. They obtained the highest overall EI score in the group but, by popular agreement, they were clearly the least self-aware in that group!

Roy then asked people to rate themselves on ‘How good is your short-term memory.’  He then gave them a memory recall task which produced a score and he asked which of the two score, was more likely to be accurate.  The parallel was drawn with personality questionnaires that asked ‘are you persuasive’ or ‘are you decisive?’.  He believes that there would be a small correlation between the objective memory score and the self-reported score as there would be between self-ratings of decisiveness – but that this does not mean we should be confident about the accuracy of the self-assessment.

With that background, the talk then covered the following:

1. a lack of consistency in the domain used to define personality.
2. Limitations in the questionnaire evaluation process
3. taking responsibility for how questionnaires are used
4. Putting forward some ideas for improvement

 

1. A lack of consistency in personality questionnaires

Roy highlighted 3 areas where more clarity would show greater rigour.  The first was better definition of the personality domain – are we analysing what people think they do (behaviour), what they think they want to do (preference), what they enjoy (motivation) or what they think (attitudes and beliefs).  Using examples from various well known questionnaires he argued that there is no clear and explicit rationale to guide item writing, no consistency between or within questionnaires and no consensus amongst theorists/authors.  The second area was the inconsistency regarding the measurement of traits versus competencies.  Are we trying to understand personality through its fundamental ingredients/atoms/elements or through its real world manifestations/molecules/compounds.  He argued that the latter would be better defined as ‘competency surveys – which explains why there is a proliferation of easily understood labels.  The issue that pre-occupied Cattell and Eysenck (i.e. how many factors we should be measuring) has largely disappeared – especially since the dominance of the Big Five – which are what Ray Cattell argued as ‘the only factors that survive clumsy factor analysis’. The third area where there was ‘loose thinking’ was inviting people to describe themselves in a particular context (at work, in this team or in some generalised concept of a team).  The Occupational Personality Questionnaire clearly asks for ‘at work’ which Roy argued could more easily be understood as a role whereas Belbin’s SPI which claims to be a role questionnaire but may more appropriately be viewed as measuring a ‘team personality’ because it transcends specific situations.

 

2. Limitations in the questionnaire evaluation process

The challenge here was how the psychometric approach treated data as ‘hard numbers’ – which is understandable since it was developed to deal with ability tests.  However, when we come to self-report he suggested that the answers should be regarded as ‘a puzzle to be understood rather than an answer to be interpreted or a prediction to be made’.  Thus results become a basis for exploration; they should stimulate curiosity and he suggested an aid to doing that which was to use the two words “I wonder” rather more and to use ‘this means/suggests’ a lot less.  He then suggested that this approach positioned the BPS reviews of psychometric tests in a more realistic light.  Of course the reviews provide a useful lens for evaluating a questionnaire’s value and appropriateness but that they were only a start for a proper evaluation – and the questionnaires with the most stars (a grading system used in the reviews) were not necessarily the best!

 

3. Taking responsibility for how questionnaires are used

The over emphasis on the hard measurement aspect of self-report questionnaires leads people to use them for making predictions.  This has already been criticised but Roy also made the point that Business Psychologists should be much more cautious about using small correlations to make predictions about individuals.  Such correlations may make sense in mass recruitment situations but when applied to individuals, the validity statistics not only explain a very small amount of the variance but also the situation/circumstances of many appointments are unique (especially in today’s changing world) and so this further reduces the justification for using profiles as predictive.  One of the roles of the psychological profession is to promote good practice of psychological tools and we should be much more vocal (or at least more effective) about the inappropriateness of using correlations of 0.3 to justify the advantages on one individual profile over another.  We know that people recognise that human personality is complex but we also know that people are easily seduced by artificial simplicity.  This is a call for our profession to be better at steering people along the path of good and realistic practice rather than following the demand.  This means better communication regarding the limitations of self-report questionnaires and how the data is used.

Roy also pointed out that the controversy of nature versus nurture still hangs over psychometrics – affecting the way that we view personality and its ‘flexibility’.  The psychometric model is built on the concept of consistency (a term that has been used as a synonym for reliability and which conflates stability with accuracy).  Taking Jung’s view that personality type can change over time (i.e. a journey of self-development whereby people can, eventually, transcend their type) it raises the question of whether change in self-report should be thrown into the unreliability bucket.  Jung certainly saw people’s self-reported type as a description of where they had got to – and this was the springboard to future adaptation and individuation.

 

4. Putting forward some ideas for improvement

Whilst this talk was critical of some psychometric practice, there were also suggestions for how things could be improved.  One such suggestion was to acknowledge what Professor Jerome Kagan identified in his book ‘Psychology’s Ghosts’.  His first chapter identifies the key issue of ‘Missing Contexts’.  This links to the difference between being and doing – are we just what we do or is what we do significantly influenced by where we are?  Roy described one psychometric solution, the Type Mapping system, which clarifies the distinction between behaviour in context and thus allows for behavioural flexibility around a sense of personality – or perhaps more accurately, around a sense of identity.  The Type Mapping system allows for different mind sets when completing a questionnaire and uses a model of ‘the Four Selfs’ which he summarised as:

*           Contextual Self – the many roles we play

*           Identity Self – the sense of me behind the role

*           Ideal Self – a preference or aspiration

*           Undiscovered Self – the part of me yet to unfold

Copied from the FIRO, the Ideal self can be accessed by asking about ‘the way I want’.  This has been a very powerful way to unpack a lot of historical baggage and it helps people to articulate a deeper sense of self and to articulate different elements of themselves.  Seeing this as a journey, Roy was struck by an interview with Jimmy Connors, one of the most successful tennis players of all time, who said that he felt a deep sense of change.  This was epitomised by his realisation that he got more pleasure from seeing his coachee (Andy Roddick) win than he had from his own victories – which he had thought could never be exceeded.  Whether this was a change in the characteristic ‘competitiveness’ or simply a new form of expressing it can be debated but, nevertheless, it is part of a big story of change and transformation – a journey towards maturity in which a view of an unchanging personality may not be helpful.

Another area in distinct need of examination and revision was the ‘measurement’ of values.  Can we trust any self-report in this area – how many people claim integrity as high on their list but this is not recognised by others (e.g. politicians?).  There are successful values questionnaires that present a normative values profile which can be hugely misleading.  Roy proceeded to demonstrate how a normative approach does not identify what is important to people.  However, he also explained how, by using some new psychometric ideas developed at Team Focus, a self-report questionnaire can probe below the surface and challenge a person’s self-report.  This has resulted in the publishing of the Value-based Indicator of Motivation (VbIM) which uses both ipsative and normative approaches to dig down into people’s less conscious views of themselves. He also mentioned how VbIM (as well as the EIQ, TDI and RHA questionnaires) could be completed as paired versions thus providing another view of the person’s characteristics or values.

 

In summary:

1. We should consider personality as a mosaic – we are describing an unfinished painting with sections which have not yet evolved.  Blind application of the psychometric model can be misleading and we need to use it to inform our psychological understanding rather than to drive it.
2. Acknowledge that self-report questionnaires are a way to encourage self-exploration and not a way to get a terminal classification.  Introducing more lenses and viewpoints to stretch, stress and challenge should be welcomed once it is acknowledged that we are not trying to draw lines between two static points – the static ‘me now’ and some static point in the future.
3. Encourage feedback – make use of the “paired” versions available with some questionnaires which then goes beyond self-report.
4. Encourage the use of different mind sets such as the “Is” versus the “Want” which encourages reflection and reveals an internal dialogue regarding ‘who I really am’
5. Recognise the difference between Being and Doing which gets into the core concept of a person’s identity.

 

RT

15 Oct 2019