What’s wrong with ‘Measuring Sexual Identity: An Evaluation Report’ (2010) from the Office for National Statistics?
Well for starters, the authors admit that its methodology is fundamentally flawed by using a single measure and if they had they used a more appropriate measure then the figures for gay, lesbian and bisexual would most likely have been higher! The report misleads over refusal rates and does not adequate addresses the age, emplyment status, education, and ethnicity biases in its figures. In spite of evidence to the contrary the authors slap themselves on the back for a success’ whereas they should be slapping each other in the face for this unmitigated disaster.
Having taught statistics in several UK universities at all levels and with a PhD in gender stereotypes and attitudes to sexuality (from an accredited university), I recognise, in this report, many of pitfalls I’ve taught undergraduate psychology students to avoid. If it was an undergraduate report, I’d struggle to give it a ‘pass’. So let’s look at some of the main problems:
Attraction, Behaviour, Orientation, Identity
The report distinguishes between sexual attraction, sexual identity, sexual orientation and sexual behaviour. So the expectation is that the report will take a multidimensional approach. However, it does not. It states that whereas legislation focuses on sexual orientation, the report choses to look at sexual identity. It also clearly states that behaviour may not form a basis for identity but goes on to argue:
‘Research during the development of the question also deemed sexual identity the most relevant dimension of sexual orientation to investigate given its relation to experiences of disadvantage and discrimination’ (p4).
Unfortunately, the research on which this conclusion is based is by the Office for National Statistics. It’s a classic example of groupthink. Much cutting edge research would dispute this assumption. I certainly dispute it. It became clear very early on in my research that a single question to measure sexual identity or sexual orientation, at best would be misleading. So I didn’t. It could be considered irresponsible to publish research based on an imappropriate tool. In fact, the curremt report probably would not be published in any peer reviewed journal.
This report warns:
“[N}o single question would capture the full complexity of sexual orientation. A suite of questions would be necessary to collect data on the different dimensions of sexual orientation, and to examine consistency between them at the individual level (p4).”
The report goes ahead and uses a single question anyway! Whether or not a person labels their sexual orientation should not be the issue where ‘experiences of disadvantage and discrimination’ are concerned. Clearly, the reluctance to feel able to or be comfortable in declaring one’s sexuality is also a form of disadvantage and an aspect of discrimination. The current approach distorts the issue through over simplification.
So what’s the difference between attraction, behaviour, orientation and identity. Well, identity is how we describe ourselves or how others label us. Well people who are attracted to the same gender might not act upon it. People who act upon sexual attraction may do so in very specific circumstances. Orientation may be indicated by attraction, behaviour or a label with which a person identifies. I would argue that behaviour and attraction are more important than the label a person uses. For instance, in sexual health there is a recognised category of ‘men who have sex with men’ (MSM), who do not identify as gay. They see themselves as straight men who occasionally have sex with other men. The rest of the time they lead ‘straight’ lives. In sexual health services and health promotions, ‘men who have sex with men’ are at risk from sexual transmitted infections, and are targeted as a specific group. However, assumptions made by this report concluded:
“Testing showed that respondents were not in favour of asking about sexual behaviour in a social survey context, nor would it be appropriate in general purpose government surveys (p4)”.
Again, this conclusion was based on reports from the Office for National Statistics. In the present study, worryingly, the authors state:
‘As in the UK, deriving an individuals sexual orientation from a suite of questions results in higher LGB estimates in the US compared with using a single sexual identity question (p15)’
It is accepted in attitude measurement that single item responses are unsuitable. Using a multiple response measure, properly administered would produce a more accurate figure. Furthermore, research suggests that this figure would have been higher.The figures cited in the report for more methodological sound research range from 5% to 9% ( Joloza, Evans and O’Brien, 2010, p15). The current report says 1.5%.
In a survey with such political impact, the decision to use a single item is ill-advised and arguably reckless. Research convenience should not compromise validity. In this instance, it does.
Sensitivity of Measurement
Having abandoned the methodologically sound approach of using a ‘suite’ of questions, one might hope that at least the report would use a sensitive measure beyond crude, simplistic nominal categories. Actually no. In the 1940s, sex researcher Alfred Kinsey developed a more sensitive ‘sliding scale’ of sexuality. Instead, the present researchers ignore this and opt for the bluntest of instruments: Straight, gay or bi. This report didn’t even bother to include transgender in its analysis.
Firstly, consider the approach of ‘measuring’ ethnicity based on a ‘Black, White or Mixed’ categorisation. How accurately would this categorisation produce a representation of ethnicity in the UK? Any reputable survey offers a whole range of options for ethnicity with quite subtle distinctions. Even then, people may declare ‘other’. I would argue that sexuality is more complex than ethnicity. So why is measurement tool in the present report, measurably more simplistic? The answers is: ‘because the study has not been properly designed to fit the subject matter’. It has little or no ecological validity, that is, it means very little in the real world, except perhaps to fuel prejudice.
The Kinsey scale requires a respondent to use a zero to six scale. Where zero equals ‘exclusively heterosexual’ and six equals ‘exclusively gay’. This gives varying degrees of bisexuality, that is, from one to five. Now clearly, these sensitive data can be collapsed into cruder categories if needs be. The problem with collecting crude data from the outset, is that we can do little else with it. It offers nothing very meaningful just the willingless of people to use a limited set of labels.
Now imagine, we take three measures of attraction, behaviour and identity all of the sliding ‘zero to six’ scale. Wouldn’t this be a far more accurate reflection of a person’s sexuality orientation? It’s just a pity they didn’t do it in their report for the Office of National Statistics. Would it produce a higher percentage of lesbian, gay and bisexual people? Well, Joloza, Evans and O’Brien (2010, p15) would probably say ‘yes’. So why didn’t they do it?
So what exactly does this study measure? Well it doesn’t measure sexual identity in the UK. It measures the percentage of people sampled who are willing to declare a sexual identity label from a limited choice in interviews, with or without others present, in a particular time frame (for a study with poor ecological validity).
But the problems don’t end there. The report also reveals questionable interpretation from by its authors.
Confidentially and the Willingness to Respond
One statistic almost jumps out of the page to indicate that there’s something wrong with this study:
[P]eople (aged 16 and over) who identified as LGB had a younger age distribution than heterosexuals – 64.9 per cent were aged under 45 compared with 48.6 per cent of people who identify as heterosexual (p16).
In other words, younger people are more likely to report a ‘non-heterosexual’ identity than are older people. With no evidence to support the notion that older people are less likely to be gay, it has to be an artifact of this research. That is, older people either don’t identify with a ‘gay, lesbian or bisexual’ so readily, or are not so predisposed to tell a stranger with a clipboard. As such, the one-shot sexual identity is not fit for purpose. It’s possible that older people are more likely to remember the days before the 1967 Sexual Offences Act, and police entrapment strategies. It’s possible that they don’t use the word ‘gay’ and may prefer ‘homosexual’. It’s possible that they don’t like to divulge personal information to strangers unless absolutely necessary. There are definitely confounding variables at play and not just age.
According to this study gay, lesbian and bisexual have better jobs and are better educated. Again, the myth of the pink surfaces. Could it not be that young, well-educated, finnacially secure people are more likely to divulge their LGB sexual identity to a stranger? This means that less-empowered people more in need of support and services are not. Again the one-shot measure doesn’t appear to do its job.
The report makes a claim in the face of its own evidence that confidentiality basing this on the refusal rate:
There is no evidence of an adverse impact on response rates confirming the general acceptance of the question. Our analysis suggests response rates are broadly in line with earlier quantitative testing. Non response to the question was low with less than 4 per cent of eligible respondents refusing to answer, saying they did not know the answer or not providing a response (p26).
Perhaps this should read ‘no adverse impact on response rates, except for older, less financially secure, not-so-well educated, non-professionals’. For those in routine and manual occupations, the most frequent response to the sexual identity question was ‘other’ at 31.1%, more ‘popular’ than heterosexual at 29.4%. Almost a half (49.1%) of those who identified as gay and lesbian had managerial or professional occupations, compared to less than a third (30.6%) who identified as heterosexual/straight? Furthermore, 38.1% of Gay/Lesbian had a degree compared with only 21.9% of Heterosexual/Straight. Doesn’t all this seem odd? Yes! It suggests a significant bias in the sampling, the method and the results. In short, the flaws are evident but largely overlooked by the authors. Failure to do this in an undergraduate report would be severely penalised. But far more is at stake here. This report may inform social policy!
Looking at ethnicity, there’s a bias here too. For Heterosexual/Straight people 90.7% are White, whereas for Gay/Lesbian/Bisexual its 93.5%. However, for the ‘Other’ category for sexual identity, 14.1% are ‘other ethnic group’. For the ‘Don’t knows’, the figure for ‘other ethnic group’ is 18.2%. People from ‘Other ethnic groups’ were almost twice as likely to say ‘Don’t Know’ as say ‘Heterosexual’ (9.3%). They were almost three times more likely to say ‘Don’t Know’ as ‘Gay/Lesbian/Bisexual’ (6.5%). This suggests either a reluctance to declare sexuality or that they did not understand what the terms meant. Either way, it’s a shortcoming of this research.
It’s interesting to note that option one on the interviewers card (market research style) was ‘Heterosexual/Straight’ and option two was the less formal ‘Gay or Lesbian’, with option three as ‘Bisexual’. It’s interesting that whereas ‘both terms in option one and three can be applied to either men or women. For option two, you can’t have a lesbian man!The options do not use comparable terminology. If different terminology had been used, would the results have been different? If the options had been re-ordered, would the results have been different?Why is heterosexual the first option? Did this slightly increase the heterosexual figure. Research into research and experimter bias suggests it might. Had the survey not been carried out in a market research format would the results have been different?Did interviewer the tone of voice affect the way in which the questions were answered. I’s done endless market research interviewers on the street and most of the time I can work out what the researcher ‘wants’ me to say. Are you heterosexual <smiles with rising intonation? or gay <frowns, with falling intonation> or bisexual <spits>? It’s a slight exageration but it does happen.
Now let’s turn to the ‘less than four per cent refusal rate’ that caused the authors to discard the other evidence.
The authors state:
‘Prior to developing and testing work on the sexual identity question, the expectation was that the higher the number of adults in the household, the higher the proportion of item non response. This is be because some household members might be reluctant to disclose their sexual identity in the presence of others. However, the results from the IHS do not indicate this (p12).’
However they don’t make the connection between non-response rates and the willingness to declare a true label:
‘Another observation here is that the proportion of people reporting to be LGB in a household decreases as the number of adults in the household increases. There is currently no explanation why this is the case but this is something that could be considered for further investigation in future (p12)’.
One explanation might be that, the more people in the house the less likely people are to declare themselves to be ‘Gay/Lesbian/Bisexual’. They didn’t necessarily refuse they may have felt the need to protect their own privacy and lied or said ‘don’t know’.
So for people identifying as the as the number of people living in the house increases, the figure for ‘Heterosexual/Straight’ increases slightly. For Gay/Lesbian it falls from 1.3% in a single person household to only 0.3% in a four person plus household. The figures for Bisexual remain roughly the same. For ‘Don’t Know/Refusal’ the figures increase slightly as the number of people in the house increases. This suggests that there is an element of self-censorship in responses.
Think about it logically. If you want to keep your sexual identity secret from other members of the house, do you ‘refuse’ and cause the other house members to ask why, or do you just lie? Or if you live a ‘heterosexual’ life for 95% of the time and have recreational sex, exactly how do you respond to the stranger on the door step with the crude market research question?
What’s clear is that the current report has not adequately addressed the numerous problems it has generated with an inappropriate methodology for a complex subject. White, Black and Mixed would not be good enough for ethicoity, so why is it good enough here, for a subject arguable more sensitive and complex?
Improvements
It’s important to remember that the Kinsey Team in he 1940s put the gay and bisexual figure as high as 37%. Of course the sampling has been criticised over the years. It probably did lead to an overestimation. Nevertheless the measure on which the Kinsey team based their research was exemplary. A one-shot question does not work for something as complex as human sexuality. The Kinsey measurement was complex and fit for purpose. It is not good enough to side step the issue of instrument accuracy with protests of convenience and acceptability to researchers. Rather than go for the easy, convenient option, get better researchers and design a better study where appropriate measures can be used. Otherwise all you get is conveniently produced meaningless results. Garbage in, garbage out.
Conclusion
So is this report, fundamentally flawed, practically worthless, irresponsible or dangerous? In my professional opinion, considering the plitical climate, I’d have to say that it’s all of those things. The ONS needs to stop engaging in groupthink and stop treating the complex notion of sexual orientation as some crass market research exercise. Patting themselves on the back, the authors conclude:
‘The introduction of the sexual identity question. . . in January 2009 followed rigorous testing and feasibility testing by ONS. The findings of this report suggest its implementation on the IHS in the first year has been a success (p26)’.
A success why what standards? Certainly not of academic rigour. We need high quality research data on which to make sense of our world and inform our social policy decisions. Sadly, this report fails to deliver and cannot be treated as anything other than a pilot study from which serious lessons need to be learned. The simplistic method does not work evidenced by the reports own figures. It fails to meet the standards of an undergraduate report, on which one can only conclude ‘must do better next time’. Sadly, the decision not to consider the ethical ramifications of publishing a flawed report is inexcusable and sheds light on the ability of the ONS to produce high quality data. It’s argubale negligence. Researchers have a responsiblity to consider how their research will be used. The ONS has failed to recognise its responsibility.
Maybe it does not commit the sin of commission of homophobia but it does commit the sin of omission in that it justifies the heterosexist ideology of rendering invisible sexual diversity.
So if we add in the refusals and the don’t knows, if we adjust the figures for age, ethnicity, education, profession and number of people in the household, what exactly would the figue be for ‘non-heterosexuals’? Well, your guess is as good as mine. It’s disappointing that this overblown, expensive pilot study has thrown up more questions than it answers, and we are back to simply ‘guessing’.
Links:
Measuring Sexual Identity: An Evaluation Report
Applied Psychology and Solution Focused Life Coaching with Dr Gary Wood 