Measuring Sexual Identity: Fundamentally Flawed, Practically Worthless, Irresponsible or Dangerous?

What’s wrong with ‘Measuring Sexual Identity: An Evaluation Report’ (2010) from the Office for National Statistics?

Well for starters, the authors admit that its methodology is fundamentally flawed by using a single measure and if they had they used a more appropriate measure then the figures for gay, lesbian and bisexual would most likely have been higher! The report misleads over refusal rates and does not adequate addresses the age, emplyment status, education, and ethnicity biases in its figures. In spite of evidence to the contrary the authors slap themselves on the back for a success’ whereas they should be slapping each other in the face for this unmitigated disaster.

Having taught statistics in several UK universities at all levels and  with a PhD in gender stereotypes and attitudes to sexuality (from an accredited university), I recognise, in this report,  many of  pitfalls I’ve taught undergraduate psychology students to avoid.  If it was an undergraduate report, I’d struggle to give it a ‘pass’. So let’s look at some of the main problems:

Attraction, Behaviour, Orientation, Identity
The report distinguishes between sexual attraction, sexual identity,  sexual orientation and sexual behaviour. So the expectation is that the report will take a multidimensional approach. However, it does not.  It states that whereas legislation focuses on sexual orientation, the report choses to look at sexual identity. It  also clearly states that behaviour may not form a basis for identity but goes on to argue:

‘Research during the development of the question also deemed sexual identity the most relevant dimension of sexual orientation to investigate given its relation to experiences of disadvantage and discrimination’ (p4).

Unfortunately, the research on which this conclusion is based is by the Office for National Statistics. It’s a classic example of groupthink. Much cutting edge research would dispute this assumption.  I certainly dispute it. It became clear very early on in my research that a single question to measure sexual identity or sexual orientation, at best would be misleading. So I didn’t.  It could be considered irresponsible to publish  research based on an imappropriate tool. In fact, the curremt report probably would not be published in any peer reviewed journal.

This report warns:

“[N}o single question would capture the full complexity of sexual orientation. A suite of questions would be necessary to collect data on the different dimensions of sexual orientation, and to examine consistency between them at the individual level (p4).”

The report goes ahead and uses a single question anyway! Whether or not a person labels their sexual orientation should not be the issue where ‘experiences of disadvantage and discrimination’ are concerned. Clearly, the reluctance to feel able to or be comfortable in declaring one’s sexuality is also a form of disadvantage and an aspect of discrimination. The current approach distorts the issue through over simplification.

So what’s the difference between attraction, behaviour, orientation and identity. Well, identity is how we describe ourselves or how others label us. Well people who are attracted to the same gender might not act upon it. People who act upon sexual attraction may do so in very specific circumstances. Orientation may be indicated by attraction, behaviour or a label with which a person identifies. I would argue that behaviour and attraction are more important than the label a person uses. For instance, in sexual health there is a recognised category of ‘men who have sex with men’ (MSM), who do not identify as gay. They see themselves as straight men who occasionally have sex with other men. The rest of the time they lead ‘straight’ lives. In sexual health services and health promotions, ‘men who have sex with men’ are at risk from sexual transmitted infections, and are targeted as a specific group.  However, assumptions made by this report concluded:

“Testing showed that respondents were not in favour of asking about sexual behaviour in a social survey context, nor would it be appropriate in general purpose government surveys (p4)”.

Again, this conclusion was based on reports from the Office for National Statistics.  In the present study, worryingly, the authors state:

‘As in the UK, deriving an individuals sexual orientation from a suite of questions results in higher LGB estimates in the US compared with using a single sexual identity question (p15)’

It is accepted in attitude measurement that single item responses are unsuitable.  Using a multiple response measure, properly administered would produce a more accurate figure. Furthermore, research suggests that this figure would have been higher.The figures cited in the report for more methodological sound research range from 5% to 9% ( Joloza, Evans and O’Brien, 2010, p15). The current report says 1.5%.

In a survey with such political impact, the decision to use a single item is ill-advised and  arguably reckless. Research convenience should not compromise validity. In this instance, it does.

Sensitivity of Measurement
Having abandoned the methodologically sound approach of using a ‘suite’ of questions, one might hope that at least the report would use a sensitive measure beyond crude, simplistic nominal categories. Actually no. In the 1940s, sex researcher Alfred Kinsey developed a more sensitive  ‘sliding scale’ of sexuality. Instead, the present researchers ignore this and opt for the bluntest of instruments: Straight, gay or bi. This report didn’t even bother to include transgender in its analysis.

Firstly, consider the approach of ‘measuring’ ethnicity based on a ‘Black, White or Mixed’ categorisation. How accurately would this categorisation produce a representation of ethnicity in the UK? Any reputable survey offers a whole range of options for ethnicity with quite subtle distinctions. Even then, people may declare ‘other’. I would argue that sexuality is more complex than ethnicity.  So why is measurement tool in the present report, measurably more simplistic? The answers is: ‘because the study has not been properly designed to fit the subject matter’. It has little or no ecological validity, that is, it means very little in the real world, except perhaps to fuel prejudice.

The Kinsey scale requires a respondent to use a zero to six scale. Where zero equals ‘exclusively heterosexual’ and six equals ‘exclusively gay’.  This gives varying degrees of bisexuality, that is, from one to five. Now clearly, these sensitive data can be collapsed into cruder categories if needs be. The problem with collecting crude data from the outset, is that we can do little else with it. It offers nothing very meaningful just the willingless of people to use a limited set of labels.

Now imagine, we take three measures of attraction, behaviour and identity all of the sliding ‘zero to six’ scale. Wouldn’t this be a far more accurate reflection of a person’s sexuality orientation? It’s just a pity they didn’t do it in their report for the Office of National Statistics. Would it produce a higher percentage of lesbian, gay and bisexual people? Well, Joloza, Evans and O’Brien (2010, p15) would probably say ‘yes’. So why didn’t they do it?

So what exactly does this study measure? Well it doesn’t measure sexual identity in the UK. It measures the percentage of people sampled  who are willing to declare a sexual identity label from a limited choice  in interviews, with or without others present, in a particular time frame (for a study with poor ecological validity).

But the problems don’t end there. The report also reveals questionable interpretation from by its authors.

Confidentially and the Willingness to Respond
One statistic almost jumps out of the page to indicate that there’s something wrong with this study:

[P]eople (aged 16 and over) who identified as LGB had a younger age distribution than heterosexuals – 64.9 per cent were aged under 45 compared with 48.6 per cent of people who identify as heterosexual (p16).

In other words, younger people are more likely to report a ‘non-heterosexual’ identity than are older people. With no evidence to support the notion that older people are less likely to be gay, it has to be an artifact of this research. That is, older people either don’t identify with a ‘gay, lesbian or bisexual’ so readily, or are not so predisposed to tell a stranger with a clipboard. As such, the one-shot sexual identity is not fit for purpose. It’s possible that older people are more likely to remember the days before the 1967 Sexual Offences Act, and police entrapment strategies. It’s possible that they don’t use the word ‘gay’ and may prefer ‘homosexual’. It’s possible that they don’t like to divulge personal information to strangers unless absolutely necessary. There are definitely confounding variables at play and not just age.

According to this study gay, lesbian and bisexual have better jobs and are better educated. Again, the myth of the pink surfaces. Could it not be that young, well-educated, finnacially secure people are more likely to divulge their LGB sexual identity to a stranger? This means that less-empowered people more in need of support and services are not. Again the one-shot measure doesn’t appear to do its job.

The report makes a claim in the face of its own evidence that confidentiality basing this on the refusal rate:

There is no evidence of an adverse impact on response rates confirming the general acceptance of the question. Our analysis suggests response rates are broadly in line with earlier quantitative testing. Non response to the question was low with less than 4 per cent of eligible respondents refusing to answer, saying they did not know the answer or not providing a response  (p26).

Perhaps this  should read  ‘no adverse impact on response rates, except for older, less financially secure, not-so-well educated, non-professionals’.  For those in routine and manual occupations, the most frequent response to the sexual identity question was ‘other’ at 31.1%, more ‘popular’ than heterosexual at 29.4%. Almost a half (49.1%) of those who identified as gay and lesbian had managerial or professional occupations, compared to less than a third (30.6%) who identified as heterosexual/straight? Furthermore, 38.1% of Gay/Lesbian had a degree compared with only 21.9% of Heterosexual/Straight. Doesn’t all this seem odd? Yes! It suggests a significant bias in the sampling, the method and the results. In short, the flaws are evident but largely overlooked by the authors. Failure to do this in an undergraduate report would be severely penalised. But far more is at stake here. This report may inform social policy!

Looking at ethnicity, there’s a bias here too. For Heterosexual/Straight people 90.7% are White, whereas for Gay/Lesbian/Bisexual its 93.5%. However, for the ‘Other’ category for sexual identity, 14.1% are ‘other ethnic group’. For the ‘Don’t knows’, the figure for ‘other ethnic group’ is 18.2%. People from ‘Other ethnic groups’ were almost twice as likely to say ‘Don’t Know’ as say ‘Heterosexual’ (9.3%). They were almost three times more likely to say ‘Don’t Know’ as ‘Gay/Lesbian/Bisexual’ (6.5%). This suggests either a reluctance to declare sexuality or that they did not understand what the terms meant. Either way, it’s a shortcoming of this research.

It’s interesting to note that option one on the interviewers card (market research style) was ‘Heterosexual/Straight’ and option two was  the less formal ‘Gay or Lesbian’, with option three as ‘Bisexual’. It’s interesting that whereas ‘both terms in option one and three can be applied to either men or women. For option two, you can’t have a lesbian man!The options do not use comparable terminology. If different terminology had been used, would the results have been different? If the options had been re-ordered, would the results have been different?Why is heterosexual the first option? Did this slightly increase the heterosexual figure. Research into research and experimter bias suggests it might.  Had the survey not been carried out in a market research format would the results have been different?Did interviewer the tone of voice affect the way in which the questions were answered. I’s done endless market research interviewers on the street and most of the time I can work out what the researcher ‘wants’ me to say. Are you heterosexual <smiles with rising intonation? or gay <frowns, with falling intonation> or bisexual <spits>? It’s a slight exageration but it does happen.

Now let’s turn to the ‘less than four per cent refusal rate’ that caused the authors to discard the other evidence.

The authors state:

‘Prior to developing and testing work on the sexual identity question, the expectation was that the higher the number of adults in the household, the higher the proportion of item non response. This is be because some household members might be reluctant to disclose their sexual identity in the presence of others. However, the results from the IHS do not indicate this (p12).’

However they don’t make the connection between non-response rates and the willingness to declare a true label:

‘Another observation here is that the proportion of people reporting to be LGB in a household decreases as the number of adults in the household increases. There is currently no explanation why this is the case but this is something that could be considered for further investigation in future (p12)’.

One explanation might be that, the more people in the house the less likely people are to declare themselves to be ‘Gay/Lesbian/Bisexual’. They didn’t necessarily refuse they may have felt the need to protect their own privacy and lied or said ‘don’t know’.

So for people identifying as the as the number of people living in the house increases, the figure for ‘Heterosexual/Straight’ increases slightly. For Gay/Lesbian it falls from 1.3% in a single person household to only 0.3% in a four person plus household. The figures for Bisexual remain roughly the same. For ‘Don’t Know/Refusal’ the figures increase slightly as the number of people in the house increases.  This suggests that there is an element of self-censorship in responses.

Think about it logically. If you want to keep your sexual identity secret from other members of the house, do you ‘refuse’ and cause the other house members to ask why, or do you just lie?  Or if you live a ‘heterosexual’ life for 95% of the time and have recreational sex, exactly how do you respond to the stranger on the door step with the crude market research question?

What’s clear is that the current report has not adequately addressed the numerous problems it has generated with an inappropriate methodology for a complex subject. White, Black and Mixed would not be good enough for ethicoity, so why is it good enough here, for a subject arguable more sensitive and complex?

Improvements
It’s important to remember that the Kinsey Team in he 1940s put the gay and bisexual figure as high as 37%. Of course the sampling has been criticised over the years. It probably did lead to an overestimation. Nevertheless the measure on which the Kinsey team based their research was exemplary. A one-shot question does not work for something as complex as human sexuality. The Kinsey measurement was complex and fit for purpose. It is not good enough to side step the issue of instrument accuracy with protests of convenience and acceptability to researchers. Rather than go for the easy, convenient option, get better researchers and design a better study where appropriate measures can be used.  Otherwise all you get is conveniently produced meaningless results. Garbage in, garbage out.

Conclusion
So is this report, fundamentally flawed, practically worthless, irresponsible or dangerous? In my professional opinion, considering the plitical climate, I’d have to say that it’s all of those things. The ONS needs to stop engaging in groupthink and stop treating the complex notion of sexual orientation as some crass market research exercise.  Patting themselves on the back, the authors conclude:

‘The introduction of the sexual identity question. . .  in January 2009 followed rigorous testing and feasibility testing by ONS. The findings of this report suggest its implementation on the IHS in the first year has been a success (p26)’.

A success why what standards? Certainly not of academic rigour. We need high quality research data on which to make sense of our world and inform our social policy decisions. Sadly, this report fails to deliver and cannot be treated as anything other than a pilot study from which serious lessons need to be learned.  The simplistic method does not work evidenced by the reports own figures. It fails to meet the standards of an undergraduate report, on which one can only conclude ‘must do better next time’. Sadly, the decision not to consider the ethical ramifications of publishing a flawed report is inexcusable and sheds light on the ability of the ONS to produce high quality data. It’s argubale negligence. Researchers have a responsiblity to consider how their research will be used. The ONS has  failed to recognise its responsibility.

Maybe it does  not commit the sin of commission of homophobia but it does commit the sin of omission in that it justifies the heterosexist ideology of rendering invisible sexual diversity.

So if we add in the refusals and the don’t knows, if we adjust the figures for age, ethnicity, education, profession and number of people in the household, what exactly would the figue be for ‘non-heterosexuals’? Well, your guess is as good as mine.  It’s disappointing that this overblown, expensive pilot study has thrown up more questions than it answers, and we are back to simply ‘guessing’.

Links:

Measuring Sexual Identity: An Evaluation Report

Advertisements

Gender, Cave People & an Apology for Psychology

If I have to hear another ‘it’s a throwback to cave people’ explanation to explain gender social roles, I’ll scream. In fact I do! Much to the dismay of people sitting in the same room.  It’s all the worse when it comes from people who should know better. I mean, we expect it from stand-up comedians but here’s an example of a  psychologist who should really know better even though s/he is speaking outside of her/his field of expertise (and appears to make a habit of it). The subject is computer games and gender.

Computer games are ideally suited to men we are informed because. . . wait for it. . .

‘[B]ack when they were cavemen, men had to focus on the animal they were trying to kill. If they were distracted by anything from a woman to their own emotions, they’d miss the target. The real appeal for men is escapism though, because they’re not as evolved to deal with emotions which is why they like games more than us’.

(It’s not clear whether the venerable ‘expert’ means that men like computer games more than they like women, or more than women like computer games. However it is clear that the use of the word ‘us’ clearly shows that the person is not speaking as a psychologist but is giving a personal opinion as a ‘not-man’)

It gets ‘better’. . .

‘Competition is important to men because it let’s them work out who’s “the best”, an instinct going back to the days when they had to prove to the cavewoman that they’d be superior providers for them’.

So where is the evidence for these sagely insights? Now I’m not aware that this particular expert has done any research whatsoever on why people enjoy computer games. The person in question doesn’t look quite old enough to be from Palaeolithic times, so it can’t be from personal experience. As for the evidence of gender roles in cave people, this largely arose from the views of a once male-dominated archaeology who often made the cardinal error of using modern-day Western living as a lens by which to view historical and cultural data. It wasn’t until the 1960s when female archaeologists had the opportunity to question the orthodox, androcentric view that an alternative view began to emerge.   The meat content of  cave people is most likely exaggerated. Some sources suggest that it was about 80% gathering (vegetarian), so those archaeological spear-like, in some instances, could just as well be scraping and digging implements. Meat was more likely a ‘special occasion’ thing which is why it appeared as paintings on cave walls. Meat consumption increased with agriculture. Plenty of sources now agree that there weren’t the super-defined gender roles of the 1950s. It’s certainly ridiculous to assume that ‘cave people’ society was based on lots of little semi-detached caves containing nuclear families with mummy sitting at home making apple sauce on the off chance that daddy comes home with a pig. It makes no sense! The societies were probably more cooperative and egalitarian with everyone ‘mucking in’.

The case for gender differences is massively overstated in popular sources (and a few academic ones). When gender differences are scrutinised in meta-analyses, taking into account confounding factors what invariably results are no differences or relatively small (statistically significant) differences. Although these are often reported as ‘significant’ in popular sources there is often a basic misunderstanding of what the word ‘significant’ means in the context of research. It means that it passes a statistical test. However, this does not necessarily translate into a real-world significance.  Furthermore, the differences that do occur can be diminished or eradicated by training. This suggests strongly that even these small gender differences are determined by social factors. Overall, the body of research on gender demonstrates that there is a greater difference within each gender than between them. It also shows that the similarities between the genders are far greater than their differences.

Whenever, ‘experts’ resort to the ‘cave person’ analogy, this is a substitute for considering the evidence. It’s a smokescreen.  It taps into a commonly held myth and therefore, on the surface, appears to ring true. Now we expect the host of ‘fakexperts‘ to resort to  ‘cavepeople’ analogies because many of them may well not be expert at interpreting research data or know where to find evidence-based resources. However, for the seemingly respectable psychologist, there really is no excuse for this kind of slap-dash, ‘say-the-first-thing-that-pops-into-your-head’ kind of laziness.  So the next time you hear cave people and gender used, uncritically, in the same sentence, question the credentials and the motives of the speaker (or writer). The same goes for the ‘Mars-Venus’ analogy. It’s just another smokescreen.

More often than not, the appearance of psychologists in the media are missed opportunities to communicate evidence-based psychology. Invariably,  what we have is not even an apology for psychology but  bull-shit based psychobabble and ‘gossipology’. So often the definition of a ‘celebrity’ psychologist is ‘someone who should know better’. We certainly deserve better!

Recommended Books on Gender:

Links:

The Dangers of Social Networking: Are We Frying Our Brains? Is it 21st Century Onanism?

Social networking: Are we in danger of infantilization?

Social networking: Are we in danger of infantilization?

Internet social networking has now become the target for age old debate on harmful media influences. Leading neuroscientist Professor Susan Greenfield has speculated that as we spend too much time on sites such as Facebook and Twitter, we run the risk of infantalizing the mid-21st mind. However haven’t we heard similar arguments before most notably with TV and video games, right back to the chivalry books of Don Quixote?  Indeed eerily similar warnings of the dangers of on-line social networking were also made for masturbation (onanism) in the 19th century. According to Lord Baden-Powell, such ‘beastliness’ was said to destroy both health and spirits so that the practitioner becomes ‘feeble in body and mind and often ends in a lunatic asylum’. Is social networking becoming the onanism of the 21st century?

Too much on-line onanism?

Too much on-line onanism?

At best, Prof. Susan Greenfield’s comments are conjecture. She admits that there is no evidence to support her assertions (see Newsnight link below). So why bother then? Well, she’d just like the government to provide some funding to satisfy her hunch.  But by contrast, oh the bells, the alarm bells from Dr Aric Sigman who claims that there is ample evidence although, according to Dr Ben Goldacre (Bad Science) Sigman only bothers to look at half of it (see link below). In short, the half that would provide us with a balanced scientific perspective is missing.  It is worth pointing out that Dr Sigman’s piece does not cite any research that he carried out (although it’s difficult to tell as he did not cite the full list of authors in his reference list, so he may be in there somewhere). The paper looks to be a  literature review with his own conclusions and he concedes (in the Newsnight interview) that it is meant to be a one-sided piece intended to provoke discussion, something he appears to specialize in (see: In Bed With Mary Whitehouse). The paper is based on the assumption that Internet social networking increases isolation and loneliness which in turn leads to health problems.

As with many of these media-scare stories, the more dramatic conclusions require a leap of faith that correlational studies indicate causation. They do not.  So for instance, does Internet social networking increase isolation and loneliness or do lonely and isolated people seek out a means to make themselves less so? This could mean that on-line social networking is a positive thing. And perhaps we should for a moment consider e-learning’, providing wider educational opportunities? No, the ‘e’ doesn’t stand for evil!

Dr Sigman speaks of displacement arguing that children are sitting in front of a computer when they could be out socializing with friends. However, couldn’t the same argument be made for watching television (er, yes it has), reading and even doing homework. I know I would have rather been out playing ‘kick the can’ as a child, when I was stuck in doors reading about the printing press and how it gave ‘the great unwashed’ ideas above their station (pretty much in the same way as the Internet does today). Worries that children can be too ‘bookish’ are now largely overlooked in favour of newer ‘dangers’. And presumably, on-line social networking has displaced ‘staring aimlessly out of the window on a rainy day’ too.

Sigman has also made the claim that we can only be friends with someone if we can shake hands with them. That’s assuming that everyone has hands and  that they can shake them! He also makes questions whether we should be chatting on-line to people across the globe instead of playing with ‘real friends’. Has he forgotten ‘penpals’ at school, the practice of writing to complete strangers in different countries? It was heartily encouraged! As a social psychologist, I would have thought connecting with people from other lands was a good thing (in contrast with good old-fashioned, small-minded, back-porch xenophobia). However, some evidence suggests young people are using online social networking primarily to enhance existing friendships and interaction with ‘e-friends’ is secondary. Unfortunately Facebook only uses th category ‘friends’ for all online contacts. It doesn’t mean that each and every is viewed the same; users may make their own distinctions. Thinking about my own experience. I have over 600 contacts in my email contact list although only 20 that I use frequently. It is not a coincidence that these are the people with whom I have the greatest face-to-face contact.  In short, much of the headline-seeking, scare mongering stems from a failure to understand how the average user uses online social networking, instead indulging in worst case scenarios. I could go on but instead invite you to read Sigman’s article and with the aid of Goldacre’s response ask you to think of alternative explanations that make up the whole of the picture. And while you’re at it consider how Greenfield throws together a heap of ideas and assumptions that do not necessarily have to be connected in the same way as she does.

So are we all twittering ourselves into oblivion?

Much has been made of the ‘rewiring our brains’ hypothesis and Prof Greenfield has speculated that in middle of the 21st Century – that is in about 40 years time –  our brains will be different. This seems a little premature to say the least. Sigman would warn us about exposing young brains to novel experienes but this also includes just about everything at school. Children have an enormous capacity for integrating new experiences. Chances are the world will be a different place and yes humans will adapt. Isn’t that learning? In the long-term isn’t it evolution?

We simply do not have the evidence to assess the extent of brain changes  and it certainly won’t just be all down to how much time we spend chatting to our real or ‘virtual friends’. Life is so much more quixotic! It’s also important to recognise that new technologies impact on different people in different ways. The diffusion of any innovation is such that we have innovators, early adopters, the late majority with laggards, er . . lagging behind.   So will we have a sliding scale of brain-mush? Will the ‘slow on the uptake’ be spared?  Again, more conjecture. It’s simply not possible, nor scientific, to conclude that any changes will be for the worst  and to pretend that ‘brain infantilization’ or ‘hairy-palmed isolation’ are  foregone conclusions (or even likely) is at best bad science. It’s just another example of not letting the evidence get in the way of  a good story.

See Links: