Learning Science: The Problem With Data, And How You Can Measure Anything

Today i am #WorkingOutLoud, sharing a section of the writing on Learning Science from the Modern Learning Capability Programme. These three pieces discuss qualitative and quantitative data, and how you can measure anything. This is early stage work, shared out of context.

Quantitative data is that which can be caught in numbers, measured precisely, and replicated. Scientists love quantitative data, as it has a certain unarguable quality. For example, if i weigh myself, and say that i am 75 kilos, you could weigh me, and i would still weight 75 kilos. Unless your scales were broken, but then i could quantify that, by doing comparative tests (weighing a bag of sugar on each, and demonstrating the difference).

But perhaps you would weigh me and say that i weigh 11.8 stone, but that’s ok too, because the two different scales of measurement are correlated.

Correlated scales use different gradations, or gaps, to measure something, but the thing that they measure is the same. So, once you have a couple of data points, you can correlate the scales. [e.g. if you measure me, then measure an apple, in both scales, then you should be able to figure out how each scale runs. This works the same for the temperature too: i can sunbathe in centigrade, Celsius, or Fahrenheit, all at the same temperature.]

Qualitative data, by contrast, is observational: it characterises, or approximates, the thing itself. So you could weigh me and say that i am 75 kilos, or you could try to pick me up, and say that i am heavy. Both are correct. The first is quantitative, expressed as a number, and measured in a replicable way, and the second is a characterisation, which may be unique to me.

My son, at four months, weighs seven kilograms. You could measure him (if you could get him to sit still for long enough) and he would weigh the same, even on your broken scales [Remember, your scales record a number, not an absolute: so your scales may say 1kg, but that’s not the same thing as a scientifically defined kilogram. Yes: there really is a block of metal, sat in a series of nested glass bell jars, in Paris, that is that actual, and only, real kilogram. Every country has a series of local weights that balance against that. You couldn’t make it up. Although it’s on the way out… because every time it is picked up, or a stray atom falls off, the ‘kilogram’ legally changes. If you polish it, you actually change the legal and scientific standard. Which is why it’s being replaced by something altogether more complex, involving clever scales, which i will let you google for brevity.].

You may say that he is light. But for me, he has got a lot heavier since he was born, weighing just under four kilos. So i think that he is heavy. And we are both right. That’s the issue with qualitative data: he is both light, and heavy, at the same time as weight exactly seven kilos. Or probably eight by the time i finish this explanation.

No problem, you may say: stick to quantitative data, and everything will be fine. Except that whilst it’s easy to measure weight, other things cannot be measured quantitively. Like happiness. Or God.

I cannot determine if you are happy or not: i mean, i could measure levels of your stress hormones, your pulse, skin conductivity, and take a look if you are frowning, but none of those would definitively tell me how you feel. But if i ask you, i would probably know.

Qualitative data is typically self reported, and concerns feelings, subjective interpretation, and belief.

Sometimes researchers try to constrain this, by presenting options: instead of asking ‘how do you feel’, which may produce an annoyingly diverse range of answers, they may say ‘Do you feel [a] happy, [b] sad, or [c] confused’. Indeed if they are good at research, they may first ask a hundred people to give free form answers, and then analyse those to produce a subset that best represents the range of expressed options, but constrains them down to e.g. 3.

If they then use that scale to survey say a thousand more people, then they could come up with a result that says ‘400 people (40%) are happy, 200 people (20%) are sad, and the rest (40%) are confused. Which sounds pleasingly quantitative. But it’s not: this is a case of generating quantitative data from qualitative research. The number describes the result, but the result is a subjective observation: just because we have a number does not change the nature.

Problems with data

There are many ways that data are manipulated, or misunderstood, to cause confusion, or to deceive, by design, or accident.

Masquerading qualitative as quantitative is one way, but do remember that it’s also done to carry out valid interpretation, and to create presentable results.

One of the biggest problems is that just because we can measure something, does not mean that it is valid data. Earlier i said that i could measure your skin conductivity to see if you are happy. But how do i know that skin conductivity relates to happiness? It’s good practice to track back, to discover if what we measure actually correlates to the thing itself. It may be that i can measure ten different people, and get ten different skin conductivity results, but that may not correlate to their state of happiness. Maybe some of them ran here, and skin conductivity relates to sweatiness and fitness!

Another problem is when we select data to prove a point: let’s say i am measuring happiness, and i survey 500 people and the final result is that 40% are happy. Chances are that as i collect the data, the results fluctuate, so in the first two hundred people, sixty percent were happy, but the final three hundred dragged the average back down. So why don’t i just survey another two hundred, and perhaps get lucky, and hit a happy team? By so doing, i am still measuring a thing, but i get to 50% happiness. Which is real?

This is why researchers need to define what they will measure at the start: we are taking a snapshot. If we move the camera, we will blur the picture.

Finally, there are issues where we selectively choose perfectly good data to support a point or conclusion that we wish to draw: conclusions should be drawn from data, not data to support a foregone conclusion.

Why you can measure anything

The qualitative to quantitative switch means that you can measure anything; how you feel about breakfast, the weight of your shoes, or the validity of democracy. But it does not mean that the scale of measurement you choose, or the mechanism of measurement, is valid. So measure anything, but do it with care. And be both wary and careful of the measurements that people give you to prove a point. Especially when they are charging you for it.