What is bias? / Data for Bluffers #7

14 February 2022

Learn how writing a pub quiz can make you better at marketing as Tom and Ed discuss how bias can be good and bad when it comes to data in sales and marketing.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Tom
Hello, and welcome to another edition of the data for Bluffer’s podcast. This week. I wanted to ask the question to ed, how do you know if the data you’re looking at is true? There’s many things that can skew how, how good our data is. And today I wanted to discuss with ed one in particular and that bias what it is, make sure we can understand it, but then really make sure that we can use that to get the most impact and value from our data. So let me ask what is bias ed?
Ed
I mean, everyone’s probably heard the words and sort of has an idea of what it, what it does mean it’s like they predisposition really to, you know, think a certain way or act a certain way, given a scenario or some information that isn’t necessarily based on that information itself, you think something is better or more worthy because of an underlying attachment you have to it, yeah. In particular, when it comes to data science, there’s sort of two heavily related concepts of bias that quite important. The first thing is the sort of recognition that humans have biases. Okay. Humans have what are often called cognitive biases. And these are almost unconscious ways of thinking that affect how humans interact with data. Okay. It’s also important to note that models have biases in some ways it’s a much more technical idea. You basically, the, a bias in a model sort of comes from how do you construct your model? How do you choose the data for it, et cetera. And then how does that affect what the results are?
Tom
So I think what you said there was really interesting, um, about humans having, having biases. And, you know, we, we often assume people in official positions often operate without bias. You know, we think, we think judges will come down on the same decision and, and there’s, there’s been some studies, um, that were conducted in America, but, and there were plenty of observations they made, but, you know, one example was judges who were fathers of daughters were seen to, to rule more harshly on, you know, sort of sexual assault cases against females. No real Supreme eyes, as I say that out loud. But I think what, what gets interesting is when we compare that to how we, you know, people are worried about bias in data, they’re often comparing that to what they, they believe, I guess, is, is an unbiased view from, you know, from the human equivalent.
Ed
Definitely. I think that’s a, that’s a really good example of sort of how a, how a bias that it’s something we all understand. It’s not, you know, we don’t criticize people for having that bias as humans. We kind of understand that idea, but at the same time, it does get in the way of producing what you might call a fair outcome. And I think it’s also a really Pressy and issue at the moment, kind of taking, taking the example little bit further that we are, we are now looking at, um, using computers, for example, to do sentencing. Right. So there’s algorithmic sentencing. Yeah. Yeah. Okay. And that’s a really good example of where your, your model that you are building is inheriting all of the bias that was in your previous system, because basically that algorithm is just trying to reproduce previous sentencing. Yeah. Yeah. So any bias that was in previous sentencing is gonna be in your new model.
Tom
Yeah. Yeah. So my, so my example of, uh, fathers of girls. So if that’s used to teach these models how to operate, then they, they buy a association are gonna act more like, uh, a father of daughters.
Ed
And I mean, that’s something that might even itself out, but if it’s a characteristic of the defendant in particular, okay, that is, that is causing the bias. Then that’s something that’s gonna get baked in and only gonna get worse as the algorithm is used.
Tom
And when it comes to writing these algorithm, them, how important is it for, for the wider team to understand the bias of an individual data scientist?
Ed
I mean, it’s, it’s incredibly important for people to understand their own biases. And also those of people they work with as a team, something you see a lot is that people have a tendency say for like things that have worked with four or methods they viewed previously, right? That’s, that’s a really strong bias is sometimes called the outcome bias, right? That if, if you do something and it works once you then assume that that’s means that was a good thing. And this is a much broader idea as well. You see this in all jobs. So not just in data science, for example, if you are, uh, running a sales team, you hire a new sales guy, your sales go up, you assume that that’s because he was a good salesman, right? It’s not because your marketing campaigns kicked in and, and that’s driving sales through another route. Or, and there, there are, you know, ways to analyze this in data and kind of get closer to the answer. But your natural inclination is often, often to think something has improved after I did took this action. And therefore the action has caused the improvement.
Tom
Yeah, I think it’s, it’s, it’s probably the, I dunno if you’ve ever read the book thinking fast thinking slow, but it, you know, it talks about system one or system in those sort of outcome bias examples. That’s, that’s our lovely system, one brain just hooking onto an answer that we don’t have to do any thinking about and, and serving us up what we, think’s the easy answer. Right. You know, this person joined results have gone up, therefore that’s what’s happened.
Ed
Yeah, exactly. And I think, I think that’s, that’s actually true of a, a, a lot of bar bias or like the majority of biases come from our brains. I don’t wanna say being lazy because that kind of makes it sound like a, a uniformly bad thing, but not having the time or being, or the information to analyze the situation fully. Now, there are some times when you really, really do need to jump to a conclusion quickly. And at which point, relying on your bias is the best thing, because you can end up at the other end of the scale where you just react to the most recent piece of information, cuz you don’t include any of that, that bias
Tom
Hearing what you’ve just said, then it gets me thinking about how we review datas. You know, so if someone comes in and presents a presents, a set of figures, you know, we’ve talked about the bias in the person, potentially creating those figures, but there’s bias in the decision maker. Uh, and what that’s leading to me to think is this is probably, this is probably an area aware diversity in an organization works really well because you need decision makers reviewing the data. Who’ve got a variety of world views, I a variety of their own biases. So when they interpret the data, they all look at it with their own biases. Is that a fair comment? Or am I, am I missing the point?
Ed
I think it’s, it’s definitely true that, you know, people’s biases can cancel each other out to a certain extent. And, and definitely if you have, if everyone has the same biases, then you end up in a what some people might call a groupthink situation where everyone is assuming the same things are true to begin with. Yeah. And therefore, anything that confirms that to talk about another time bias or a confirmation bias, anything that confirms that is considered good and true and valid and anything that goes against that is considered with some form of skepticism. Now it might not be thrown out completely, but the barrier for, for evidence is much higher.
Tom
So you, you, you mentioned another one there. So we, we briefly talked about outcome bias. You’ve just brought up confirmation bias, confirmation bias. I, think’s always a really interesting one, especially for sales and marketing teams. Um, because you often, that’s often a really big one that, that in those departments you are fighting, you know, because when you speak to someone, you know, if, if they believe it based on a past experience quickly to believe it, if not, you’ve got to fight that. Right. And as an ingrained bias, it can be a, it can be a hard one to, to displace.
Ed
I mean, confirmation bias. It’s kind of important to understand where it comes from. So confirmation bias, particularly at work where it comes from is that people don’t wanna be told they’ve been doing the wrong thing.
Tom
Yeah.
Ed
Okay. For, you know, the last six months. And on top of that, if everything’s moving in the right direction, it can be very hard to understand that maybe it’s not moving there as quickly as possible.
Tom
You know, that that’s probably where slightly straddling topics a bit, but loss, aversion comes in, you know, people think, well, we could change, but, but it is going up and I’d actually rather our trend of up slowly then risk any change that might, you know, cause things to drop, if you like that, we’re all averse to losing anything.
Ed
Well, yeah, this, this goes, this goes to what we sort of briefly mentioned earlier about the idea of that biases protect us from making rash decisions. You know, you might have some data which suggest you should be doing something differently, the possible or negatives of that going wrong. Greatly outweigh potentially the positives of adding 1% a month to your sales.
Tom
Good. So let’s get back to the examples then. So we we’ve covered we’ve, you know, we’ll covered them briefly, but we talked about outcome bias, um, which was, you know, salesperson joins sales go up. So, you know, we think that that, that they’re linked. Um, we just talked to confirmation bias. What, what are the other ones that I guess, catch people out? What, you know, what are the other ones that, that we should be aware of when we’re, when we are looking at data or, you know, when we are creating data.
Ed
So there’s a, there’s a few that I think are, are important. Um, there’s something that people might call say availability bias, but it’s BA basically idea that if you can recall something you are more likely to then act upon that. Now there’s kind of an obvious case in that statutory, this is like, you can’t act on something you haven’t remembered, but also when you remember something easily, you often consider it to be more important than things that you’ve struggled to remember. Or even if you get remind, if you have to be reminded of that thing. And this interacts with confirmation bias, because, because we quicker to believe and remember data that supports our points of view avail, you kind of leads to our, our confirmation bias is being reinforced all the time.
Tom
It sounds linked to something that we talk about when we speak to customers, which is the, the bad mine H phenomenon, which is why people ignore adverts. You know, we, you know, we, we see anywhere between five and 10,000 ads a day and we ignore most of them, um, because our, our brains filter out the unfamiliar. So is, is availability bias kind of linked to that?
Ed
I think so. Yeah. So that’s kind of like a frequency, illusion idea that suddenly once we really notice something, so once it becomes available in our mind, we then notice it everywhere. Yeah. That, that isn’t availability by, which is saying, okay, because I can remember this, it stands out to me.
Tom
If I, if I go and buy a red mini, all, I then see on the roads, you know, for the next few years are red minis. And it’s not because there are any more red minis on the road than there were the day before. It’s, it’s just the fact that our, our brain has unfiltered it because if our brain prompted us for everything and, and made us consciously aware of everything, you know, we’d never walk more than a foot to, out of bed, I guess.
Ed
But I think that’s, yeah. And I think, I think this is availability bias to something really, to consider when you are looking at sort of data solutions as well. If someone’s got you thinking about locations as a way of improving your marketing, any data that comes to you about that, you’re gonna be like, oh, that’s like what? That someone was telling me the other day. Right. And that causes it to stick in your mind more. And it, and it is also important to say that for data teams that I think there’s always a danger that because you want to look like you’ve got a lot of data and you’ve, you know, you’ve done a lot of analysis you put E or you produce what are basically quite easy bits of information that reproduce people’s biases and therefore make it even more available and even worse in the future.
Tom
Yeah. Okay. Are there more that are worth kind of talking through?
Ed
Um, I think one, that’s sort of a little bit linked to outcome bias and is again, really quite important in decision making. And that’s sort of what some people might call like a self-serving bias. This is just our tendency to kind of take, take responsibility for, uh, things that go well and to put things that go badly down to luck, right?
Tom
Yeah.
Ed
When really it, you know, the, the truth is that you’re normally somewhere in the middle. So things that go well are, you know, somewhat down to your actions, but also a bit down to luck and things that go badly are down to your actions and down to luck.
Tom
Does some of that come back to, I guess I wanna say storytelling, I don’t think storytelling is the right word, but, you know, let let’s, let’s say for example, we’ve, we’ve launched a new campaign and something has gone really well. And, you know, I wanna report that back to the business that, you know, the obvious question is why is that gone so well? Um, and, and you look for the answer, you know, so, well, I did ABC and then we did this. So it’s probably ABC. Um, is that, is that what we’re talking about?
Ed
So there’s kind of two, two pieces to unpack there. Firstly, is this the, uh, general fear of standing up and going? It went well, but I don’t know why. Yeah. So no one wants to not have an answer. Yeah. And then, okay. So what is, or what is the answer you fool on the initial? You know, your favorite answer is the thing that you did, your li your less favorite answer is, oh, it turned out it was nothing to do with us. It was because this viral video that we released 15 years ago finally, like took off on the internet. Right.
Tom
I get that all the time on YouTube. You just, you see this, it serves you in a video and it’s got millions of views and you look at it and it’s like eight years old.
Ed
And you could be sure if that, you know, if that was a viral marketing video, then someone would be stopping a meeting going, oh, it’s because we, you know, yeah. Readjusted us paid spend or something like that.
Tom
One I always look at, and I guess the question is, is, is this a form of bias? But, uh, I think we’ve read it before briefly, but is this kind of concept of SP correlations? You know, we, we see something, um, and something else happens at the same time. So we, we assume they’re linked, you know, like you say, we’ve just, we’ve just changed our, our paid media approach. And we’ve seen a massive, massive outcome, um, a massive, massive increase, which could be related, but equally, yeah. There’s a lot more variables and it, and it may not be, is, is that, is that a form of bias or is, is that something different?
Ed
Uh, I think that is, that is a form of bias. Like as, as humans, we’re, we, there are sort of evolutionary reasons why this might be the case. If you think you see a pattern that looks like a tiger, it’s better to get out of there than to make sure it’s a tiger before you start running. Yeah. That’s an example where bias is, is really helpful because striving for the perfect answer is gonna have costly consequences. Yeah.
Tom
Yeah. Okay. And anymore we should focus on from a, from a bias point of view.
Ed
One would be there’s, there’s tends to be a bias to assume that everything, you know, is quite well known, really good way to experience this is if you’ve ever tried to write a quiz. Okay. When I was a student, we used to look after people staying at the college, we used to write quiz is for them. And it’s really hard. Like when you’re ever writing a quiz, you, you think every question is either impossible because you don’t know the answer or incredibly easy, because you do know the answer and it, and, and in the data world that, that sort of happens. Right. So you do a bit of research, you find something out.
Tom
Yeah. Okay.
Ed
And then very quickly it can become normalized that, okay. Now everyone knows that. And in the marketing world, what that breaks down to is you think, oh, we are doing this thing. You do it for a couple of months and you are still the only people doing it, but for you, that’s now old news. And you look for the next new thing. Yeah. Just because you assume that you are behind the curve.
Tom
And, and I guess also it really means there’s a lot more value that these teams can put out there that they assume is of little value, because it’s been normalized in their head, but actually for, for their, for their audience, their, you, their customers or whatever it may be. There’s probably a lot more, a lot more content they can do. And it’s just not viewing it through that lens of, ah, this is too easy. Everyone will always always know it.
Ed
Everyone’s probably experiences as well. When you start a new job and you go into a different team and suddenly it’s like, oh, you do this all the time. And everyone there is kind of like, well, obviously, and your, and your response is okay, I’ve never really thought that before, because you know, in, in the previous company that just, wasn’t a, a, an angle that people looked at it from,
Tom
If nothing else out the back of this, this conversation, that’s gonna give me a kick. And actually I might try and write a quiz for someone, um, just to kind of reinforce that. So that, that might, that might be my top tip outta this. Right, right. Go, go, write a quiz and then reflect on how you view your everything else you do in the world in that it’s either impossible or too easy.
Ed
I just wanna Chuck another one in the ring, which is related to what we just spoke about. So in long projects, you often you do the big stuff at the beginning. And then towards the end of the project, you sort of lose sight of the, the problem you were trying to solve and get, and end up focusing on what, yeah. Probably to the product are, are sort of trivial things because people talk about them a lot in the team and they, without them a lot, because they’re easy to talk about it comes up at every meeting because it’s easy to come up with a quick, uh, you know, a quick opinion on it. This is sometimes called, uh, the bike shed bias or bike shedding, which is the idea that, you know, people, people designing a, a nuclear power plant or something like that spend as much am designing the bike shed as they do the main reactors, because the main reactors are hard. You send someone off to do some research and they gotta, they come back in a few months. Meanwhile, you’re having meetings in every meeting. Someone’s got something new to say about the bike sheds.
Tom
So in our, and we’ve we in a previous chat, we talked about AI and we tried to kind of unpack that subject a bit. And within that, we talked about heuristics and, you know, you know, essentially rules of thumb is, you know, I guess, is, is bias. Really another form of a hu is, is a heuristic. Or, or is it, is it more complicated than that?
Ed
No, I think that’s, that’s definitely the case. And if people who are sort of more involved in, in data science will have heard of the bias variance trade off. When we talk about bias in data, that is exactly what we’re talking about. We’re talking about the bias is the heuristics that go into the model. The variance is the amount we depend on the data we put into the model, as the name suggests there is a tradeoff between those two things. So in, in every model and, and the truth is in every decision making process, there is a trade off between your heuristics. What are you putting into your, your decision making process, what you come to it with, and the amount you’re responding to the data that’s put in front of view is, is really hard as a human in particular, to get rid of all of those heuristics and try and respond to what’s right in front of you.
Ed
And I would go so far as I think it’s probably dangerous to try and do that. You know, this is an example which always amazes me that we have this in our legal system, but, you know, in, in like in court, the jury’s always told, you know, don’t, don’t read anything you’ve read outside about the case and all that. Um, you know, only you can only hear the evidence that’s here. And then sometimes, you know, if a lawyer speaks out of turn about some evidence that wasn’t allowed in, they’ll say, you’ve gotta forget that you heard that
Tom
Basically. Yeah. Yeah.
Ed
And I always, I always think like how, I mean, surely people know that’s impossible. Right.
Tom
You know? Yeah, yeah, yeah. It is a, yeah. So it’s almost in, in that example then between the, you know, the, the heuristic or the rules and, and the bias that we’ve talked about before to try and find some line between them, it could, it is potentially the ones you’re aware of and the ones you’re not right. Because when you are, when you’re, when you’re writing something, you are, you are factoring in some rules of thumbs and heuristics, you know, about, but a lot of the stuff we’ve talked about might be, might often be rating in a subconscious way. You know, they might be, you know, you might be not putting it in there and thinking on it, but actually subconsciously it’s, it’s getting baked in.
Ed
Yeah, definitely. And this is, this is where, when you’re looking at data products, data services, you’ve gotta be critical of, of what’s going into them from a, a heuristic perspectives. What are they assuming is true. That’s gone into it. And then out of that also, do they have any evidence that the thing works? Yeah. It’s not necessarily that you’ve overcome the bias, but you don’t really care about the bias. If the result is a marketing strategy that’s working really well,
Tom
I guess, I guess there’s a, a portion of critical thinking needed then when you’re looking at reports, right. Because, you know, let’s just get, say, you’ve given a report, you know, whether that’s by your internal BI team, whether it’s by an external tool that you know, where wherever it comes from, but actually a lot of these dashboards that people are shown, contain a lot of and information. Uh, and sometimes a disproportionate amount of effort might have gone on reporting about the bike shed, you know, to, to use your example earlier, which will give it, you know, a disproportionate, um, waiting, you know, so, and, and that’s easy if, you know, if it’s easy, if you use an example, like, you know, these are the results of the nuclear actor, then these are the results of the, the, the bike shed. But actually if you’re talking about maybe technical performance of, of a campaign or, you know, some lower level metrics, it can be harder to decipher what is the, what is the bike shed? And what’s the reactor, uh, and really people should be applying level a critical thinking just to try and understand, well, okay, there there’s four charts on that, but really how, how important is that towards the bigger picture of what we’re trying to talk about here?
Ed
You know, this goes back to a, a conversation we’ve had previously about setting targets for data projects. There’s a, there’s a tendency to, to set targets based on what’s easy to report. Attribution is a favorite example of this, right? You can do attribution in a certain way. So then you set KPIs, which are about, you know, clickthrough rates, for example, on particular adverts rather than brand building, you know, and that, and that impact on sales, which is much slower process. That’s definitely part of the same thing that you could, you can end up targeting water on ultimately trivial metrics. So a good example of this, some people have called it the corners problem before, and that’s the, basically, if you wanna predict who’s gonna win a football match, you can do just as well, or I think even better by looking at how many corners each team gets, rather than how many goals they score. The problem is if you set up your football team to try and win corners, you aren’t winning any football matches, right?
Tom
Yeah. It’s a good example focused on focused on the metric rather than the objective.
Ed
Exactly.
Tom
I think it’s been fascinating. I’ve, I’ve learned loads. I think it’s been really interesting. Um, but we’ve covered a lot. So what was your one takeaway be if you were to recommend, you know, one thing that’s gonna help people get more out of their data, you know, when, when considering bias, what, what would you be, what would you be saying to people?
Ed
I would be, I would be saying, like, be, be critical of things that come in front of you think about your own biases, think about the biases in the data and ultimately think what you want out of it. I think that’s the only a, to really overcome buyers is to think, what do I want to get out? Or what do we, as a business, want to get out of this product or service and therefore, is it doing that or is it likely to do that? And in particular, if someone’s presenting to you data on how well their system is working, think about, okay, but is that really what we want? You know, are they the metrics that we will look at at the end of the day, and then partly this also, you know, when that isn’t the case. Okay. But does their story make sense? Is their explanation of how it is gonna help you make sense?
Tom
I’m, I’m gonna add one as well. Um, and I think from my side and welcome to shoot me down, but if you’re gonna review data allocate time for it, making sure that you, you allow five minutes to look at something that on this face of it looks really simple to consider all the things that that EDS said. Don’t just look at it and apply your own confirmation bias and move on
Ed
In the inverse situation. If someone’s telling you something that you don’t believe, don’t write it off quickly. Yes.
Tom
I won’t add another one. Cause otherwise we’ll just keep doing like a sandwich, but I’ll add one more and
Ed
Ideas,
Tom
Tennis. Um, great. So that’s it for a, another, another episode of the data for podcast. I hope you enjoyed the episode as, as much as I did at least, uh, as ever, uh, you know, follow like subscribe and we will both see you for another episode in two week time.

Friends in conversation | Herdify

Sign up to the Herdify newsletter