How Bias Compromises AI (With Examples)



    AI Data Science 101

    Trust plays a large role in many of the decisions we make, and it can be difficult to trust something that can be as opaque as AI. When so many critical, high-risk decisions are being left to a machine whose inner workings aren’t easily understood, it’s not hard to see why AI has been so heavily scrutinised.

    One of the major concerns surrounding AI is that societal biases are making their way into the data sets being used to build AI models. These biases are then perpetuated in very real and often public situations. There are a variety of high-profile examples of AI bias causing serious problems, and these problems matter to the development of AI.

    There are three main elements that can lead to an AI developing bias. If you allow them to happen, trust in your AI will undoubtedly be compromised. 

    Bias in AI - The three elements

    Bias in AI is a real concern, as it can impact the way these models think and learn. Bias ingrained into algorithms leads to discrimination, sexism, racism, and more. There are three areas where AI develops bias.


    Data is integral to the creation of AI. When data is collected and input into AI models, it needs to accurately represent its real-world application. And while data sets utilised in the creation of AI are often of a sufficiently large size, it’s normal for them to be compromised in some way by underlying human and social biases.

    AI systems can’t differentiate between whether the data they’ve received is subjective or objective, so those biases will be picked up by the algorithm and learnt. The biases within the data could be found in any of the factors included in the data set, but it may be found even deeper than that. The data used in AI decision-making runs deep, including factors such as where people live, their interests, spending habits, and more. And while you might not think bias would apply when building the model, there are so many factors at play that it’s difficult for bias to not innately be included by design.

    It can be difficult to retain a meaningful set of data if you try to eliminate every source of bias from all of the different factors included.


    While algorithms can’t create more bias in a model, they can amplify existing bias. AI are designed to maximise accuracy and will use all factors to do so. However, this can lead to biases being exacerbated.

    For example, if a training subset of images depict both men and women in the kitchen, but happen to feature more women, the AI may decide that all people in kitchens are women to try and improve accuracy – despite there being men in the kitchen in the training set. Therefore, underlying gender stereotypes are reinforced by the AI.

    To combat this, algorithms can be developed to ignore certain information, such as background elements that might identify a kitchen.


    The third element lies with those developing the AI – the data scientists and technologists. While they naturally want to achieve the most accurate result possible with the data available to them, it’s important to think about the broader context of the data involved.

    The development of AI needs to change in this regard. For instance, experts in bias and ethics can offer valuable insight and should be involved in the design process. AI engineers themselves should be better educated in ethical matters to avoid future instances of bias.

    AI engineers dealing with this discover themselves coming up against a fundamentally human challenge – the definition of fairness and bias itself. What is fairness? What does it mean to be unbiased? The key is finding an answer that satisfies all, or as many as possible, and incorporating that into AI design.

    Though these three elements of AI bias are all related, the people building the AI are the most important factor behind eliminating bias from the data sets. When it comes to deciding what it means to be biased, that decision shouldn't just be left to the data scientists - organizations should have a clear view from the top of what bias means, perhaps utilizing an ethics charter to outline the definitions. Changing the way AI engineers think about how an AI’s decision can affect people’s lives, over simple accuracy, is crucial. As it stands, AI has been responsible for some highly publicised mishaps due to bias.


    Examples of bias gone wrong

    Bias finds its way into AI through many means, and can often lead to dramatic and widely-publicised consequences. Here are some examples of AI bias negatively affecting an AI system and having to be rectified – or removed entirely.

    Amazon's recruitment tool

    In order to automate their hiring process, Amazon began building an AI tool designed to crawl through past candidate resumes and pick out key terms to then recommend future candidates. The AI would essentially use these key terms to determine the ‘best’ of the candidate resumes and then pass them along.

    However, the AI was biased. Perhaps, because it had trawled through resumes predominantly belonging to men, it had unwittingly come to the conclusion that only men were preferable candidates. It started marking down resumes belonging to women, and outright disregarding applicants who had studied at all-female colleges. 

    Thankfully, Amazon executives lost faith in the project and abandoned it.

    UK home office's visa algorithm

    A controversial decision-making algorithm was scrapped by the UK home office after it created a supposedly hostile environment for people applying for UK visas.

    The algorithm, which has been in use since 2015, was accused of being racist after taking decades of institutionally racist processes and using them in its decision-making. The tool would target particular nationalities for immigration raids, while creating a ‘fast lane’ allowing for white people to be accepted at a faster rate.

    Although the home office claimed that the removal of the algorithm was not an admittance of the allegations of racism, it is believed to be the UK’s first successful challenge to an AI decision-maker.

    Microsoft's Twitter chatbot

    When Microsoft revealed ‘Tay’, their Twitter chatbot AI designed to be an experiment in conversational understanding, they had hoped it would learn from the people it engaged with. Technically, it did.

    Twitter users who spoke with Tay managed to corrupt the AI system within 24 hours, causing it to eventually spout racist and misogynistic rhetoric on the social media platform.

    The AI was designed to become smarter through conversations, but as people conversed with it using all manner of remarks, the AI responded in kind and Microsoft was forced to remove it. It’s not quite as simple as the AI learning from the worst however – many of the tweets were deliberate duplications of other users' tweets, as the AI bot could replicate tweets if asked.

    While the chatbot seemed to espouse all kinds of views, some particularly extreme, Microsoft’s company website stated the model was built using ‘relevant public data’ which had been ‘modelled, cleaned, and filtered’. This scenario raises an important question for AI development – how can AI be taught using public data without it taking on the racist, sexist, and other prejudice data sets from particular users?

    The impact of bias

    When building an AI model, trust is key. It’s the most human element. However, bias can compromise your AI, making it difficult to trust, especially when it’s hard to understand where the data came from.

    We understand the impact of bias in AI development, and how to reduce it in the development phase. Understanding the source of the bias is absolutely essential, which is why it's important to know the three elements behind AI bias. Trusting the model comes after that.

    Boy with robot