Which humans raised our LLMs? The built-in bias the C‑Suite needs to know about.

As a team we made a commitment to learn together in order that we accelerate our learning, and have common reference points across our team. One of the milestones in this journey has been “Which Humans”. This is a research paper from the Department of Human Evolutionary Biology at Harvard University. It became a milestone for us because it reinforced a key point about the frontier LLMs that we thought it was important everyone knew, as it both hints at the potential for the ‘Divide’ (the inequity in AI access and opportunity between different parts of society, and different parts of the world), to become even wider, but also brings to life the limitations of how LLMs learn - which is important for anyone looking to use them in the workplace or at home to understand its limitations, and biases.

When you consider LLMs, and what they do, it’s common to refer to them producing ‘human-like’ outputs. The key question the researchers here asked was; which humans exactly do these responses mimic?

The answer of course is “the humans they learned from”. If I grow up in Mauritius, I’m going to learn from the society and people around me, but also mixed in will be the popular culture I’m exposed to online and on TV. Despite globalisation and the proliferation of Western media, we have largely seen the preservation of clusters of countries who broadly share similar cultural values and norms. The World Values Survey shows (below) how although there are changes over time, there is still a broad clustering that seems untouched by globalisation, at least on a macro level. In other words, we are all still (broadly) the products of the societies that raised us.

Source: https://youtu.be/ABWYOcru7js - The Inglehart-Welzel World Cultural Map - Live Version - World Values Survey 7 (2023). Source: http://www.worldvaluessurvey.org/  

So, who raised our LLMs? In other words, which humans do our LLMs learn from, and therefore whose values, and behaviours do they exhibit?

The answer is WEIRD ones.

That’s Western, Educated, Industrial, Rich and Democratic. But mostly Americans to be honest. Well, at least in the case of Chat GPT.

The Harvard researchers took Chat GPT responses, and compared them to how individuals in different countries responded to the same queries. They then mapped out the degree of similarity and difference of those countries to the LLM response. Ethiopia, Pakistan, and Kyrgyzstan were the least similar by a long way, and the closest were the United States and Uruguay, followed by a cluster of countries including Canada, Northern Ireland, New Zealand, Great Britain, Australia, Andorra, Germany, and the Netherlands. (By the way, there is a great rabbit hole to go down comparing USA and Uruguay. Long story short; Uruguay’s stats really add up!)

To understand this is not hard. LLMs are trained on textual data - the internet, books, and other reference materials. When you look at the volume of available textual data (the internet, and digitised sources), it is predominantly WEIRD societies that generate and use this. Therefore the LLMs are closest to the WEIRD societies that they have learned from. They are primarily modelled on Americans, for Americans, and then reinforced by usage from the countries most similar to them; not forgetting that the implementation and refinement of these models probably took place in Silicon Valley too.

So does this matter?

Surely, if it’s WEIRD countries who have the affluence and access to use Chat GPT, does it matter if Chat GPT thinks like the people who mostly use it?

Well, possibly not if you’re a majority ethnicity Brit, an American, or even a Dutch person - but it does mean that the 3.6bn people on the planet without internet access aren’t included in this training data, and where ethnic or other minorities in a society are under-represented in online presence and content, they will also be underrepresented in these models. This makes the AI divide very real. Tools set up, designed, and tailored to predominantly white, western educated nations, just aren’t going to be as effective or useful to societies that are not like that.

Where does that end? The level of funding that Anthropic, OpenAI, Google and others have to invest in their models is enormous. What is their interest to diversify its thinking patterns, and how can a local LLM designed to cater for its citizens possibly compete with that? (A list of regional or language specific LLMs available at the end of this article). This research only focused on ChatGPT and didn’t consider other models, but it’s safe to assume that if the originator is a well funded global company originating in the West, that the story is primarily the same all over.

Understanding the LLMs means understanding its training, not its motivations

The more I see the model researchers talking about their models, the more it is clear that they only ‘understand’ their own products, in the same way that a psychiatrist ‘understands’ their most unique patient, or a lion keeper ‘understands’ their alpha male lion.

This Harvard paper also proposes human psychological evaluation methods as a way of understanding LLMs. This might sound bizarre, but of course if human psychology is manifested in the decisions, behaviours and communications of humans, then of course LLMs will also mimic that, but without the underlying psychological, cognitive, neurological and hormonal mechanisms that actually drive our psychology, just a good eye for what human psychology ‘looks like’. So although the human psych evaluation helps us understand the outputs, it doesn’t help us understand how the LLM arrived at that decision.

So although one would hope that over time, the frontier models would train across all of humanity, and tailor their approaches to the cultural norms preferred by the specific user. But we don’t even know if LLMs have the ability to distinguish between those nuances of humanity and to reconcile being able to have multiple cultural viewpoints or values held at the same time. If there are very few or no humans who can meaningfully do that, why would an LLM be any different? It may be that we if we want ‘human-like’ responses, we will have to be clear about which humans’ responses we want it to predict.

What’s the ‘so what’ for enterprises?

First - Use this example to help your people understand the limitations of LLMs. They are not all knowing, all seeing, wise beings. They are probabilistic prediction engines, trained to think and behave like certain humans from the past.

Second - It’s simply important to understand and be wary of the inherent bias that this creates. This might manifest itself simply during day to day usage of asking AI ‘what good looks like’ in different situations or scenarios. So long as you are aware that this isn’t going to very well represent a full society view, and in fact, will most likely give you an American viewpoint, then at least you’re informed and can act or react accordingly.

We know of companies conducting early stage ‘user testing’ of products, or copy, using LLMs. This is a smart move because you’re able to get many more, very rapid alternative perspectives, and more testing is always better than less. You can ask it to play a whole range of roles that you’d never be able to recruit for - and so refine your product much earlier before putting it front of expensive human test subjects. However, we now understand that there’s a very clear limitation that means we can’t rely on AI to be truly representative even within a particular society, let alone for a global world view.

Brand, product and marketing colleagues of mine now, and in the past never would have accepted the outcomes of focus groups if they’d been exclusively populated by the relatively narrow group of society that our LLMs are trained on.

More global enterprises should also consider how they might support local offices to better tailor the outputs of frontier models through simple custom system prompts (instructions) for agents, custom GPTs or Gems.

Almost finally - We are increasingly turning towards Small Language Models for specialist applications like medicine, realising the limitations of the LLMs for specific use-cases. Our view is that you should also consider non global models, or specifically trained small language models for applications where reflecting a specific culture, values and norms is important.

Finally - there is the moral compass to consider

At Valliance we choose responsible optimism: actively engaging with powerful AI to shape outcomes, not retreating. We aim to apply it carefully, transparently, and accountably. We commit to open knowledge and community benefit so AI’s gains are widely shared, not hoarded.

So, as a close, we’d ask you to consider the Perplexity-AI-generated resources below and look for opportunities to support, raise awareness of, or even fund. The more respected voices in the enterprise world that recognise these challenges and campaign for them to be solved, the narrower the divide will become, and the more of the world will be served by AI fit for them, and will represent them to the rest of the world.

#AI Transparency - At Valliance we are committed to being transparent about where and how AI is used in our work.

This article was written by a human, but with the ability to ‘ask questions’ of the source PDF research paper using Adobe’s AI Assistant in Acrobat.

The resources below are generated by a research prompt in Perplexity which has the ability to query internet sources. Be aware that the results below may be misleading or inaccurate, and have not been checked against the sources, which have been included. As you can see, I also didn’t ask it to write in British English!

[AI GENERATED CONTENT FROM THIS POINT]

Localised LLMs:

Southeast Asia

• SEA-LION: Developed by AI Singapore, SEA-LION is trained on Southeast Asian language data—including Bahasa Indonesia, Thai, and Vietnamese—with proprietary tokenization and significant participation from regional content.carnegieendowment+2

• PhoGPT: Built by VinAI for Vietnamese, this model was pretrained from scratch on a large Vietnamese-language corpus and further fine-tuned with local conversational data.carnegieendowment

• MaLLaM: A family of Malay language models trained on tens of billions of local tokens, aiming to remove English-centric bias.scmp+1

• ILMU (Malaysia): Another Malay-focused model, cited for leveraging local data sources to build meaningful applications for the Malaysian context.scmp

Africa

• African Next Voices (ANV) Project: An ongoing initiative to train models on 9,000 hours of spoken language from multiple African languages, including Yoruba, Igbo, Hausa, and more, with open-access datasets.nature+2

• AfriBERTa: Focused on African languages, trained on locally curated data, with an emphasis on underrepresented dialects.arxiv

• GSMA Africa AI Initiative: A collaborative project across African telecom providers to build LLMs for local applications, trained with regionally relevant data.mtn

Community Projects

• Masakhane: Grassroots collective developing datasets and models for over 11 African languages, focusing on community participation and localized data.kabodgroup

Key Characteristics

• These models are often open-source to ensure broad local participation and adaptation.sea-lion

• Their training datasets are sourced intentionally from local texts, oral records, and annotated corpora to reflect daily life, idioms, and cultural practices specific to their communities.nature+1

• The success and scaling of these models depend heavily on the availability of high-quality local data, often gathered via community initiatives and public sector support.kabodgroup+1

For further detail, you may visit the SEA-LION project, Masakhane community, and the African Next Voices websites, as they provide both technical details and dataset resources for their respective respective LLMs.

Key Organizations and Initiatives you could consider funding, supporting or driving awareness of;

• Masakhane Research Foundation: This Africa-centered NLP collective organizes, funds, and develops language models for African languages with a focus on local ownership, community-led research, and data sovereignty. They prioritize cultural nuances, local language characteristics, and open-source development for equitable AI innovation.cacm.acm+1

• Māori Data Sovereignty Network: Active in promoting indigenous control over data—including for AI and LLM research—especially for New Zealand’s Māori and other indigenous peoples, emphasizing cultural relevance and ethical data practices.adalovelaceinstitute

• Maritaca AI (Brazil): This organization builds LLMs tailored for Brazilian Portuguese and local societal needs, ensuring models reflect regional norms, use-cases, and cultural context.cacm.acm

• BharatGen (India): Launched by the Indian federal government, BharatGen develops multi-lingual, multimodal open-source LLMs trained across a wide variety of Indic languages, supporting underrepresented dialects and community-driven applications.cacm.acm

• Funding Initiatives: In the US, projects like the "Hard Data to the Model: Personalized, Diverse Preferences for Language Models" funded by the National Science Foundation explicitly address the need for personalization and increased diversity in LLM training data by supporting research targeting underrepresented groups.mendoza.nd

Academic and Open Research Efforts

• ERC-funded DataDivers Project: A European project working to measure and enhance dataset diversity for fair, robust NLP and AI models internationally.cordis.europa

• University of British Columbia Research Group: Actively working on enhancing LLMs with culturally diverse knowledge and promoting their adoption in research and actual applications.theconversation

These groups represent just a few examples. There is growing recognition in both policy and research communities of the need for LLMs to better reflect linguistic, cultural, and regional diversity in both data collection and model design.adalovelaceinstitute+2

[AI GENERATED CONTENT ENDS]