AI systems don’t invent bias, they learn it from whatever people fed them, and HR has been feeding biased data into its systems for decades, long before any of it was called “training data.” Performance reviews score women, Black employees, and older workers differently for the same work. Historical hiring records reward whatever pattern hired people looked like before. Point an AI system at that history without checking it first, and it will learn the same pattern, just faster and at scale. Data Society’s Catie Maillard argues the real fix is asking better questions about what you’re feeding it and keeping a person in the loop to catch what it gets wrong, not trusting AI less.
Maillard’s answer to what people should watch for with AI at work started as a list of questions, the kind she says people should be asking before they lean on a system, not after something goes wrong.
“What are the downsides? What am I missing? What should I make sure I do to make sure I can catch hallucinations? What information should I give you to make sure that we’re not discriminating against folks? That’s the other piece to consider, particularly around AI. The difference is, for some reason, because it’s a system, we don’t accept that it’s going to have human biases, and we often take AI outputs at face value, not remembering that it’s really just building on all the content in humanity.”
Catie Maillard, Global VP of People, Data Society
The instinct she’s naming is a real, well-documented one: people are more likely to trust a recommendation when it comes from a system than when it comes from a person, even when the system was built entirely from people’s past decisions. A spreadsheet doesn’t get the benefit of that doubt. A chatbot often does.
 Â
The clearest example of what Maillard is describing is still Amazon’s experimental hiring algorithm, built by training a model on ten years of resumes submitted to the company. Because the tech industry those resumes came from was male-dominated, the model learned to penalize resumes that included the word “women’s,” as in “women’s chess club captain,” and downgraded graduates of two all-women’s colleges. Amazon’s engineers tried to correct it and ultimately couldn’t guarantee the model was neutral. They scrapped it.
That algorithm was designed to find candidates who looked like the company’s past hires, not to discriminate, and the company’s past hires were disproportionately men. The bias was an accurate description of the data, not a bug in the model.
Maillard’s example hits closer to home for most People teams than a hiring algorithm: performance reviews.
“And if you’re in HR, you’ll know that at any point in time, all of the feedback in our performance reviews is biased. All of the feedback in historical writing has been biased. So the information fed into Claude is not necessarily going to give you the results that you need. There is no choice but for us to have humans in the system, either leading it or looking at use cases that don’t use Claude to think.”
Catie Maillard, Global VP of People, Data Society
Textio’s analysis of feedback from more than 25,000 employees across 250 companies backs up exactly what Maillard is pointing to. Women receive 22% more feedback about their personality than men, and 30% more exaggerated feedback. Younger employees get called “ambitious” two and a half times more often than colleagues over 40. Black women receive nearly nine times as much unactionable feedback as white men under 40, and Black men get roughly a third less feedback overall than white women. None of that is hypothetical or historical. It’s what’s sitting in most companies’ HR systems right now, and it’s exactly the kind of text a People team might reach for to summarize a review cycle, draft calibration notes, or train a tool to flag who’s ready for promotion.
Maillard’s conclusion is to ask the questions she opened with, what could go wrong, what am I missing, what would catch a hallucination, what would catch discrimination, before the data goes in, not to stop using AI in HR altogether. That means someone reviewing what a model is actually trained or prompted on, not just what it outputs. It means treating “the AI said so” with exactly as much scrutiny as “a manager said so,” because underneath the interface, that’s what it usually is.
Before trusting an AI system with HR data, audit what it’s actually trained or prompted on, not just what it outputs. Data Society’s AI Advisory work helps HR teams build that audit process, and AI upskilling programs teach teams what to look for.
