Virtual humans meet real emotions: the VHESPER research project

Professor Ad Vingerhoets, emeritus professor and expert in the psychology of tears, approached them with a peculiar problem: how do you study crying without having reliable, authentic visual stimuli? 

The tears challenge 

Nick van Apeldoorn, the leading virtual human researcher: 'It's a fascinating problem, really. Tears are part of a very small research community, perhaps twenty people worldwide who focus on this topic. The challenge is that you can't easily capture natural crying on camera. People don't want to be filmed when they're genuinely upset, so existing stimulus material is essentially non-existent. Traditional methods involved photoshopping tears onto photographs, which is hardly ideal for studying authentic emotional responses.' 

The potential solution? Virtual humans. Through the Virtual Humans for Emotional Prosperity in the European Region (VHESPER) project, BUas researchers are building emotional building blocks that could revolutionise how we understand and create empathetic responses to digital characters. 

A cross-border collaboration 

VHESPER emerged from a network of expertise spanning multiple European universities. The project brings together The Humboldt University in Berlin, the Coginitive Systems Lab from the University of Bremen and Howest and BUas contributing its distinctive specialism in game technology and virtual human development. 

Nick van Apeldoorn: 'What makes this collaboration work is that each partner brings something unique to the table. We have the technical expertise in creating hyperrealistic virtual humans through our AGM (Academy for AI, Games and Media) Research & Development team. Our partners contribute psychological research frameworks and cognitive science perspectives. It's about combining the "how" with the "why", we can create the technology, and they help us understand its impact, and we aid them in their research'. 

Student-led innovation 

At the heart of BUas's contribution is a distinctive approach: student-led research that combines academic rigour with practical innovation. Working alongside MindLabs, a local creative technology hub, master's students from institutions including the University of Amsterdam, Utrecht University, and Leiden University have been instrumental in developing the research. 

Nick van Apeldoorn: 'This year, we've had two exceptional master students who graduated summa cum laude and cum laude respectively. They didn't just assist; they genuinely drove the research forward.' 

In VHESPER we conduct experiments to explore how people perceive and experience virtual humans displaying various emotions. One particularly complex achievement has been developing a sophisticated tears system that operates on physics-based principles. Created by Niels Voskens, a former Games student now working with the team, the system gained immediate international attention when posted on LinkedIn, being picked up by major tech websites within hours. 

Nick van Apeldoorn: 'The tears system is remarkably sophisticated. It includes tear film overflow effects, gravity-based movement, and even subtle details like the wet trail tears leave behind. This level of detail matters because the more realistic you want your virtual human to appear, the higher the standards become. A hyper realistic face with robotic movements simply pulls people out of the illusion.' 

Virtual humans need to find their own niche 

The VHESPER research challenges a fundamental assumption in virtual human development: that virtual humans should be developed to be exactly like us. 

Nick van Apeldoorn: 'My current thesis is that virtual humans shouldn't be designed to replicate human behaviour perfectly. Just as we don't want social robots to be too human-like, virtual humans need to find their own niche, doing precisely what we want them to do, rather than trying to pass as real people. It's a tension between being human enough to connect with, but distinct enough to serve their purpose and not mislead people.' 

This philosophical approach underpins the practical research. The team has tested responses to six different virtual humans displaying various emotions, discovering that recognition varies considerably. Some emotions, like anger or disgust, are poorly recognised, partly because people don't typically associate tears with these emotions. 

The research employs both intuitive stimuli (how researchers believe emotions should appear) and, in later phases, more psychologically grounded approaches. Working with Ursula Hess from Berlin and Denis Küster from the Cognition and Affect Lab in Bremen, the team has built experimental frameworks that students then execute and analyse. 

Looking forward 

Whilst VHESPER focuses on building emotional building blocks for virtual humans, a parallel regional initiative, the Regiodeal project, explores practical applications. This includes developing virtual news presenters that can automatically generate video content from text input, raising important questions about trust, misinformation, and the future of digital media. 

Nick van Apeldoorn: 'We're also conducting a Delphi study, a technological forecasting approach, where we're asking leading scientists and entrepreneurs across Europe to forecast on where this technology will be in ten years. We've deliberately limited it to ten years because frankly, we can't imagine what twenty years ahead looks like.' 

As VHESPER progresses towards its 2026 conclusion, the building blocks being developed at BUas, from tears systems to comprehensive emotional expression libraries, are laying groundwork for applications that span journalism, education, healthcare, and beyond. The challenge, as Nick van Apeldoorn sees it, is ensuring these powerful technologies find their appropriate niche: human enough to connect, distinct enough to serve. 

For more information about the VHESPER project, visit https://www.sia-projecten.nl/project/vesper