2023 has truly been a successful year for Generative AI, we saw a surge in content production that flooded various platforms and media. This sudden popularity was facilitated by the increased availability of easy and accessible AI toolkits that are based on the large language models [1]. Notably, 2023 could be seen as the year when AI transformed into a universal tool, catering to individuals with minimal prior interest or knowledge in technology. Looking ahead to 2024, Generative AI continues its momentum with the announcement and release of more advanced technologies. This time, the focus is not only on generating still images from a few lines of text but also on producing high-quality moving images [2]. It’s important to note that the process of converting text into images works both ways, where an image can be transformed into lines of text. However, in this case, the system may operate without revealing that an image has been submitted and restricts access to the generated text. As image classification becomes a ubiquitous feature in our daily environment, it is crucial that we understand and comprehend how such an intelligent system operates and integrates into our mundane lives.

This article describes my personal journey as a computational artist who uses AI and datasets in my work. I have created a series of artwork that examines the issue of dataset biases in machine learning, where datasets that are not diverse or accurate enough can result in biased outcomes.

Before large-language-models became available, AI in artistic practice was more focused on creating a system that can make choices based on the small set of images it is trained on. This demanded more technical skills, but few accessible tools like Wekinator by Rebecca Fiebrink [3] made it easier to build intelligent systems from a near bottom-up approach where one can create their own dataset and decision algorithms in a completely transparent process. This way of building AI models is called supervised machine learning. It is less computation-intensive, but quite limited to specific uses, and it can work locally without an internet connection. In supervised machine learning the artist has full control of what input it takes and what possible outputs it can produce. This method of building control has been popular among sonic artists to build digital AI agents that can co-control the synthesizer to produce sounds that are reactive to human gestures. The key aspect of supervised machine learning for artists is that one can readily understand and get firsthand experience of creating AI system themselves. The supervised learning is where I also started the AI related artwork.

I first discovered supervised machine learning for building artwork in 2017, when I attended a workshop at Goldmsiths University. Since then, I have been working on creating an AI that can recognise facial expressions, especially emotions and specifically people’s smiles. My first work was a door that would only open if people smiled at it. I wanted to create an experience where people could express emotion to a non-human entity and have their emotion read and judged by the computer. To do this, I started taking multiple ‘selfies’ of myself smiling and not smiling. Then I pixelated all the images and collated them as a dataset. The computer looks at each pixel to understand what it is seeing. The more pixelated the image, the less information to process and the faster the interaction. Instead, if the image is too high in resolution, the result is more accurate, but the speed is slower. I tried different levels of pixelation until I found a good balance between accuracy and speed, making sure that the interaction is in real-time. However, even with the high resolution images the system was still not reliable enough. It would be affected by the background and make errors based on the lighting, colors, and so on. So, I found another algorithm that can identify the structure of the face called faceOSC [4] and extract key features of my face, such as mouth width, eye opening size, mouth height, and so on. By using both the pixel values and the feature information the system become much more reliant.

FaceOSC uses computer vision algorithm that can recognise features of the face and creates a wireframe of mask to provide more detailed information of the expression. On the left the image is showing FaceOSC information of a face with a smile and the right side showing neutral to sad safe.

The system works by connecting to an electric motor and spinning one way when it senses a smile and the other way when it doesn’t. The door was shown in the department as part of a work-in-progress show. During the show, something unusual happened when a white female colleague came near the door, and it started to open even though she was not smiling. I guessed that the system might have mistaken her distinct facial features, especially her large lips, for a smile, so I quickly asked her to join the dataset! What happened next was that the door did not recognise my smile unless I made a big effort to open my mouth wide. This was a very intriguing experience of curating a dataset. At the end, I trained the door with some more colleagues, but it never worked as well as when the dataset had only my face. When the project was shared on a popular online magazine, it became viral on twitter, where people retweeted and commented on a video. In 2017, using AI as a control in artwork was still quite novel, and it provoked many negative and scared reactions from people, mostly because of the uncanny way of expressing and being controlled to fake emotion to a computer. In the infrastructure level, using AI to influence or guide our digital behaviour was already happening, such as on Amazon or Facebook, but the door was an ideal medium to show this control infrastructure in a direct way because it had a clear binary function of either opening or closing, or allowing in or not.

Video of a smile door prototype

Article of the smile door by FastCompany in 2017

Twitter comments of the smile door prototype.

A few months after that, I had a remarkable experience at UK visa and immigration centre in Croydon, where I went to for my visa renewal. Normally at the application desk, a person would take a picture of me and include it in the visa application, but this time they had replaced this with a machine that could capture and verify the photograph. To my astonishment, the machine failed several times to recognise my face, on the screen it kept saying “open your eyes” when my eyes were clearly open! Only after the third tries with each time opening my eyes bigger, I managed to meet the compliance. Having had the previous experience in building the facial recognition system myself, I was astonished but also able to comprehend the situation quite well. The machine obviously hasn’t trained enough with the whole range of ethnicity. What was perhaps annoying is that there wasn’t another way to proceed but really just to widen my eyes to meet the compliance. It is not easy to get the help or to figure out who is responsible for these faulty technologies. This experience motivated me to keep exploring this topic and create more works that can highlight the issues of dataset and enable people to interact with data in a physical way.

Picture of the photo booth machine at Croydon immigration office.

The latest iteration of the series is an interactive door that also responds to people’s smile by opening and closing. The latest installation was called “Simple steps toward a happier space” and it was built with a support from Studio Egret West (SEW) for their new exhibition AI everything everywhere. For this exhibition I had an opportunity to revisit the work and make it bigger and more complete than the original prototype from 2017. The door was set up at SEW’s office showroom and the exhibition started for the London Festival of Architecture and went on for several months. This time I used a bigger dataset that is publicly available on the internet called FER-2013. This dataset has about 30,000 RGB images of faces with different expressions. The dataset has 7 types of expressions: Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral. And instead of using a simple supervised learning model, I used convolutional neural network (CNN) that can classify the image much more accurately (5 frame per seconds) even when operated on a small single board computer (SBC) like Raspberry Pi. CNN is a common model for real-time computer vision perception for robotics. The visitors smile at the door’s pinhole that has a small spy-camera, and when they smile, the compliance score is shown in a segment display, when they reach the minimum score the opens and the visitor can enter.

Simple steps toward a happier space installed inside the SEW office.

Pinhole camera and the instruction signage on the door.

I decided to make the threshold of the compliance score to 70/100, which seemed to me like a reasonable enough score that would let everyone’s smile be detected and that they will not have to force their smile too much. I tried the system with the SEW staff and I saw that even with 30000 faces in the dataset, some biases still exist (like I experienced at the UKVI!) and I lowered the threshold to 65. After that, even though some ethnicities still had very low scores, the door opened for everyone, and I was satisfied with 65.  The exhibition was a great opportunity to have conversations with people about the dataset and use of AI in our built environment. It was an effective way to provoke about AI, and I enjoyed seeing people’s interest in the dataset, the score display worked particularly well, because visitors were not only curious about the door but also paid attention to the compliance score that was shown. I think that the exhibition, gallery was a suitable space to challenge the dataset, and allow the AI to fail sometimes. If someone got a very low score or even got rejected by the system, it is much easier to discuss it as an artwork (which wouldn’t be possible at a Visa centre!), and as I built the model from scratch, I can be accountable and explain why such discrimination might be happening.

On the last week of the show, when the RIBA student winners came to see the exhibition, the worst failure happened when a coloured student kept failing to meet the standard of the AI model based on FER2013 and the door wouldn’t open. After almost 10 attempts the door finally opened, but the score was very low too. It was the only time that I saw someone being totally rejected, and I didn’t know how to respond, as I had a similar experience that was powerful. I told the student about the dataset and apologised for what was happening, but luckily the student was curious and not too astonished by the work, but then this was exactly what I wanted to exhibit and people to experience.

Footage of the installation not recognising a visitor’s smile

AI models are becoming more effective and available, and we will witness a huge rise in their use in different situations and environments. They will analyse our features and recognise us, and because of their nature they will label us with categories even for things that are not easy to classify. The more I learn about AI, the more it feels like a human study. As the algorithms get closer to how our brain works and functions, it almost feels like creating a human cognition. So, when I create a system, it makes me think about how I process visual cues in life. When I encounter a stranger in the street, what do I notice? I notice the gender, the skin color, the hair length, the height, the eyes, the nose, the mouth, then the clothes, and in a fraction of a second, I make bundle of assessments and decisions and forecasts, which are all based on my previous experiences and interactions with people. I will compare what I see and what I have interacted with before, and estimate the person’s traits, personality, ethnicity, and so on. The only difference is that when I make mistakes, I am accountable for them but with the system it’s not so traceable. My future project will include other ordinary objects such as a mirror, to show how deeply the system can be integrated in our everyday interaction of objects. Now I am making a mirror that only reflects one’s image when they meet certain criteria. The criteria involve not only emotions but also ethnicity, genders, and more.

A video showing a first prototype of a mirror that has CNN AI model inside it for recognising happiness

………………………….

References:

1. https://en.wikipedia.org/wiki/Large_language_model
2. SORA-Open AI https://openai.com/sora
3. http://www.wekinator.org/
4. https://github.com/kylemcdonald/ofxFaceTracker

………………………………………

https://freddiehong.com/

All images copyright and courtesy of Freddie Hong

Art, Artificial Intelligence, Consciousness, Creativity, Culture, Technology