for relationship building using emotion recognition



Elective: Designing Intelligence in Interaction


Involve intelligence (AI, ML) in a product which interacts with a human


Relationship building with a drone based on facial recognision


Training the Neural Network in Netlogo with our own created training data. Contact with student team BlueJay building a domestic drone.



Robots are increasingly becoming part of our daily lives which makes human-robot interaction an important topic to design for. In this study, we explore the technologic possibilities of a domestic drone building a relationship with a human using emotion recognition. To our knowledge, emotion recognition has not been used for human-robot interactive relationships with drones. A prototype was made that was able to detect the emotions using a webcam and a neural network. The person would see a screen with the eyes of the drone responding differently to their emotion based on their relationship level. By our research, we are one step closer to drones becoming part of our daily lives.


Key learning points: Machine Learning

The lecture consisted of eight lecture that would provide the information to create a prototype. For our team the most valuable lectures where from Matthias Rauterberg, creating a basic understanding of what is intelligence based on our already present knowledge. Secondly the lecture from Barakova explained the theory behind pattern recognition using neural networks and Jun Hu provided the practical side of creating a neural network in Neuroph Studio. Within the group, I was responsible of creating the network and training it. When the training data is correct, it was rather simple to create a trained network. Although the implementation within Processing gave some problems, the use of real intelligence in design projects is now a possibility for me. The course provided a clear difference of heuristic programming, which normally is seen as intelligence and real intelligence by learning algorithms. Using both within the project, allowed us to create an interesting interaction with intelligence.

Furthermore, I kickstarted the project, because I knew the BlueJay team. I could lead the project by making concept choices based on the interest of BlueJay. Based on the input from Barakova and Jun Lu, we had to change focus from the initial prototype to fit the courses criteria. Based on the MVP process, we were able to bring the prototype one step further than initially expected. Already being able to mimic the emotion of a person using our neural network, would be sufficient for the course. Since we had extra time, we were able to quickly add a second layer of intelligence by the memory of persons and the relationship the drone would have with the person. I was responsible for the logic of the emotions linked with the relationship level. Together with Tim, I figured out the logical flow of creating the heuristics within Processing. The relationship level was also implemented in the final prototype. I see my role in the team as the team leader, since I divided the tasks, presented the work, made and sustained contact with BlueJay and took the initiative to finalise the report you are reading now. I made sure we had a good process so we were sure to have a working prototype at the presentation.

The most valuable learning point for me is that implementing real intelligence in a product, isn’t as hard as it seems. Furthermore I do now understand how and when to use intelligence over heuristics. I hope to be able to use it within my FMP for the creating of community software. Intelligence could help to detect patterns of relationships with communities and based on that perform suggestions. I am satisfied and surprised by the current prototype, since we were able to link all the codes and programs together and create an interactive prototype. Already with a small sample, we were able to match around the same accuracy as a professional company creating emotion recognition.