research project

“Sounding out” and “Seams”, are two separate installations that share their content using different perspectives. On the one hand “Sounding out” has as its core the “energy band analyser”; it is a live interactive installation in which the audio signal from the participant is captured by a microphone and analysed in real time. On the other hand, “Seams” is not interactive in the sense that no live information is produced or detected (except for the movement of the head of the participant wearing the goggles) but it is an immersive art experience that shows the contents of my research; this installation is a way for people to visualise the patterns of emotions in sound, a sort of phenomenological approach to my research content and findings. This second installation is meant to address the topic from a different perspective in order to give people a figurative as well as practical overview of this study.



sounding out

In this artwork, I created an inclusive framework for real-time recognition of emotions from acoustic properties of speech not using content information. I then linked these properties to a visual output to arrange an interface of the emotions.

WRITTEN MATERIALS

ABSTRACT & INTRODUCTION ( The first 3 pages of the thesis)

THESIS FULL TEXT

sounding out PICTURES


FLORENCE JUNE 2018

sounding out VIDEO MATERIALS

FLORENCE JUNE 2018
RECORDINGS OF LIVE PARTICIPATION 

 

 seams

“Seams” is an immersive audio-video representation of the processes of analysis and detection of emotions used in “sounding out” and in the research carried out in my studio using the “energy band analyser”.

 

seams PICTURES

THE BEGINNING - PRIVATE TECHNICAL TESTS 

FIRST ATTEMPT - PREMIER - LIBRARY CONSERVATORY OF CESENA APRIL 2018


seams VIDEO MATERIALS

2D VERSION OF WHAT PARTICIPANTS SEE IN THE HEAD MOUNTED DISPLAY WITH FIXED STEREO VERSION OF SOUNDTRACK - 
(NO SONIFICATION AND NO SPATIALIZATION OF SOUND WHICH ARE AVAILABLE ONLY IN LIVE EXHIBITION)
 
 


The videos below are those shown inside the televisions with the different sonification of the audio.

During the 360 degree immersion (using 3D goggles) the participant, turning the head on the left, hears sad voices and sonification of sadness, facing the centre, s/he hears happy voices and sonification of happiness and turning the head to her/his right, s/he hears angry voices and sonification of anger.

The interaction is only possible during the exhibition through the use of a max/MSP patch designed for the intent.

Below you can press play and stop on each video alternately to try to create the feeling of when you are turning your head in a direction (left/center/right) to understand the change in the sound environment.

left - sadness

centre - happiness

right - anger


seams Audio soundtrack

Below you can listen to each track alternately (the audio is the same as the above 3 videos) to understand the differences between the audio sonification of the 3 emotions and to focus only on sound.

Sonification of Happiness


Sonification of Anger


Sonification of Sadness