Goal: 

As the working title suggests my goal is to enhance live performances of musicians and bands through connecting sound, light and other visuals like animation or maybe even generative processes. This could also be described as audio reactive visuals.

Possibilities:

Through integrating different software and hardware into a live performance, there’s the possibility to create a more immersive experience for the audience, as well as the musician(s) themselves. Best case scenario, visuals that react directly to instruments and voices (audio reactive visuals) give a unique character to the performed music and sound that not even a pre-programmed live set or an experienced light tech can provide. Also it can be used in a wide variety of genres. Styles that have a very expressive character and lots of dynamic range can benefit from it, but I can also imagine singer/songwriters using it. Sitting there alone on stage, with their guitar and microphone and just one moody little light reacting to the words they sing for instance. 

Difficulties:

Currently there are a lot of plans and ideas but just as much work ahead. I have barely any practical experience with the technical aspects necessary to realize a project like this. My first steps are to get familiar with programming languages like Pure Data and Max/MSP. Then I need to find a way to make an analogue input interact with these softwares and also routing an output signal to an interactive light for instance. This all has to happen in real-time with the biggest difficulty being the synchronization of light and sound. Latency might be a problem.

Research question: Is it possible to measure underwater sound by using spacious audio microphone techniques? Is it possible to reproduce an underwater spacial audio?

1. Introduction
   
2. Definitions
   
3. Acoustics and psychoacoustics of spatial audio
   – How human ear localise a spatial sound?
   –
4. Acoustics and psychoacoustics of underwater sound
   -Propagation of signal
   –
5. Building a hydrophone microphone
   
6. Building a hydrophone Microphone Array
   
7. Application of underwater recordings
   
8. Resources:
   
   Websites:
   – Capturing 3D audio underwater
   – Microphone Array
   – On 3D audio recording
   
   Literature:
   • Jens Blauer – Spatial hearing [The MIT Press, Harvard MA]
   • Michael Williams – Multichannel microphone arrays
   • Justin Paterson & Hyuonkook Lee – 3D Audio [Routledge]
   • Abraham A. Douglas – Underwater Signal Processing [Springer]
   • Franz Zotter, Matthias Frank – Ambisonics [Springer]

About spatial audio (development of the concept)

A phantom “image” in stereo
Human ear perceives the sound coming from the front while sending the same signal to both speakers. Changing a gain between the two speakers gives an impression on movement of sound. The phantom changes a position in between two speakers. To not break a central phantom image speakers are usually placed with an angle of 60-degrees. Increasing the angle beyond 60 – degrees might cause a hole in between the speakers, so we lose the perception of central sound and start perceiving sound from independent sources (left and right speaker).

Center speaker
The movie industry decided to add a central speaker in order to compensate the perception of the frontal sound image for the listeners that are not seated at the sweet spot. For those a central speaker works as an anchor and helps to have a more defined and stable sound image.

Surround speakers
In order to improve the movie experiences (by having more realistic set up for a listener), the additional speakers on the sides were introduced. One of the first formats was the LCRS (left, center, right, surround): 5 speakers, 4 channels (the two of rear speakers combined in a single audio channel). In movie theatres surround audio channels are reproduced by an array of speakers. All speakers on a left wall and a right wall reproduce either left or right surround channel. It’s made that way to prevent a phantom hole (the point when you don’t hear any sound coming out) as well as to give to every listener nearly this same surround experience. There is a loose of resolution, the surround sound will become more diffused. In order to reproduce low frequencies the additional speaker was added with its own audio signal. The reason for a separate channel is that humans are not able to localise the direction of low frequencies, so that means that single speaker would not affect a space of the sound image and additionally (since we deal with low frequencies) this channel should be reproduced on higher SPL level than the other channels. Also from practical point of view all movie theatres would have to buy just one speaker instead of replacing the entire sound system with larger speakers. That’s how 5.1 was born: 5 main channels + 1 for low frequencies.

Other Horizontal Layouts (6.1/7.1)
There were other layouts creates, by adding additional channels.
Sony SDDS used 7.1 setup in 90’s (with 5 front speakers and 2 surround channels).
Dolby created a Dolby Digital EX – a 6.1 format with a rear surround speakers.
The traditional 7.1 was created with 3 frontal channels and 4 surround channels, increasing the space resolution on the sides/rear.

Immersive Audio
In order to experience not only the horizontal plane (front, sides, rear), immersive systems were developed. That allows listeners to perceive the sound from all directions, by adding the height component (by adding speakers on the ceiling or on the floor).

Auro 11.1/ Auro 13.1 a format that includes horizontal plane with 5.1 or 7.1 plus a height layer with a 5.0 setup plus a speaker above the listener.

IMAX 12.0 a format that considers 7 channels at the horizontal plane and 5 height channels, no LFE channel.

NHK 22.2: consists of 2 LFE channels, a lower layer with 3 speakers (below horizontal plane, with sound coming from the ground), a horizontal layer with 10 channels (5 frontal, 5 surrounds), a height layer with 8 channels and 1 “voice of god”.

2. Definitions

Binaural cues = the most prominent auditory cues, that are used for determining the direction of a sound source. 

Channel-based audio = audio formats with a predefined number of channels and predefined speaker positions.

ITD (interaural time difference) = measure the time difference of a sound arriving in the two ears

IID/ILD (interaural intensity difference) = tracks the level difference between the two ears

Phantom image =

Shadowing effect

Vector Base Amplitude Panning (VBAP) = recreation of perception of sound coming from anywhere within triangle of speakers

4. Acoustics and psychoacoustics of underwater sound

_{Localization underwater – what makes it so difficult?}

• From ITD cue perspective: the speed of sound: in air = 343 m/s, underwater = 1480 m/s (depends mainly on temperature, salinity)
  
  “A speed of sound that is more than 4x higher than the speed of sound in air will result in sound arriving in our ear with much smaller time differences, which could possibly diminish or even eliminate completely our spatial hearing abilities. “

• From IID perspective: there is a very little mechanical impedance mismatch between water and out head
  
  “Underwater sound most likely travels directly through our head since the impedances of water and our head are very similar. and the head is acoustically transparent. “
  
  “In air, our head acts as an acoustic barrier that attenuates high frequencies due to shadowing effect.”
  
  “It is not entirely clear whether the underwater localization ability is something that humans can learn and adapt or if it’s a limitation of the human auditory system itself.”
  

Sehen, Hören, Riechen, Schmecken, Tasten: die fünf Sinne des Menschen sind allseits bekannt. Doch unsere Wahrnehmung funktioniert nicht immer so geordnet und klar trennbar, wie diese Einteilung vermuten ließe.

Bereits 2004 erschien The Handbook of Mulitsensory Processes (Gemma A. Calvert, Charles Spence, Barry E. Stein). Die Publikation beschreibt Wahrnehmung als multisensory experience und bricht mit dem traditioniellen sense-by-sense-approach.

Zu Beginn der 2010er-Jahre machte der Neurowissenschaftler Daniel W. Wesson durch Zufall eine spannende Entdeckung. Eigentlich untersuchte er bloß die Geruchswahrnehmung von Mäusen, doch als er eines Nachmittags seine Kaffeetassse auf den Labortisch stellte, bemerkte er einen punktuellen Anstieg der Aktivität im Tuberculum olfactorium der Mäuse – das ist der Bereich des Gehirns, in dem u. a. Gerüche verarbeitet werden.

Gemeinsam mit seinem Kollegen Donald A. Wilson ging er nun der These nach, dass es “Smounds” gebe – eine Sinneswahrnehmung, die sich aus Smells und Sounds zusammensetze. Tatsächlich konnten die Forscher anhand einer Studie an Mäusen nachweisen, dass im Tuberculum olfactorium Smells and Sounds überschneidend verarbeitet werden, ganz im Sinne der multisensory experience.

Doch die Beschäftigung mit dem Zusammenspiel mehrerer Sinneseindrücke hat eine lange Geschichte. So entwickelte schon der russische Komponist Alexander Skrjabin (1872-1915) die Idee für ein Gesamtkunstwerk, das “Mysterium”, das, einmalig und mehrtägig in einem eigens dafür errichteten Tempel aufgeführt, eine allumfassende Sinneswahrnehmung ermöglichen sollte. Ein Ritual mit esoterischen Einflüssen, dessen Umsetzung nicht nur an Skrjabins Tod scheiterte. Skrjabin galt als Synästhetiker, er konnte laut eigener Aussage Klänge als Farben sehen. Sein Gesamtwerk sollte jedoch nicht auf das “Mysterium reduziert werden.

“Smounds” sind auch heute noch Gegenstand künstlerischen Schaffens, rücken aber aktuell auch in der Stadtplanung in den Fokus. Doch dazu in den nächsten Wochen mehr.

Quellen/Links:

Calvert, Spence, Stein(2004): The Handbook of Mulitsensory Processes

https://www.scientificamerican.com/article/making-scents-of-sounds-n/

https://www.jneurosci.org/content/30/8/3013

https://www.semilakovs.com/cd/booklet-text/

https://de.wikipedia.org/wiki/Alexander_Nikolajewitsch_Skrjabin#Werke

https://de.wikipedia.org/wiki/Syn%C3%A4sthesie

“ You can’t totally control noinput music because it’s all about feedback. Things like turning the tuning knob, even by one millimeter, make a big difference to the sound. …It’s very hard to control it. The slightest thing can change the sound. It’s unpredictable and uncontrollable. Which makes it challenging. But, in a sense, it’s because of the challenges that I play it. I’m not interested in playing music that has no risk. “ (Toshimaru Nakamura)

Reading a paper on feedback systems, from Dario Sanfilippo and Andrea Valle: “FeedbackSystems: An Analytical Framework”, I found interesting a given key to understanding feedback itself and the relation between the different parameters. Specifically I focused on a paragraph that emphasizes the non-linearity of a feedback system and how different sonic parameters, normally unrelated in an audio system, are actually related to each other.

As they mention in the paper, parameters like amplitude and pitch, are coupled in a condition where they mutually affect each other in an interaction system. Their combination is leading to new entities that are not the summation of their parts, instead a new element that coexist only because of their synergy. Each element has its own identity, no one has the role of coordinator, but all play as equals to achieve a goal that takes shape only through their essential cooperation. This is also resembled in the loose relations of non- or flat hierarchical organisations, where each participant´s ideas are equal to and influence each other. The emerging results are unpredictable to a certain degree.

Inside this process, the smallest changes of one parameter or in the setup influences the overall result unproportionally. Therefore it´s difficult to reproduce the results although it´s possible to train playing it as an instrument, to attain a certain amount of control. By testing the influence of different parameters, you can understand roughly what a single parameter/event changes in the whole system.

It is important to underline that every performance is unique and you can´t know before if it´s going to be a success or a failure. Each setup is unique in its possibilities, even using the same gear never sounds identical. Still a sort of connection between the sonic phenomena and the interaction with the performer within the system is achieved, which´s framework is outlined by the composer.

In contrast to playing a conventional instrument, the performer and system meet equally as agents. The performer interacts within a complex net, changing the dramaturgy of the sound itself. Sound and form are not to be differentiated, the concept of coherence depends on the level of understanding the immanent evolution of events and their relation to each other.

Feedback offers great variety of different possible results from seemingly chaotic to minimalistic. Thanks to its flexibility, it is usable to explore these extremes in a continuum. The risk of loosing control becomes part itself of the compositional process, if on one side the indeterministic behaviour puts you on a subordinary position, on the other side feedback systems tend to stabilize themselves giving you the possibility to explore the potential of this unstable “holy” dimensions.