Emerging Ideas

Leon Radschinski-Gorman: Sounds are Objects – Understanding Creativity with Sound through Embodied Cognition

Leon Radschinski-Gorman graduated from the MA Art and Science course, University of Arts London where he researched how relationships between sound and image can be reconciled from a cognitive science and neurological point of view. This essay presents a cross-modal and embodied account of creativity with sound in order to understand better the processes involved in being creative.

Abstract:

This essay presents a cross-modal and embodied account of creativity with sound in order to understand better the processes involved in being creative. First, examples of language will be presented to demonstrate the essential metaphoric vocabulary for sound experience; interpreting George Lakoff’s and Mark Johnson’s work on conceptual metaphor, this language will be used to evidence fundamental conceptualisations of sound with reference to creative choices in audio-visual work. The embodied groundings for these metaphors will be presented to explain why, for instance, sound pressure wave frequency is able to be mapped to verticality, giving us the metaphors ‘high’ and ‘low’ for pitch. The argument that creativity is dependent on embodied experience will be examined by tying together conceptual metaphor theory with research in the neurological basis of synaesthesia, in particular Vilayanur Ramachandran and Edward Hubbard’s theory that synaesthesia is an exhibition of ‘cross-wiring’ in the brain that in fact we all possess, just not in specific regions dealing with perception, and is essentially a demonstration of metaphoric thought from a neurological point of view. Examples of work that can empirically show utilisation of cross-modal metaphor (uses of ‘tone painting’ and work entailing the visual representation of sound) will be compared with what the work may have been presented as if it were produced under a different conceptual metaphor system, i.e. by an artist from a vastly differing culture that structures common sound experiences through different conceptual metaphors; e.g. pitch as thickness in Turkish as opposed to pitch as height in English. The ability for us to make and understand these connections across modalities creatively, independent to language, supports Ramachandran’s and Hubbard’s claims that synaesthesia is an inherent part of normal cognition. In conclusion this paper suggests that sound creativity is inseparable from embodied experience, and that creative output of artists working with sound is fundamentally structured by cognitive metaphor.

Contents:

1

Sound’s Metaphorical Vocabulary

Introduction to language we use about sound

2

Experiential Groundings for Metaphors and Manifestation in Audio-Visual Work

Sound creativity structured by embodied experience

3

Synaesthesia and the Neural Theory of Metaphor

The neurologically cross-modal foundation for metaphor

4

Further Cross-Modal Influences

Top-down effects on sound experience

5

Conclusions

……………

Introduction

What is Sound Creativity?

In this essay, creativity with sound refers to practices where sound is a primary method of expression. This includes in practice but is not limited to music writing (lyrics and instrumentation), music production, sound art, visual representation of sound, sonic representation of image, sound design, sound recording, foley (post sync sound effects in film), sound engineering, and also the act of talking about sound. In essence, it refers to the cognitive processes and decisions involved when engaging in these practices.

The methodology used to investigate creativity with sound is this:

  1. Present sound’s vocabulary to introduce metaphorical conceptualisation of various types of sounds.
  2. Present experiential basis of those metaphors.
  3. Present research detailing metaphor as a synaesthetic phenomenon.
  4. Present examples of work throughout to tie research areas together to conclude experiential basis for creativity. 

Notes on format:

  • The essay follows Lakoff and Johnson’s conventions to illustrate conceptual metaphor mappings, capitalised so: TARGET IS SOURCE. This is helpful as it shows clearly the structuring domain: SOURCE, which structures the more abstract domain: TARGET (Lakoff and Johnson, 1980, p. 20).
  • Occurrences of metaphors relating to sounds will be written within single apostrophes to highlight the use of the metaphor (e.g. ‘high’ sound). As they are so ingrained we often do not realise they are metaphors at all.

1

Sound’s Metaphorical Vocabulary

Communication is based on the same conceptual system that we use in thinking and acting, language is an important source of evidence for what that system is like” (Lakoff and Johnson, 1980, p. 3).

In this first section, the prevalence of metaphor use in sound’s vocabulary will be highlighted by presenting examples of descriptive terms often used in English. The examples introduce an experiential conceptualisation of sound to be elaborated throughout the essay. Use of metaphor in other languages and implications to creativity will be mentioned later.

It should be noted that not all descriptions of sounds are metaphorical, onomatopoeia is often used (fizzy, sizzling, banging) as are comparative adjectives (breathy, tinny, wooden), otherwise, much of the language reveals a significant use metaphor to conceptualise sound.

The following examples were documented and collected through materials and media dealing with sound: books about sound recording, reviews of music, descriptions of music and sound, and personal communication (Chion, 1994; Van Campen, 2012; Viers, 2008; Milner, 2010; Viers, 2012; Huber, 2013; Describing Sound – A Glossary, 2013):

 

Some of these metaphors will proceed to be referenced throughout the essay. 

Nomenclature

As well as common terms such as ‘high’, ‘harsh’, ‘soft’ other metaphors are sometimes used in specific contexts. For example, ‘wet’ or ‘dry’ might be quite subjective unless the sound in question is particularly comparable to water, rainfall, etc. Or, if used in music/audio production, ‘wet’ and ‘dry’ refers to the degree with which an audio signal is processed through some kind of effect, e.g. distortion or reverb; a ‘dry’ signal being unprocessed (Huber, 2010, p. 475). This means individuals with a background in audio will more readily relate to this metaphor and a better understanding of that metaphor would exist in the small section of a population. The ‘wet’/’dry’ metaphors and what they mean may not be immediately obvious to those without this prior knowledge as they only exist as nomenclature in a particular field, i.e. no experiential or particularly obvious relationship to an existing sound metaphor. It is worth noting the existence of these specialised metaphors as they show sound conceptualisation can be augmented through creative use of language.

2

The Intrinsic Experiential Grounding for Sound Metaphors and Manifestation in Audio-Visual Work

The vocabulary of sound suggests an understanding of sounds as objects: they exhibit textural qualities (the sound can be ‘smooth’), they exhibit visual qualities (the sound can be ‘bright’), they have physical force (the sound can be ‘piercing’), they exist within a space relative to the listener, (‘foreground’, ‘background’, ‘up’, ‘down’, ‘high’ and ‘low’) and they are understood temporally and so can change position over time (‘rising’ and ‘falling’). When writing a song, they are organised within a space, the sounds are arranged in terms of a physical arrangement which is projected as – desirably – an aesthetically pleasing result sonically. The space can have boundaries (the sounds are ‘in’ a song) and this bounded region in which the sounds exist can be elaborated metaphorically and spoken of as a structure/building (songs can be built, put together, have solid foundations, can be deconstructed, etc).

In addition to language, experiential conceptualisation of sound is manifested in creative  works dealing with the visual representation of sound. As will be discussed, ‘high’ sounds are often represented with high imagery, this demonstrates not only a linguistic metaphor, but a primary conceptual metaphor in English speaking cultures: PITCH IS HEIGHT. Primary metaphors often form the basis from which other metaphors are built and come from experiential correlations, Lakoff describes this in his paper The Neural Theory of Metaphor:

During learning, spreading activation strengthens synapses along the way. When the activation spreading from A meets the activation spreading from B, a circuit may be formed. The more A and B fire together the stronger the synapsis forming links in the circuit. As we shall see, this is the basic mechanism by which primary metaphors are formed (2008, p. 5)

Primary metaphors can be elaborated dynamically, for example we also conceptualise pitch as brightness, as expressed in the common metaphor ‘bright’ for ‘high’ pitch. PITCH IS BRIGHTNESS has no obvious experiential basis but can be explained due to the fact height and brightness are experientially coherent; coherence is important to the development, elaboration, propagation, and understanding of new conceptual metaphors (Lakoff and Johnson, 1980, pp. 87-105). This coherence can be further clarified through syllogism: height (concrete experience) = pitch (abstract experience), brightness (concrete experience) = height (concrete experience), so brightness = pitch.

Looking at other languages reveals the importance of conceptual metaphor systems to creative decisions. In Turkish, Farsi and Zapotec languages the dominating metaphor for pitch is in terms of ‘thin‘ (high) and ‘thick‘ (low) (Shayan, Ozturk and Sicoli, 2011, p. 98), in the Manza language it is ‘small’ and ‘large’ (Dolscheid, et al., 2013, p. 614). This is important because as other languages demonstrate unique conceptualisations of sound, it follows that creative choices and understandings of creative work will differ: different conceptual systems lead to different ways of being creative. To clarify, a sound ‘rising’ in pitch could likely be represented visually with an upwards movement by an English speaking artist because, through their conceptual system, that correlation is coherent. A Turkish artist on the other hand might represent that sound by illustrating an image becoming thinner. This could potentially mean that parts of a work like Disney’s Fantasia may actually appear audio-visually incoherent when viewed by certain other cultures (although audio-visual concurrence producing coherence will be discussed later). Song writing and composition is in turn structured by these frameworks; lyrics will often reflect the conceptual metaphor PITCH IS HEIGHT; for example in Radiohead’s High and Dry the word ‘high’ is sung ‘high’ (Yorke, 1995), this is known as tone painting (or text/word painting) and has been used for centuries to cross-modally blend linguistic meaning into musical form; Giovanni Pierluigi da Palestrina’s 1567 Pope Marcellus Mass uses this technique to map a passage about Christ’s descent (descendit) from heaven to the musical descent of pitch (Palestrina, 1964; Zbikowski, 2009 p.360).

Cross-modal metaphor structures creative choices; had Pope Marcellus Mass been written by somebody without conceptualisation of pitch as height, it can be reasoned that the musical product would be fundamentally different.

Following are examples of conceptual metaphors demonstrated through both language and representations of sound evidencing their influence on audio-visual creativity, along with their suggested embodied groundings.

AMPLITUDE IS HEIGHT

Examples: ‘high volume’ ‘low volume’

  1. MORE IS UP is a primary metaphor, coherence between more and up is deeply integrated into our conceptual system (Grady, 1997, p. 285), so: more volume = higher volume. MORE IS UP is likely grounded in the experience that adding more to something increases its height (Lakoff and Johnson, 1980, p. 16; Grady 1997, p. 285).
  2. Closer proximity of listener to sound source increases amplitude and relative height and vice versa.

In work Unraveling Boléro by Anne Adams:

Unraveling Boléro is a painting and visual representation of Boléro by Maurice Ravel.

As Boléro becomes progressively louder, the illustrations reflect this by becoming taller.

AMPLITUDE IS DISTANCE

Examples: ‘background’ ‘foreground’ ‘distant’ ‘close’ ‘near’ ‘far’

  1. Closer proximity of listener to sound source increases amplitude and decreases distance and vice versa.

In Michel Gondry’s music video for the Chemical Brother’s Star Guitar:

The video features a fixed camera perspective looking out of a moving train window, all the elements of the song are represented in sync by various features in the landscape.

The ‘background’ synth sounds are represented as factory chimneys, for example.

Louder sounds such as the bass drum are represented by block-like buildings in the foreground.

AMPLITUDE IS QUANTITY

Examples: ‘more volume’ ‘less volume’

  1. An increasing number of sound sources usually accompanies an experienced louder sound level.
  2. Coherence with AMPLITUDE IS HEIGHT, recall: MORE IS UP. 

AMPLITUDE IS SIZE

Examples: ‘large’ ‘small’ ‘big’ ‘fat’ ‘massive’ ‘tiny’

  1. Closer proximity of listener to sound source increases amplitude and increases physical size and vice versa.
  2. Coherence with AMPLITUDE IS HEIGHT/QUANTITY; objects seen to increase in height also increase in size. 

PITCH IS HEIGHT

Examples: ‘high pitch’ ‘low pitch’ ‘rising pitch’ ‘falling pitch’

  1. The movement of the larynx upwards for higher pitches, and downwards for lower pitches.
  2. The feeling of lower frequencies resonating lower in the body.
  3. The production of lower tones using a ‘chest voice’ (physically lower), in contrast with the higher tones using the ‘head voice’ (higher).
  4. The feeling of very low bass sounds vibrating the floor.

In Disney’s Fantasia:

Swooping fairies reflect the swooping (‘downwards’) notes of the bass clarinet.

In Michel Gondry’s Around the World music video for Daft Punk:

Blocky characters in tracksuits represent the bass line, with each ‘descending’ note the characters descend a set of stairs.

PITCH IS BRIGHTNESS

Examples: ‘bright’ ‘dark’ ‘dull’

  1. As detailed previously, coherence between height and brightness generate this metaphor.

In Disney’s Fantasia:

The sparkling high notes of the celesta are accompanied by bright fairy dust.

In Michel Gondry’s Star Guitar music video:

A ‘high’ synth sound that ‘lowers’ in frequency is represented by daytime becoming night.

This case is interesting as the visual representation of the ‘low’ sound appears fitting even though in English ‘dark’ is not often used for ‘low’ pitch, although ‘bright’ is often used for the converse. This emphasises that these visual representations are not just reflections of language.

SOUNDS ARE TANGIBLE OBJECTS

Examples: ‘harsh’ ‘smooth’ ‘rough’

  1. Harsh texture produces a respective sound when handled etc.
  2. Sounds are produced by physical things naturally, often the terms used to describe sounds are either onomatopoeic or comparative. For example, white noise might evoke rainfall, producing descriptions like ‘wet’, ‘heavy’, or even ‘cold’. And comparative adjectives can be used too: sound can be wooden, glassy, metallic etc. These examples show that non-metaphorical descriptions of sounds can be used to augment and decorate the metaphors.
  3. This metaphor is of course coherent with every other metaphor, as physical objects can demonstrate all the qualities previously mentioned.

A key point is that when artists are making these representations, they make choices because they seem right, it makes sense that ‘upwards’ sound goes with upwards movement, it is integrated into the very language we use, this makes the process of representing a sound with an image seemingly obvious, but highlighting that a large object is chosen for a ‘large’ sound raises an important inquiry: how is it possible, and what in the brain is responsible,  for the mind to be able to abstract ‘upwardness’ from the perception of sound pressure waves vibrating faster than converse ‘low’, slower, vibrations? How is this correlation solidified individually and how does it become deeply integrated across cultures?

So far, an experiential basis for creativity with sound has been investigated mainly through examples of cross-modal representation and language, but in order to advance the case that sound creativity, like language, is inherently embodied, a study of the physical neurology that allows for this kind of cross-modal cognition will be presented. The next part of this essay investigates this neural cross-modality and its role in audio-visual creativity.

3

Reconciling Synaesthesia, the Neural Theory of Metaphor, and Embodied Cognition

Synaesthesia is a perceptual phenomenon in which stimulation of one sense automatically cross-activates experience of another sense (Van Campen, 2008, p. 1). For example, for a synaesthete, hearing the note C could produce the perceptual experience of the colour red, F could produce blue, and so on. Importantly, the correlational experiences that synaesthetes perceive are consistent to that individual. Other common forms include numbers being coloured, sounds producing tastes, days of the week being coloured, etc.

Two forms have been distinguished by Vilayanur Ramachandran, a leading scientist in neurology and psychophysics: ‘higher’ synaesthesia and ‘lower’ synaesthesia (Ramachandran and Hubbard, 2001, p. 14). A ‘lower’ synaesthete that has grapheme>colour synaesthesia might experience effects from seeing the grapheme ‘4’, but not the Roman numeral ‘iv’; the ‘higher’ synaesthete would experience the effects for both, showing that in some people the experience is driven by the concept and not just the visual form (ibid., p. 14). The various manifestations of synaesthesia has been attributed to ‘cross-wiring’ in different areas of the brain (Ramachandran and Hubbard, 2003, p. 54), and ‘cross-wiring’ as a function of synaesthesia has been experimentally supported by DTI scans showing greater white matter diffusion in grapheme>colour synaesthetes (Romke and Scholte, 2007).

Research shows that as infants, the human brain exhibits significant cross-modality resulting in a ‘sensory soup’ (Van Campen, 2008, p. 29). A few months into development, ‘pruning’ occurs, leaving separate areas of the brain more specialised in their respective roles (Ramachandran and Hubbard, 2001, p. 3; Van Campen, 2008, p 30). A popular hypothesis is that defective pruning caused by a gene or genes results in continued greater cross-modality either selectively or more generally throughout the brain (Purple Numbers and Sharp Cheese, 2003). Ramachandran and Hubbard assert that the genetic aetiology of synaesthesia can be exploited to understand the existence of metaphoric thought in humans, in turn providing an insight into creativity (Ramachandran and Hubbard 2001 pp. 4-5). Since synaesthesia is experimentally more common in creative individuals (Dailey, Martindale and Borkum, 1997; Domino, 1989; Root-Bernstein and Root-Bernstein 1999, cited in Ramachandran and Hubbard, 2001, p. 17), whom are notable for prolific use of metaphor, they hypothesise that genes responsible for defective pruning of adjacent brain areas that result in synaesthesia, when expressed arbitrarily throughout the whole brain and not just in regions dealing with the senses, can account for greater ability to link together seemingly unrelated ideas. Ramachandran and Hubbard state:

If mutation-induced cross-wiring selectively affects the fusiform or angular gyrus someone may experience synaesthesia. However, if this mutation is more  diffusely expressed it may produce a more generally cross-wired brain creating  a greater propensity and opportunity for creatively mapping from one concept to another (2001, p. 17).

This neural connectivity explanation can be compared to the neural theory of metaphor:

Pre-existing Neural Connections and Experiential, Developed, Neural Connections

Despite historical attempts to demonstrate connection between pitch and colour, for example experiments, tests and inventions by Plato, Aristotle, Newton and many others over the centuries (Van Campen, 2009, p. 45-53; ‘Color organ’, 2017), the mappings will not be consistent and will not resonate amongst audiences like pitch to height representations do. The reason is this: though the mapping of pitch to brightness might appear no more abstract, in experience there is no frequent enough correlation naturally between pitch and colour, so neural connections resulting in a consistent conceptual metaphor to develop across a culture will not form. Lakoff writes:

In situations where the source and target domains are both active simultaneously, the two areas of the brain for the source and target domains will both be active. Via the Hebbian principle that Neurons that fire together wire together, neural mapping circuits linking the two domains will be learned. Those circuits constitute the metaphor (2008, p. 16).

Since correlation does not cause cross-modal mapping between sound and colour due to lack of natural experiences, the only way it can be consistently exhibited is through a formed neural link in the brain due to synaesthesia, either genetically or epigenetically (Van Campen, 2009, p. 36; Ramachandran and Hubbard, 2001, p. 6).

The line between synaesthesia and universal cross-modal cognition is blurred when considering the following case study and its relevance to language and creativity:

The Baluba-Takete/Bouba-Kiki experiments

These experiments, originally conducted by Wolfgang Köhler, demonstrate intrinsic cross-modal abstraction of sound from image. The tests present two images, one bulbous and round, the other sharp and jagged. Participants were asked which is ‘Baluba’ and which is ‘Takete’, 95% to 98% assigned the former to the rounded shape and the latter to the jagged one (Köhler 1929, cited in Ramachandran and Hubbard, 2001, p. 19). The experiment was repeated by Ramachandran and Hubbard with the terms ‘Bouba’ and ‘Kiki’ resulting in identical results (2001, p. 19). Ramachandran explains the phenomenon during the Science Network’s Beyond Belief conference:

That sharp inflection ‘ki‐ki’ in your auditory cortex, the hair cells being excited, … mimics the visual inflections, sudden inflection of that jagged shape. … These photons in your eye are doing this shape, and hair cells in your ear are exciting the auditory pattern, but the brain is able to extract the common denominator … and we now know this happens in the fusiform gyrus of the brain because when that’s damaged … people lose the ability to engage in “bouba‐kiki”, but they also lose the ability to engage in metaphor. … Something very interesting is going on here in the angular gyrus, because it’s the crossroads between hearing, vision and touch, … and I think it’s a basis of many uniquely human abilities like abstraction, metaphor and creativity (LennyBound, 2008).

Bouba-Kiki provides an insight into language development, it suggests naming of objects is not arbitrary, rather sounds we choose when naming can be abstracted from the physical form of the object in question. It has been demonstrated that even before language acquisition, children are able to coherently engage in audio-visual cross-modal abstractions (Dolscheid et al., 2014, p. 1259). This independence from language is evidenced in certain representations of sound in examples mentioned earlier as well as following examples.

The ingrained cross-modal cognition revealed by Bouba-Kiki is emphasised when studying sound to image representations. In Disney’s Fantasia during The Nutcracker Suite, exceptional coherence with the Bouba-Kiki experiments is demonstrated: ‘smooth’ legato strings are visualised as rounded dancing plants (Bouba). Followed by angular sharp plants for staccato parts (Kiki).

In Norman McLaren’s Dots, he paints on film and also paints on to the sound track of the film to produce the sounds. The sounds are synced with the various abstract shapes seen on screen. For example a bass sound is represented by a large blob which looks like Bouba, and a converse, ‘sharp’, sound is represented by a sharp, jagged shape.

These kinds of representations are possible and make sense for the same reason the Bouba-Kiki experiments produce such coherent results. Our brains are automatically capable of abstracting sound from image as well as image from sound in a truly synaesthetic way. Though the conceptualisation of sound is based on physical experience, the perception of sound and image is rather more bidirectional, the following part of this essay evidences this through presenting synchresis and the McGurk effect as exhibitions of multisensory integration.

4

Further Cross-Modal Considerations

Synchresis

An important note about audio-visual experience is that perception is a dynamic experience. This essay has presented a subservience of sound experience to more concrete physical and visual experience in order to present the thesis that sound creativity is structured by embodied experience. But there is also a perceptual top-down structuring whereby sound perception will modulate the perceived properties of an image (Chion, 1994, p. 63). This is often studied in film theory and the term ‘synchresis’ comes from the prominent film theorist Michel Chion. A large image played in sync with a ‘large’ sound will highlight the large aspects of the image, which in turn highlight the ‘large‘ aspect of the sound ad infinitum, a phenomenon film editor and sound designer Walter Murch calls conceptual resonance in his foreword to Chion’s Audio-Vision: Sound on Screen (1994 p. xxii). To illustrate this, in Michel Gondry’s Star Guitar, a block like structure has been used to represent a bass drum, the image reflects a kind of blockyness identifiable in the sound of the drum. Additionally though, the action of seeing and hearing the blockyness in sync emphasises and resonates that blocky characteristic for both visual and auditory experiences.

Exploiting synchresis could be achieved by juxtaposing the sound with the image to cause a kind cognitive dissonance that could be utilised for effect, or drawing attention to certain visual or sonic characteristics through considered use of the corresponding visual or sonic forms in sync.

The McGurk Effect

The McGurk effect demonstrates synchresis experimentally: a phoneme (e.g. ‘ba’) played in sync with the image of somebody saying (‘ga’) produces a new perceived sound in the observer (‘da’) (McGurk and Macdonald, 1976, p. 746). The illusion is unique in that even by being aware of the effect, it does not make it any less likely to be perceived. The effect shows modulation of sound perception through visual stimuli, which then theoretically also can be produced via synchresis between sound and image conceptually. For example, a ‘falling’ sound with a rising image, the effect this has is a kind of dragging of visual and sonic movement, the sound pulling on the image through synchresis to produce a percept of the image not falling quite so convincingly. This can be noticed in Fantasia, in one scene, as shimmering lights descend, the music playing is not noticeably rising or falling in pitch, but due to synchresis, an impression of descending pitch can be noticed in the soundtrack.

6

Conclusions

This essay presents a number of conclusions:

  1. Creativity in sound is inherently cross-modal and reliant on conceptual metaphor. Neural connections between concepts across areas in the brain irrespective of sensory region provide a synaesthetic network of metaphors we employ to understand sound, describe sound and work creatively with sound.
  2. A subconscious picture of sound objects, their arrangement respective to one another, their ability to conform to various types of conceptual metaphors, their textures, size, shapes and movement is the hidden resource we use to be creative and develop ideas sonically.
  3. Being aware of these systems can allow conscious circumventions and exploitations of structuring metaphors, and can result in the revelation of new and hidden ideas.
  4. The argument from cognitive linguistics that language is embodied is supported by this essay.
  5. Creativity with sound is independent to language, though language does serve to structure it. As was mentioned earlier, this is shown experimentally with children that exhibit the ability to partake in cross-modal activities before learning language, and it is also shown in certain representations of sound. When examining the ways sound is represented and comparing it to the language used by the artists, it is interesting to note that sometimes the correlations chosen do not necessarily reflect metaphors in the native language. ‘Dark’ can certainly be used to describe ‘low’ sound, but it is not a normal part of sound’s vocabulary. Seeing that correlation on screen though resonates and appears to make perfect sense.

………………..

Bibliography:

‘Around the World’ (2003). In The Work of Director Michel Gondry [DVD], USA: Palm Pictures.

Aldhous, P. (2008) Boléro: ‘Beautiful symptom of a terrible disease’. Available at: https://www.newscientist.com/article/dn13599-bolero-beautiful-symptom-of-a-terrible-disease/ (Accessed 22 June 2016).

Baron-Cohen, S. (1996) ‘Is There a Normal Phase of Synaesthesia in Development?’, Psyche, 2(27). Available at: http://hstrial-tridenttechnical.homestead.com/BaronCohen1996.pdf (Accessed 01 February 2017).

Calvert, G. (2001) ‘Crossmodal Processing in the Human Brain: Insights from Functional Neuroimaging Studies’, Cerebral Cortex, 11(12), pp. 1110-1123. DOI: https://doi.org/10.1093/cercor/11.12.1110

ccrmalite (2014 7 March) Douglas Hofstadter: The Nature of Categories and Concepts. Available at: https://www.youtube.com/watch?v=Kr3QDMkMGmQ (Accessed 10 January 2017).

Central European University (2013 22 October) George Lakoff on Embodied Cognition and Language. Available at: https://www.youtube.com/watch?v=XWYaoAoijdQ (Accessed 10 January 2017).

Chandler, D. (2002) Semiotics, the Basics. 2nd ed. New York: Routledge.

Chion, M. (1994) Audio-Vision: Sound on Screen. New York: Columbia Univeristy Press.

‘Cocktail party effect’ (2016) Wikipedia. Available at: https://en.wikipedia.org/wiki/Cocktail_party_effect (Accessed 22 July 2016).

Coëgnarts, M., Kravanja, P. (2012) ‘From Thought to Modality: A Theoretical Framework for Analysing Structural‐Conceptual Metaphors and Image Metaphors in Film’, Image and Narrative, 13(1), pp. 96‐113.

‘Cognitive linguistics’ (2017) Wikipedia. Available at: http://en.wikipedia.org/wiki/Cognitive_linguistics (Accessed 20 July 2016).

Collier, W., Hubbard, T. (2004) ’Musical scales and brightness evaluations: Effects of pitch, direction, and scale mode’, Musicae Scientiae, 8(2), pp. 151‐173.

‘Color organ’ (2017) Wikipedia. Available at https://en.wikipedia.org/wiki/Color_organ (Accessed 20 Jan 2017).

‘Conceptual metaphor’ (2017) Wikipedia. Available at: http://en.wikipedia.org/wiki/Conceptual_metaphor (Accessed 20 July 2016).

‘Conduit metaphor’ (2017) Wikipedia. Available at: https://en.wikipedia.org/wiki/Conduit_metaphor (Accessed 21 July 2016).

‘Cortical homunculus’ (2017) Wikipedia. Available at: https://en.wikipedia.org/wiki/Cortical_homunculus (Accessed 21 July 2016).

Crystal, D. (1995) Phonoaesthetically Speaking: An examination of why people regard some words as inherently more beautiful than others, English Today. 11(2).

Culhane, J. (1983) Walt Disney’s Fantasia. New York: Harry N. Abrams Inc.

CVM (2014 15 March) Komposition in Blau. Available at: https://vimeo.com/89193540 (Accessed: 10 January 2017)

Describing Sound – A Glossary (2013). Available at: http://www.head-fi.org/t/220770/describing-sound-a-glossary (Accessed 23 December 2016).

Dolscheid, S., Hunnius, S., Casasanto, D., Majod, A. (2014) Prelinguistic Infants Are Sensitive to Space-Pitch Associations Found Across Cultures, Association for Psychological Science. 25(6), pp. 1256-1261.

DOI: 10.1177/0956797614528521

Dolscheid, S., Hunnius, S., Casasanto, D., Majod, A. (2015) ‘When High Pitches Sound Low: Children’s Acquisition of Space-Pitch Metaphors’ Cogsci 2015: Mind, Technology, and Society, Cognitive Science Society, California, 23 to 25 July 2015.

Dolscheid, S., Shayan, S., Casasanto, D., Majid, A. (2013) ‘The Thickness of Musical Pitch:

Psychophysical Evidence for Linguistic Relativity’, Association for Psychological Science. 25(6), pp. 613-621.

DOI: 10.1177/0956797612457374

Fantasia (1940) Directed by Armstrong, S. et al. [DVD] USA: Walt Disney Pictures.

Forceville, C. (2008) ‘Metaphor in pictures and multimodal representations’, in: Gibbs, R. (ed.) The Cambridge Handbook of Metaphor and Thought. New York: Cambridge University Press, pp. 462-482.

Gibbs, R., Lima, P., Francozo, E. (2004) ‘Metaphor is Grounded in Embodied Experience’, Journal of Pragmatics. 36(7), pp. 1189‐1210.

Greene, J and Hicks, C. (1984) Open Guides to Psychology: Basic Cognitive Processes. Milton Keynes: Open University Press.

Huber, D (2010) Modern Recording Techniques. 7th ed. Oxford: Taylor & Francis Ltd.

‘Ideasthesia’ (2016) Wikipedia. Available at: https://en.wikipedia.org/wiki/Ideasthesia (Accessed 20 March 2016).

‘Image Schema’ (2017) Wikipedia. Available at: https://en.wikipedia.org/wiki/Image_schema (Accessed 20 July 2016).

Johnson, M., Rohrer, T. (2003) ‘We Are Live Creature: Embodiment, American Pragmatism, and the Cognitive Organism’, Body, Language, and Mind, 1, pp. 17‐54.

Johnson, M. (1987) ‘The Body In The Mind: The Bodily Basis of Meaning, Imagination, and Reason’. Chicago: University of Chicago Press.

Lakoff, G. (2008) ‘The Neural Theory of Metaphor’, in Gibbs, R. (ed.) The Cambridge Handbook of Metaphor and Thought. New York: Cambridge University Press. Citation from updated edition of paper available at: http://dx.doi.org/10.2139/ssrn.1437794

Lakoff, G. and Johnson, M. (1980) Metaphors We Live By. 10th ed. London: University of Chicago Press.

Lakoff, G. and Johnson, M. (1999) Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought. New York: Basic Books.

LennyBound (2008 22 January) Part 1 – V.S. Ramachandran at Beyond Belief 2.0. Available at: https://www.youtube.com/watch?v=N9hy7oOhHxk (Accessed 10 January 2017).

LennyBound (2008 22 January) Part 2 – V.S. Ramachandran at Beyond Belief 2.0. Available at: https://www.youtube.com/watch?v=1_9fR_yYEXE (Accessed 10 January 2017).

LennyBound (2008 22 January) Part 3 – V.S. Ramachandran at Beyond Belief 2.0. Available at: https://www.youtube.com/watch?v=7hIK3D1r4ro (Accessed 10 January 2017).

‘Linguistics’ (2017) Wikipedia. Available at: http://en.wikipedia.org/wiki/Linguistics (Accessed 20 July 2016).

McAdams, S,. (ed.) Bigand E. (ed.) (1993) Thinking in Sound: The Cognitive Psychology of Human Audition. USA: Clarendon Press.

McGurk, H. and Macdonald, J. (1976) ‘Hearing lips and seeing voices’, Nature. 264, pp. 746-748.

doi:10.1038/264746a0

McNerney, S. (2011) A Brief Guide to Embodied Cognition: Why You Are Not Your Brain. Available at: https://blogs.scientificamerican.com/guest-blog/a-brief-guide-to-embodied-cognition-why-you-are-not-your-brain/ (Accessed 28 June 2016).

‘Metaphor’ (2017) Wikipedia. Available at: http://en.wikipedia.org/wiki/Metaphor (Accessed 20 July 2016).

Milner, G. (2010) Perfecting Sound Forever: The Story of Recorded Music. London: Granta Books.

Murch, W. (1994) ‘Foreword’, in Chion, M. Audio‐Vision: Sound on Screen. New York: Colombia University Press.

National Film Board of Canada (2011 24 November) Dots. Available at https://vimeo.com/32645760 (Accessed: 12 January 2017).

National Film Board of Canada (2012 23 March) Neighbours. Available at: https://vimeo.com/39056719 (Accessed: 12 January 2017).

‘Neural Basis of Synesthesia’ (2017) Wikipedia. Available at https://en.wikipedia.org/wiki/

Neural_basis_of_synesthesia (Accessed 20 July 2016)

Palestrina, G. (1964) Pope Marcellus Mass. London: Novello and Company Ltd. (Original work published 1567).

‘Perception’ (2017) Wikipedia. Available at: https://en.wikipedia.org/wiki/Perception (Accessed 20 July 2016).

‘Prototype theory’ (2017) Wikipedia. Available at: https://en.wikipedia.org/wiki/Prototype_theory (Accessed 20 July 2016).

PsychologicalScience (2015 07 April) George Lakoff: How Brains Think: The Embodiment Hypothesis. Available at: https://www.youtube.com/watch?v=WuUnMCq-ARQ (Accessed 12 January 2017).

Purple Numbers and Sharp Cheese, (2003) BBC Radio 4, 23 April. Available at: http://www.bbc.co.uk/programmes/p00gpxdm (Accessed: 02 January 2017).

Ramachandran, V (2007) 3 Clues to Understanding Your Brain, [Recorded Lecture] Available at: https://www.ted.com/talks/vilayanur_ramachandran_on_your_mind (Accessed: 10 January 2017).

Ramachandran, V., Hirstein, W. (1999) ‘The Science of Art: A Neurological Theory of Aesthetic Experience’, Journal of Consciousness Studies, 6(6-7), pp. 15‐51.

Ramachandran, V., Hubbard, Ed. (2003) ‘The Phrenomenolgy of Synaesthesia’, Journal of Consciousness Studies. 10(8), 2003, pp. 49‐57.

Ramachandran, V., Hubbard, E., Butcher, P. (2002). ‘“Higher” and “lower” forms of synesthesia may arise from cross-wiring at different cortical stages’, Journal of Vision. 2(7)

Ramachandran, V., Hubbard, E. (2001) ‘Synesthesia – A Window Into Perception, Thought and Language’, Journal of Consciousness Studies, 8(12), pp. 3‐34.

Romke, R., Scholte, H. (2007) ‘Increased structural connectivity in grapheme-color synesthesia’, Nature Neuroscience. 10, pp. 792-797. DOI: 10.1038/nn1906

Sagiv, N and Ward, J. (2006) ‘Crossmodal Interactions: Lessons From Synesthesia’, Progress in Brain Research. 155, pp. 259‐271. DOI: 10.1016/S0079-6123(06)55015-0

Shams, L & Kim, R. (2010) ‘Crossmodal Influences on Visual Perception’, Physics of Life Reviews, 7(3), pp. 269‐284.

Shayan, S., Ozturk, O., Sicoli, M. A. (2011) ‘The Thickness of Pitch: Crossmodal Metaphors in Farsi, Turkish, and Zapotec’, Senses and Society, 6(1), pp. 96‐105.

Stafford, B. (1999) Visual Analogy: Consciousness as the Art of Connecting. Massachusetts: The MIT Press.

‘Star Guitar’ (2003). In The Work of Director Michel Gondry [DVD], USA: Palm Pictures.

Talks at Google (2007 16 July) George Lakoff | Talks at Google. Available at: https://www.youtube.com/watch?v=jNLP88aTg_8 (Accessed 11 January 2017).

The Middle Way Society (2013 11 November) The embodied view of meaning. Available at: https://www.youtube.com/watch?v=jgpWtdOHlEI (Accessed 12 January 2017).

Van Campen, C. (2008) The Hidden Sense: Synesthesia in Art and Science. Massachusetts: The MIT Press.

Viers, R. (2012) Location Sound Bible: How to Record Professional Dialogue for Film and TV. California: Michael Wiese Productions.

Viers, Ric. (2008) The Sound Effects Bible: How to Create and Record Hollywood Style Sound Effects. California: Michael Wiese Productions.

Whittington, W. (2007) Sound Design & Science Fiction. Austin: University of Texas Press.

Whittock, T. (1990) Metaphor and Film. Cambridge: Cambridge University Press.

Yorke, T. (1995) High and Dry. United Kingdom: Parlophone.

Zbikowski, L., (2009) ‘Music, language, and multimodal metaphor’, in Multimodal Metaphor. Berlin: De Gruyter, pp. 359-378.

zencat01 (2014 13 November) Consciousness, Synesthesia, and Art (V.S. Ramachandran). Available at: https://www.youtube.com/watch?v=yJbtgxs-wZc (Accessed 30 January 2017).

Get the Full Experience
Subscribe today and view all issues in full from just £10 for 3 months.

Subscribe Today
Comments are closed.