Cinema in the age of instant creation

by Boris and Brecht Debackere


Live cinema, a term originally used to describe the live musical accompaniment to silent film, has been gaining popularity in the last few years. It is no longer defined as a backing to film but as a quest for a symbiosis between sound and image, performed in real time.

How should an audiovisual practice in the age of instant creation manifest itself? No real language or theory of Live Cinema has been written so far. It is clear that Live Cinema intersects with many other art forms: performance, film/cinema, music, “colour organs”, light art, theatre, motion graphic design, programming. From this point of view, a interdisciplinary art form.

Music has always influenced visual artists and filmmakers; composing with both image and sound is a recurring theme throughout art history. Merging the two within compositions of Colour music, light organs, light performances and abstract animation points towards an aesthetic of temporal abstraction which digital computer graphics nowadays can ideally explore.

A closer look at the historically connected forms shows us that the many attempts to formalize a universal syntax of a medium lead to false practices neglecting the basic qualities of the medium itself. Alternatively the language of a Live Cinema performance should emerge from the elements the artist wishes to work with, a system set up and initiated by the creator, with its own properties of behaviour and evolution. So instead of trying to define an absolute theory, Live Cinema could benefit much more from outlining different “open” strategies.

The digital revolution caused a shift in the use of audiovisual media towards real-time human interaction with the medium by introducing instant control for the creators as well as for the audience. From real-time editing to computer games with filmic atmospheres, from multi-angle viewpoints on DVD to digital television, they all put the maker or viewer in a more active role.

Live Cinema ≠ VJing

Real-time video in all its shapes and forms is a hot item anno 2006. One would be mistaken to think all aspects of the diverse practice of real-time video manipulation, creation, editing and processing is covered by the term “VJing”. Where Live Cinema performance is situated within both traditional music performance and visual music practice, VJing started out ambiguously and undesignated. Its roots lay both in the club scene with television rigs being used as alternative lighting and atmosphere tools, and MTV that used the term for its on-air presenters, modelled on the radio DJ equivalent. In other words, VJs were creating and defining the ambience in different clubs.
The founding father of ambient music, Brian Eno, describes ambient as a low-attention type of art that caters to an audience that isn’t there specifically for your product but for a whole environment. From this point of view, it become clear that VJing is a low-attention form that holds the middle between interior decoration and lighting.

Live Cinema performances, on the other hand, are an active form of real-time and real-space audiovisual creation. It is not one of the elements defining the ambience of a place but the sole point of attention. The spectator fulfills an active part in the performance, where his or her presence plays an essential role in placing the artwork in real space and experiencing the uniqueness of instant creation.

Colour Music

The idea of colour music is one that spans centuries of history and branches out into many different theories and practices. In a search for natural analogies between sound and colour, as a physical phenomenon, many have been tempted to propose theories based on “numerical mystics”, thus resembling the music science in Ancient time and Middle Ages that searched for musical laws in the cosmos and considered harmony as an ontological system, outside the human psyche.
When Pythagoras discovered a numerical relation between musical tones his intuition led him to propose that the planets, while running in their orbits, produced “Musica Mundana” or “music of spheres”.

Pythagoras and later also Aristotle speculated on the correlation between musical scale and the rainbow spectrum of hues [1]. These ideas infiltrated the philosophy of the epochs following.

In his book Harmonice Mundi (Harmony of the Worlds) (1619), Johannes Kepler attempted to notate the concrete “music of spheres” score, explaining proportions and geometry in planetary motions by relating them to musical scales and intervals. “The heavenly motions are nothing but a continuous song for several voices, perceived not by the ear but by the intellect, a figured music which sets landmarks in the immeasurable flow of time.”

Isaac Newton, who based his theory of gravity on Kepler‘s work, was also attracted by the idea of a divine predestination of musical order in the universe. Newton‘s studies of light clearly demonstrated that light alone was responsible for colour. He conceptually arranged the colours around the circumference of a circle and linked the seven colours with the seven tones in the octave.

From 1725 onwards, Bernard Castel published several descriptions of a so-called Clavecin Oculaire. His writings were based based on Newton’s original sound to colour analogies and the reflections of Athanasius Kircher, the populizer of the “Laterna Magica”, on whether it is possible to see music. Eventually Castel realized his “colour organ” constructed out of candles, light filters and a harpsichord keyboard. The conception of colour music was illustrative of the culture of The Age of Enlightenment. By simply isolating musical tones and translating them into colour, Castel reduced and neglected audible and visible qualitative differences. As Hegel pointed out, “the worthless games with idle analogies have caused valid contempt to natural philosophy”. As for Castel, he regarded analogies as a “guide” in the search of truth and used them in his theory of colour music.

The enthusiasts of colour music were mainly scientists, producing ideas lacking aesthetical motivation in favour of metaphysical dreams of a perfect universe. The idea was scattered throughout many different contradicting theories and instruments, all claiming the ability to map sound to image. Still, the magic of the concept started to appeal to many artists as well.

Eventually, at the dawn of electricity, the colour organ was upgraded with electrical light.Two notable colour music performances were given around the turn of the 20th century and apparently neither of them was very well received [2]. Alexander Rimington, Professor of Fine Arts at Queen’s College, performed on the colour organ in St. James’s Hall in London in 1895. His one-to-one mapping of a single note to a specific colour proved problematic and the London critics were not kind, commenting on the “restless flicker” on the screen [2].
In 1915, a colour organ was used in the performance of Alexander Scriabin’s symphonic poem Prometheus: A Poem of Fire , but the performance (at least the visual aspect) was not very successful. A review in the New York Times (March 21, 1915) pointed out what may have been the performance‘s most prominent shortcoming: “So far as the lights were concerned, it could not be discovered how they added to or intensified the meaning of the music.”

Visual music can also be linked to the psychological phenomenon of synaesthesia, a term used to indicate a condition in which the stimulation of one sensory modality gives rise to an experience in another.

The most common experience of this is the “coloured hearing”, but the mapping between tone and colour differs notably between synaesthetes.
Visual music or colour music can best be put forward as a metaphor for an associative transformation from one sensory experience into another. When taken literally a one-to-one colour to music translation is not convincing. But the spirit of the age and the lack of adequate technological means naturally led to finite colour music approaches.

Abstract Film

Due to the invention of photography and its subsequent role as a representational medium, painting was freed from the burden to copy and preserve reality. Painting thus evolved, no longer representing what’s there, but free to move on and search for means to represent personal impressions of reality and expressions of intense emotions.
Wassily Kandinsky, credited with making the first modern abstract work, tried to express the soul of nature and humanity. Probably due to his synaesthesia, Kandinsky called this the “Innerer Klang” or “inner sound’ [4], and linked it to the experience of music. The lines and colours resembled harmony and rhythm, and the titles of his works, and those of many other abstract painters, often referred to musical concepts, stressing the attempted visual music.
Inspired by the musicality of Kandinsky several young artists working in the field of non-representational art seemed to have been inspired either by his writings or by the time spirit. Unsatisfied with the static nature of painting they were looking for a moving alternative for their work. Although the composition of a painting was supposedly directing the gaze of the spectator, it was not precise enough; there wasn’t enough control.
Amongst them, two Bauhaus artists, Ludwig Hirschfeld-Mack and Kurt Schwerdtfeger, developed their “Reflektorischen Farblichtspielen” out of rhythmically moving stencil keys that were used to shape projected light in real time.

One of the protagonists of abstract film, Walter Ruttmann, observed the growing interest in time-based technologies and concludes in his essay “Malerei mit Zeit”: “The gaze that in intellectual matters is increasingly driven towards a focus on a temporal occurrence can no longer find the rigid, reduced timeless forms of painting relevant”.[5]

His first abstract short, which is considered to be the very first abstract film presented to a general audience, “Opus I”, was reviewed bearing the title “A New Art, The Vision-Music of Films”. The author, Bernhard Diebold, was deeply impressed, as was his young friend, Oskar Fischinger, who would later on produce an impressive body of abstract visual music films and the Lumigraph colour organ.

Among Ruttmann’s contemporaries were Hans Richter and Viking Eggeling, two other painters looking for a way to use the time element in their work. Swedish painter Viking Eggeling was working on a universal language of symbols and searching for the rules of a plastic counterpoint [6]. He accidentally met Hans Richter and the two started sharing their ideas. First with abstract sketches, later on scroll paintings and finally on film in order to fully take advantage of the time dimension.

All these artists aimed to equal a musical experience through abstract visuals with Fischinger’s “Radio Dynamics” explicitly asking for “Please! No music” or Ruttmann’s “Opus I” starting with an explanation of his intentions.

Experiment in colour-rhythm still seemed to be confined as a link between the images and the music. They managed to move away from the simplicity of the colour organs by structuring their works by means of dynamics, tempo, rhythm, counterpoint and more in general, composition.
But the time it takes to create an abstract movie, to carefully plan its structure and composition, to animate the shapes and develop the film. In short, the time it takes to have control over ‘time’ is gained at the expense of the real-time factor. The outcome of a seemingly free visual interpretation of music results in an extremely rigid and fixed structure on celluloid to be enjoyed by disembodied spectators in a darkened room: the environment is negated and the entire focus is directed towards the screen.

The amalgamation of visual music

The relationship between early abstract or absolute film and the visual music experiments with colour organs is a complementary one. In a way they were each other’s counterpart, but they gradually evolved into a unifying whole over the course of the 20th century.

The visual music of colour organs was limited by both its starting point, being the direct coupling of music and colour, and by the technology, e.g. electric lighting. In the case of Thomas Wilfred’s ‘Clavilux’, a visual music instrument he built to produce real-time shapes and colours or ‘lumia’, the shapes and forms the machine was capable of producing were defined by the shape of the tungsten filament in the light bulbs he used. As one critic, Adrian Bernard Klein, rightfully observed: “If we are to have form in colour music, that form must be as much the intention of the artist as it is in painting, and not dependent on chance circumstances.”[7]

The control over formal expressions that the color organs were lacking was exactly the strongpoint of the animated abstract film. Most of the animators were traditionally schooled painters working in abstract expressionism before they moved on to film.

Nonetheless, the immediacy of a visual instrument was lost and the importance of a spectatorial presence denied due to the inherent time-difference between the exhaustive production-process and the moment of presentation.

In the 1950’s the overhead projector made its way from the classrooms into the performance world. Its potential to project and amplify anything thrown on the glass plate to immense proportions made it the ideal tool for the live visual accompaniment of concerts. It was primarily used for the so-called ‘wet shows’, where artists used paint and oil to produce colourful psychedelic imagery. Although its image producing capacities were limited, it did provide performing visual music artists with an unrivalled freedom to improvise and experiment without the restraints of a fixed musical link.

With continuous experiments in both abstract film and color organs, the technological advancements of the Second World War added two key elements to the further development of visualization in general.
One of them was the radar, offering a real-time rendering of a visual representation and at the same time creating an early form of telepresence.

The other invention was the first motion controlled camera, built by John Whitney with analog computer mechanisms and used to mathematically plot graphics onto film. [8]

For the complete mergence of real time and real space to occur within a performance with seemingly unlimited visual potentials, a few things were still missing.
A technological development that had been under constant improvement at that time was television. Through television, it was possible to reproduce moving images in real time, bringing abstract film one step closer to abstract visual music performance. The real-space problem was still to be tackled. Apart from the ‘eidophor’, a television projector in the making, there were no means for the performer to produce and control shapes and color on location. The answer came in the form of the video synthesizer, an instrument that manipulates the analog video signal.

With the development of graphic computer imaging the ultimate visual machine was born, capable of generating visuals and sound and/or processing pre-recorded footage, distributing the result in real time through both the video projectors and the PA system.

Digital Media

As every medium renders a translation of its predecessors, the computer transcribes multiple media at once. The inherent way this machine digitizes all input to a binary form holds the expectation of a truly multi-sensory medium.
Nevertheless the computer’s basic hardware properties: input – processor – output, brings us back to the primary question. What kind of translation process can be applied from one sensory experience (input) to another (output)?

In computer terminology, mapping is often used to describe the system of connections between different elements. According to Thomas Ciufo, “one unique potential of these new computer systems is that of sound-to-image and image-to-sound mapping, making sound and image structurally integrated. To achieve this integration, specific aspects of the audio can be analyzed and used to directly control the manipulation of specific aspects of the image, and visa versa.”

Unfortunately this approach of mapping reduces the computer‘s potential back to a digital version of the color organ. Examples can be found in commercial products such as iTunes and Winamp that create screensaver-like imagery driven by the music that is being played.
The resulting visuals of these two applications might not be interesting as such but they do exemplify two essential aspects of the computer: creating a ‘virtual space’ in ‘real time’. These two properties are an opportunity to surpass the time consuming production of the early abstract and absolute filmmakers (real time) and the physical, mechanical limitations in producing imagery (virtual space) by the color organ builders.

Every new medium re-introduces the belief in mystical possibilities of universal media translations. From Pythagorean theories to the digital computer, people felt tempted to rely on exterior ‘truths’ or technologies for their decision-making In relation to the computer, Norbert Wiener, pioneer in the study of stochastic or random processes, defined it as “to shift responsibility from man and put it on the device, which is supposedly inconceivable, but possesses the indisputable objectivity.”

Strategies for Live Cinema

“The creation of an audiovisual experience trough the improvisation with an abstract images and sound- generating computer system.”


“Creating a virtual multi sensory universe.”

As we have seen with the colour organ experiments, a one-to-one mapping is limited and often favours the structures of one medium. Since all previous media are accumulated within the computer, no ‘real’ distinction can be made on the grounds of their nature but only by sensorial perception. As a result, no single medium can claim superiority in order to achieve a symbiosis.

In order to create a meaningful synergism we propose an emerging, “bottom up” system, where the starting point is a concept for a system t hat will be implemented in a computer program. This system is like a meta-actor and becomes the intermediary through which image and sound are linked to each other. The audible and visible elements have no direct connections but are connected through the intermediary.

Every element, the performer included, is altering the state of the whole and the relations whitin. In this manner, the system functions as a hub, regulating, limiting and/or allowing the input of all three: performer, image generator and sound generator. It comes to life from the moment the interaction is initiated between performer, sound and image. This makes structuring a piece, as a “top down” composition, irrelevant. The result is a controlled feedback system.

Since no universal tool exists, the performer needs to concretize his ideas and concepts in a custom application. A computer program that is capable of generating images and sounds, taking into account how and what the performer wishes to control and manipulate during the performance. The amount of controllable parameters should very well be considered, so that it doesn’t results in an unmanageable system.
The tool however should not be limited to a single composition and is in this respect comparable to a musical instrument. This implies that it takes time to master the tool and only through rehearsal will the actual expressive powers or limitations of the tool present themselves.

Quite often the flow of things doesn’t happen in the strict order that has been described. It’s most likely that the process of creating this tool is a constant dialogue between programming and playing.
Since it is a virtual tool, the possibility to easily create different versions and make refinements is a huge advantage. Colour organs relied on physical mechanisms and, once constructed, they were harder to change.

©2006 Boris & Brecht Debackere

“Cinema in the age of instant creation”
was originally published in Transmedia Textseries 04.

If you like to receive a hard copy,
surf to and email your request.


[1] W.Moritz:
The Dream of Color Music, and Machines that made it Possible,
Animation World. Issue 2.1, April, 1997.
[2] W.Thomas:
Light and the Artist.
Journal of Aesthetics and Art Criticism, June, 1947.
[3] G.Levin:
Painterly Interfaces for Audiovisual Performance.
MIT Master Thesis, 2000
[4] W.Kandinsky:
Concerning the Spiritual in Art.
The Project Gutenberg, March 2004 (10th edition),
[5] H.Helfert:
Technological Constructions of Space-Time, Aspects of Perception,, May 2006
[6] R.Russett:
Experimental animation, origins of a new art.
New York: Da Capo Press, Inc. 1976 (2de druk)
[7] A.B.. Klein:
Colour-Music: The Art of Light.
London: Crosby Lockwood & Son, 1930, p. 195.
[8] W.Moritz:
Digital Harmony: The Life of John Whitney, Computer Animation Pioneer,
Animation World Magazine, Issue 2.5, August 1997