Synopsis: “What’s This?” is a data art audiovisual installation that explores the concepts of real-time visualization and sonification of streams of big data, extrapolating Baudrillard’s ideas of simulation and simulacrum as control mechanisms of the modern world. The simulacrum is true.
This installation consists of a data interface built in Max/MSP that transforms in real-time data streams of financial and weather APIs into MIDI data (notes and control). These signals communicate with a DAW through a virtual MIDI driver, that transforms them into audio signals using various synthesis techniques and digital effects processing, exploring the control of these tools in real-time. The same data streams also feed a visual interface (Max/Jitter) that implements several interpretations of these streams.
Since the beginning of time, mankind tries to understand the world. In order to do so, we build mental models and simulations of reality. By analyzing and describing the world through these models and simulations, we break the link with reality, which inevitably is more comprehensive and complex than what these reductions can capture. By making decisions and taking action based on these miniaturized representations of reality, we risk constant deception.
One of the clearest examples of this process is the concept of the map that represents the territory. In the beginning, the territory preceded the map, reaching its peak as Borges’ fable tells us: “the cartographers of the Empire draw a map so detailed that it ended up covering the territory exactly”. However, this has not been the case for quite some time. The abstraction of the map is no longer of the concept, the double, or the mirror but rather a simulation of a real without origin or reality: a hyperreal – the map precedes the territory.
In today’s world, this condition goes even further, trying to make the real coincide with the simulation models. But it is no longer a matter of “map and territory” because the difference between them has been lost, eliminating the metaphysics of abstraction: “There is no longer the mirror of being and appearances, of the real and its concept. There is no longer imaginary coextensivity: it is genetic miniaturization that is the dimension of simulation.” Thus, we can say that the hyperreal is produced by means of matrices, memories, and command models, which are nothing but miniaturization of the real that can be reproduced over and over again from these. We must then clarify the difference between simulation and representation:
“Such is simulation, insofar as it is opposed to representation. Representation stems from the principle of the equivalence of the sign and of the real (even if this equivalence is Utopian, it is a fundamental axiom). Simulation, on the contrary, stems from the Utopia of the principle of equivalence, from the radical negation of the sign as value, from the sign as the reversion and death sentence of every reference. Whereas representation attempts to absorb simulation by interpreting it as a false representation, simulation envelops the whole edifice of representation itself as a simulacrum.”
In this way, Baudrillard systematizes the different phases of the image as follows:
- it is the reflection of a deep reality.
- it masks and deforms a profound reality.
- it masks the absence of a deep reality.
- it has no relation to any reality: it is its own pure simulacrum.
“In the first case, the image is a good appearance – representation is of the sacramental order. In the second, it is an evil appearance – it is of the order of maleficence. In the third, it plays at being an appearance – it is of the order of sorcery. In the fourth, it is no longer of the order of appearances, but of simulation.”
From these reflections, a comparison is hereby made with data streams from two major categories: meteorological (temperature, humidity, wind) and financial (gold/currencies/cryptocurrencies). Note that the weather data is from 4 central locations of the stock exchange market: New York, Tokyo, London, and Frankfurt. These act as two extremes of the simulation: the first where the image is a reflection of a deep reality (a first-order simulation) and the other where the image no longer bears any relation to reality and is, therefore, its own simulacrum. The sonification and visualization of this data are also a simulacrum because, in order to carry them out, we created matrices, memories, and control models that can be reproduced indefinitely as long as there is input data.
Since the very concept of simulation goes beyond the abstraction of the real, the result of this installation is cryptic, not indicating all the steps that the models used, but referencing a logic that self-structures itself according to the input data. There is no direct representation of the data, but rather a self-managed restructuring of the data through its own models.
In essence, this installation is also a hyperreality in itself.
Created by João Guimarães and Vasco Santos
In the video below you can watch a video demonstration of the installation and an interview at the Semibreve Festival where the installation was on permanent display from October 28th – October 31st, 2021.
João Guimarães 