Cycle is a series of technology-assisted performances, incorporating the use of robotics and sound. It was inspired by the interrelating concepts from Graphic Notation and East Asian Calligraphy/Ink Wash Paintings. In each unique recurrence, Cycle explores the theme of spontaneity and individuality transpired within a structured framework as the performers present their own interpretation of a set of instructions.
Each performance lasts approximately three minutes, give or take a minute; the performers end it at their own discretion. During the performance, a sole performer walks around the ‘Ink Stick Rotation Machine’ (ISRM) in a seemingly undefined way. The ISRM grinds an ink stick on an ink stone according to how the performer walks. Ambient sounds and vibrations generated from the constant moving contact of the ink stick and ink stone are amplified by speakers through a microphone located on the sides of the ink stone in real time.
In the performer’s interpretation from a set of rules constructed by the graphic score’s composer, control over the manner of performance is removed from the composer’s authority which alludes to a spontaneous creation of the performance by leaving it to ‘chance’. Unlike music represented in traditional notation, different performances of one graphic score do not have the same melody yet still articulate similar notions expressed in the score. In the cases of Ink Wash painting, the rules in posture, way of holding the brush, and practiced strokes, the results cannot be fully controlled by the painter and are still unpredictable due to human error and the nature of ink and water – their interaction take on a life of its own.
The audience sees and listens – nothing really comes out of watching the performance. Yet, even if the audience does not understand the concepts implicated in this work due to requiring some background knowledge about the act of grinding an ink stick, to experience Cycle, they merely have to practice being in a state of calmness and ambiguity. Just like when a painter or calligrapher prepares ink by manually grinding the ink stick, it is to ebb their flow of thoughts, momentarily forget about the things that are happening outside of the performance and just watch and listen. The performance would be both like a ‘performance’ and a non-religious ‘ritual’ at the same time. The feeling that one would sense is like when one is a non-Buddhist listening to the chants of Buddhist monks. Strangely calming, yet it could get annoying when one listens to an ununderstandable language for too long.
For the performers, I would hope that they would be in a world of their own without minding the presence of the audience and focus on their body walking in a circular path, yet I can imagine that they would perhaps be nervous in front of an audience, especially if they are performing for the first time. As a recurring theme in my work, ‘walking’ is a simple movement that can be of disinterest and a distraction all the same. It not only refers to the bodily action of moving your legs as a mode of transport but also signifies the act of repetition, which is structural, and the mundane. As the performer walks after a few times, the performer may build up a personal routine or choose to walk a different manner each time.
After my research on Graphic Notation and East Asian Ink Wash Paintings, I have drawn connections between these two distinctively different genres in art and show their overlapping characteristics in which my artwork attempts to embody conceptually. I likened graphic notation to instructions that were rather open-ended yet specific in certain ways, hence, I decided on creating a performance that Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink.
Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink. With the addition of the sound of the motor, I thought that the sound would be a nice hybrid between the organic and inorganic materials.
In the late 1950s and the first half of the 1960s, many prominent international avant-garde composers such as Roman Haubenstock-Ramati, Mauricio Kagel, and Karlheinz Stockhausen, as well as experimental composers such as John Cage, Morton Feldman, and Christian Wolff started to produce graphic scores that used new forms of notation and recorded them on sheets that were very divergent from traditional music notation in size, shape, and colour. This new way to convey ideas about music alters the relationship of music/sound to the composer and musician. “In contrast to scores with traditional notation, graphic notation emphasized concepts and actions to be carried out in the performance itself, resulting in unexpected sounds and unpredictable actions that may not even include the use of musical instruments.” (Kaneda, 2014)
Here, I focus on how graphical notation evolved from John Cage’s musical practice and then on Treatise, one of the greatest graphical scores, by Cornelius Cardew.
Influence of Zen Buddhism in Cage’s Work
In Cage’s manifesto on music, his connection with Zen becomes clear: “nothing is accomplished by writing a piece of music; nothing is accomplished by hearing a piece of music; nothing is accomplished by playing a piece of music” (Cage, 1961).
This reads as if a quote from a Zen Master: “in the last resort nothing gained.” (Yu-lan, 1952). Cage studied Zen with Daisetz Suzuki when the Zen master was lecturing at Columbia University in New York. Zen teaches that enlightenment is achieved through the profound realization that one is already an enlightened being (Department of Asian Art, 2000). Thus we see that Cage has consciously applied principles of Zen to his musical practice: he does not try to superimpose his will in the form of structure or predetermination in any form (Lieberman, 1997).
Cage created a method of composition from Zen aesthetics which was originally a synthetic method, deriving inspiration from elements of Zen art: the swift brush strokes of Sesshū Tōyō (a prominent Japanese master of ink and wash painting) and the Sumi-e (more on this in the next section) painters which leave happenstance ink blots and stray scratches in their wake, the unpredictable glaze patterns of the Japanese tea ceremony cups and the eternal quality of the rock gardens. Then, isolating the element of chance as vital to artistic creation which is to remain in harmony with the universe, he selected the oracular I Ching (Classic of Changes, an ancient Chinese book) as a means of providing random information which he translated into musical notations. (Lieberman, 1997)
Later, he moved away from the I Ching to more abstract methods of indeterminate composition: scores based on star maps, and scores entirely silent, or with long spaces of silence, which the only sounds are supplied by nature or by the uncomfortable audience in order to “let sounds be themselves rather than vehicles for man-made theories or expressions.” (Lieberman, 1997)
John Cage: Atlas Eclipticalis, 1961-62
Atlas Eclipticalis is for orchestra with more than eighty individual instrumental parts. In the 1950s, astronomers and physicists believed that the universe was random. Cage composed each part by overlaying transparent sheets of paper over the ‘Atlas Eclipticalis’ star map and copied the stars, using them as a source of randomness to give him note heads. (Lucier, 2012)
In Atlas, the players watch the conductor simply to be appraised of the passage of time. Each part has arrows that correspond to 0, 15, 30, 45, and 60 seconds on the clock face. Each part has four pages which have five systems each. Horizontal space equals time. Vertical space equals frequency (pitch). The players’ parts consist of notated pitches connected by lines. The sizes of note heads determine the loudness of the sound. All of the sounds are produced in a normal manner. There are certain rules about playing notes separately, not making intermittent sounds (since stars don’t occur in repetitive patterns), and making changes in sound quality.
Cornelius Cardew: Treatise, 1963-67
After working as Stockhausen’s assistant, Cornelius Cardew began work on a massive graphic score, which he titled Treatise; the piece consisting of 193 pages of highly abstract scores. Instead of trying to find a notation for sounds that he hears, Cardew expresses his ideas in this form of graphical notation, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated (Cardew, 1971). The scores were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes. (Tilbury, 2008)
As John Tilbury writes in Cornelius Cardew: A Life Unfinished (2008), ” The instructions were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes.”
“A Composer who hears sounds will try to find a notation for sounds. One who has ideas will find one that expresses his ideas, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated.” – Cornelius Cardew
In the Treatise Handbook which guides the performer on the articulation of the score, Cardew writes that in Treatise, “a line or dot is certainly an immediate orientation as much as the thread in the fog” and for performers to “remember that space does not correspond literally to time.” (A Young Persons Guide to Treatise, 2009)
East Asian Ink Wash Painting
The Enso, or Zen circle, is one of the most appealing themes in Zen art. The Enso itself is a universal symbol of wholeness and completion, and the cyclical nature of existence, as well as a visual manifestation of the Heart Sutra, “form is void and void is form.” (Zen Circle of Illumination)
Despite there being many specific technicalities in Cage’s work, these are all qualitative instructions which are open-ended, ultimately leaving it up to the performer’s or conductor’s judgement on how they would play the piece as implied by Cardew’s ideas. In a sense, the individuality of each performance of the graphic score by different performers emerges. This is mirrored in appropriating the creation of the Enso in Cycle by the performer. Every painter draws a circle but every circle is different. Bodily and mindfully engaged in drawing the circle, the circle becomes an allegory of the individual.
The performer not only becomes both the painter and the medium in creating the circle, the performer is also a musician with the indirect control of the device that grinds ink – the instrument with a naturalistic sound created from the contact between the ink stick and the ink stone. To quote Cage’s approach to what defines music, it is the “the difference between noise and music is in the approach of the audience” (Lieberman, 1997).
The act of grinding the ink stick becomes the juxtaposition between the ritualistic and the improvised. Also, ink that is produced after each performance are of different quality each time as no two performances will last the exact same time nor will the performers be able to replicate their performance exactly.
Communication between the phone and the computer is through OSC. The ISRM is made up of an Arduino Uno, which controls a stepper motor, which is directly connected to the computer with a USB cable. The speed and direction of the performer would be measured by the built-in sensors in a phone on the performer. Data from the orientation sensor and accelerometer of the phone is computed in a C++ program on the computer which maps the speed and direction of the performer to that of the ISRM.
Controlling the Stepper Motor with C++
The Arduino part was pretty straightforward as there was the Firmata library for the Arduino that enabled serial communication with a C++ program. However, there was no stepper library in C++, so I translated the Arduino stepper library to C++. Working through the technical details of the stepper motor that I had with some trial and error, this was the circuit that I used to test controlling the stepper motor through a C++ program.
Here’s me testing the program out:
To hold the ink stone, ink stick, and the stepper into a single functional entity, I started off with a preliminary design of a 3D model in Blender, which eventually I was going to 3D print.
I got the idea of the rotation wheel and axis from the driving wheels of steam locomotives, but I was not satisfied with the motions of the rotating mechanism in the first prototype. It caused the ink stick to rotate in a rather awkward manner that did not keep the ink stick facing the same direction. I also removed the water tank as I felt that it was visually obstructive and had no better purpose than to provide the ink stick with water, which I did not manage to figure out a fail-safe method of channeling the water into the ink stone. I thought of using a wick to transfer water from the tank to the ink stone, but water transfer was too slow, or a small hole with a pipe dripping water to the ink stone, but the rate of dripping will change when the water in the tank decreases due to decrease in pressure. Also, it would damage the ink stick if I let it touch the water for too long periods of time, hence I scraped the water tank from then on and decided to manually add water before every performance.
There were many difficulties trying to get the holder for the ink stick to fit. I realised that it was never going to fit perfectly as the dimensions of the ink stick itself was not uniform; one end of the stick could be slightly larger than the other end, which made it either too loose or too tight when I tried to pass through the entire length of the stick through the holder. I resolved this by making the holder slightly larger and added sponge padding on the inside of the holder so that it would hold the ink stick firmly no matter the slight difference in widths. The ink stick was shaky when it rotated so I increased the height of the holder to make it more stable. I also added a ledge on each side of the holder for rubber bands such that the rubber bands could be used to push the ink stick downwards as it gets shorter during grinding.
Before arriving at the final design, there were just wheels that were only connected to each other through the rod. The rotation did not work like expected of a locomotive wheel and I realised that it was because the wheel not connected to the motor had no driving force that ensured it spun in the right direction. Therefore, I changed the wheels to gears.
The printed parts did not fit perfectly and that was not because of the wrong measurements as there was a factor of unpredictability in the quality of 3D printing. I tried using acetone vapour on the parts that need to move independently of each other to smooth the surface, but the acetone vapour also managed to increase the size of the plastic. The plastic became more malleable so I easily shaved them down with a penknife.
This process was too slow and I ended up using a brush to brush on the acetone directly to the plastic parts and waited for a few seconds for it to soften before using a penknife. Super glue was then used to hold parts that were not supposed to move together. The completed ISRM:
I used electret microphones that were connected to a mic amp breakout, then connected to a mixer for the performance. I got an electret microphone capsule to use with the Arduino but I did not know that the Arduino was not meant to be used for such purposes and the microphone was not meant for the Arduino.
So, I got another kind which could directly connect to output as I did not want to use the regular large microphone which would look quite ostentatious with the small ISRM.
Trying to amplify the sound of making ink (sound is very soft because I only had earphones at that time, and I was trying to get the phone to record sound from the earphones):
Sensor Data & Stepper Motor Controls
I initially thought of creating an android application to send data to the C++ program via Bluetooth, but there was the issue of bad Bluetooth connectivity, especially the range and speed of communication. Hence, I switched to using OSC to communicate the data. Before finally deciding on using an OSC app, oscHook, I made an HTML5 web application with Node.js to send sensor data. It worked well except for speed issues as there was a buffer between moving the phone and getting the corresponding data that made it rather not ‘real-time’, and it also sent NaN values quite often which would crash the program if there were no exception handlers.
For controlling the speed of the stepper motor, I mapped the average difference of the acceleration of the y-axis (up and down when the phone is perpendicular to the ground) within the last X values directly to the speed of the motor. Prior to this, I looked at various ways to get the speed and direction of walking, from pedometer apps to compass apps. As different people had different sensor values with the phone, I created a calibration system that adjusted the values of the mean acceleration when the performer is not moving and when the performer was moving at full speed. This ensured that the stepper will be able to run at all speeds for all performers.
Link to Git Repo.
Performance & Installation
Videos of performances were playing on the screen for the second day of Symbiosis. The TV was covered with white cloth on the first day. The ISRM was placed on a white paper table cover with the microphone next to it.
Instructions for Performers
Besides having to run a calibration before their performances, I requested the performers to wear “normal clothes in darker colours” to make a contrast with the white room walls. I decided not to specifically ask for black as it was too formal and intimidating. Although the performance exudes the sense of a ‘ritual’, it was not meant to be solemn or grievous, as was such cultural connotations of fully black clothes in a rather ritualistic setting.
During the performance, the performers were to heed these instructions:
- Walk around the room.
- When you stop, stop until you hear the sound indicates that the motor is at its lowest speed.
- End the performance when it is three minutes since the start.
Prior to completing the program that controls the stepper motor, I wanted to attach the phone to a belt and hide it under the clothes of the performers such that they would be walking hands-free. I realised that it was quite abrupt to merely end the performance with the performer standing still as there was no indication if the performer was pausing or stopping entirely to the audience. Hence, after realising that by placing the phone parallel to the ground caused the motor (and in turn the sound) to stop in an elegant manner, I decided that the performer would hold the phone (which I covered in white paper to remove the image a phone) in their hand and have them place it on the ground to signify the end of the performance.
There was a total of eight performances by three people, Yun Teng, Leah, and Haein. These are videos* of the performances by each of them on the Symbiosis opening night and their thoughts on their experience of performing:
*The lights in the room were off during the day, hence videos of the earlier performances look quite dark. If you do not hear any sound from the video, please turn up the volume.
“Being asked to perform for this piece was an interesting experience. For me, it was seeing how (even on a conceptual level, as it turned out) that my physical movement can be translated through electronics and code into the physical movement of the machine and the audio heard. Initially, although we were given simple instructions to follow and even, to some extent, encouraged to push these instructions, I was at a loss to how to interpret them, and just walked in a circular fashion. I tried to vary the pace, speed and rhythm of my walking in order to create variation, but ultimately fell back into similar rhythms of fast, slow, and fast again. It would have been interesting to perhaps push this even further if the machine was more sensitive to height changes, or arm movements – as a dancer who is used to choreography, this was a challenge for improvisation and exploration. In addition, due to the size of the room, the space was limited and hence the walking could only take place in certain patterns.” – Yun Teng
“At first, the walker was uncertain, distracted and anxious. She explored the link between sound and her unchoreographed strides and expected the connection to be instantaneous and obvious. However, it was not. There were delays and inconsistencies; the electronic and mechanic could not accurately reflect the organic. A slight panic arose from the dilemma of illustrating the artist’s concept to the audience and accepting its discrepancies as part of the performance. Slowly she started to play around with the delay, stopping suddenly to hear the spinning sound trailing on, still at high speed, and waited for it to slow down. Rather than a single-sided mechanical reaction to movement, the relationship between the walker and the machine becomes visible and reciprocal. Rather than just walking, now she also had to listen, to wait, and by doing so interact with the machine on a more complicated level. Through listening, she felt the shadow of her movements played back to her by the machine. The observation sparked contemplation on the walker’s organic presence versus the machine’s man-made existence and the latter’s distorted yet interesting reflection of the former.” – Leah
“The whole practice first was received as confusing and aimless as there was too much freedom for one to explore. It was challenging to perform the same act (walking/running) for more than two minutes. At first, I performed more than four minutes, unable to grasp the appropriate time, but it decreased as I repeated the practice. This repetitive performance was quite meditative and physically interactive with the work that caused me to wonder about the close relationship between myself and sound piece (which changes according to my walking speed). The most pleasant part of the performance was that I got to control the active aspect of the work and directly interact with it.” – Haein
The audience was very quiet, probably so that they could hear the sound that was very soft even at its loudest. When they first came in, they did not know what to do as there was no visible sitting area (so I directed them to sit at places that allowed the performer to roam most of the room). It was a huge contrast to the audience that interacted with my previous work as only the performer gets to have a direct interaction with the ISRM. Even then, the ISRM was visibly moving during the performances.
Just hours before the opening night, the ISRM broke at (fig. A & B). It was a mistake on my part as I was reapplying super glue (fig. B) to the base as it had somehow loosened from the previous application of super glue. In hindsight, I did not make extra parts (I did print extras of certain, not all, parts but they of no use when I did not bring them on site, nor were they ‘acetoned’ to fit together.), could not manage to salvage the parts, and I knew that I would not be able to reprint the parts in time. In the end, I slightly altered my work as the ISRM could no longer function as intended. Instead of having the microphones stuck to the sides of the ink stone, I stuck them to the stepper motor instead. Although the sound no longer had an organic element from the ink stick and ink stone, it was completely mechanical now.
After undertaking this project, I have learnt not to limit myself by my tools, but to explore different methods and tools before limiting myself in the creation of the work. I had a misconception that 3D printing was the most efficient way. In some ways, it was because it was the printer that was doing the hard work, not me and I did want to try 3D printing. Despite that, I should not have limited myself by my lack of consideration in using other materials to build the ISRM, such as the traditional way of putting together wood and gears. On the other hand, I do not regret my attempts to build an android app (which I quickly decided was not worth my time for the simple thing I was trying to accomplish) and a web application for sending the sensor data from the phone with Node.js as it is something new that I learnt even though I did not use it in my final work.
Fortunately, I managed to finish the design of the ISRM and print it out in time, but I felt that I should have focused more on the ISRM instead of coding in the earlier phase of the project timeline. 3D printing takes a lot of time, as I have experienced through this project, and any botched prints needed to be printed again as they are rarely salvageable even after being in print for hours. It is also tricky to get the settings right (i.e. infill) such that the printing time is minimised without compromising the quality.
Apart from the many technical things, I also learnt how to organise a performance art (this is my first performance art) and through making this artwork, there many more implications and questions that arise from what I created. For the performance, there were many things to keep track of, such as rehearsing with the performers beforehand, the attire of performers, the schedule of performances, getting the camera to film for documentation and managing the audience. In conclusion, despite being unable to carry out the performances as I have originally planned, I am glad that I have managed to still put together what is left of the entire work even when the ISRM failed to work correctly and the original intentions behind the artwork are still largely intact.
References & Bibliography
Works Cited in Background Research
A Young Persons Guide to Treatise. (12 December, 2009). Retrieved 2 November, 2015, from http://www.spiralcage.com/improvMeeting/treatise.html
Asian Brushpainter. (2012). Ink and Wash / Sumi-e Technique and Learning – The Main Aesthetic Concepts. Retrieved 2 November, 2015, from Asian Brushpainter: http://www.asianbrushpainter.com/blog/knowledgebase/the-aesthetics-of-ink-and-wash-painting/
Cage, J. (1961). Silence: Lectures and Writings. Middletown, Connecticut: Wesleyan University Press.
Cardew, C. (1971). Treatise Handbook. Ed. Peters; Cop. Henrichsen Edition Limited.
Department of Asian Art. (2000). Zen Buddhism. Retrieved 11 December, 2015, from Heilbrunn Timeline of Art History. New York: The Metropolitan Museum of Art: http://www.metmuseum.org/toah/hd/zen/hd_zen.htm
Kaneda, M. (13 May, 2014). Graphic Scores: Tokyo, 1962. Retrieved 2 November, 2015, from Post: Notes on Modern & Contemporary Art Around the Globe: http://post.at.moma.org/content_items/452-graphic-scores-tokyo-1962
Lieberman, F. (24 June, 1997). Zen Buddhism And Its Relationship to Elements of Eastern And Western Arts. Retrieved 10 December, 2015, from UCSC: http://artsites.ucsc.edu/faculty/lieberman/zen.html
Lucier, A. (2012). Music 109: Notes on Experimental Music. Wesleyan University Press.
Tilbury, J. (2008). Cornelius Cardew (1936-1981): A Life Unfinished. Copula.
What Ink Stick Should You Choose For Japanese Calligraphy? (2015). Retrieved 3 December, 2015, from Japanese Calligraphy: Modern Japanese Calligraphy inspired in Buddhism and Zen: http://www.theartofcalligraphy.com/ink-stick
Williams, M. L. (1981). Chinese Painting – An Escape from the “Dusty” World. Cleveland Museum of Art.
Yu-lan, F. (1952). A History of Chinese Philosophy. Princeton, New Jersey: Princeton University Press.
Code References & Software
by Akira Fiorentino and Uyen Tran Hong Le
XPLOR is an immersive exploration game / visual experience, viewed from a top-down perspective, where the player moves in 2D but the environment is in 3D.
The player controls a virtual fish-like life-form, guiding it around in an abstract environment populated by large, pulsating blob-like creatures.
The goal is to stay alive by consuming the food whilst avoid getting absorbed into the giant creatures. Eating the food will increase the score. The player has 3 lives and once they’re all lost it triggers a game over, which displays the score and prompts the player to restart.
Our first focus is to develop a game that employs generative randomness, used in the creatures. They are symmetrical shapes made up using one equation called the Superformula, discovered by Johan Gielis in 1997. In the game, every time the creatures go off the screen, their appearances (colours, shading colours, resolution) and behaviours (amount of oscillation, speed and direction of rotation) are randomised.
Randomness also applies the positions of the food objects. Altogether, this creates a sense of unpredictability, capturing the player’s attention.
The second focus is on visual aesthetics, which we wanted to explore ways to create a mesmerising piece of art game with a futuristic and mystical style. This is shown through the alluring shading of the creatures, glowy food spheres and a dark background that subtly changes colours, in contrast with the tiny particles to create a nice gleaming effect.
We believe our program can appeal to both tech-savvy people, who will take interest in the super formula and its implementation, as well as people who aren’t familiar with programming concepts, who will simply enjoy the visual experience. In particular, our program could easily appeal to a very young audience (ages 2 to 10), given the simplicity of the controls, and the colorful visual appeal.
Our main sources of inspirations are flOw and Spore, in terms of the game mechanic, aesthetics style, top-down view with 2D gameplay in a 3D environment.
The generative randomness are inspired from the idea of things changing when the player isn’t looking (Creepy Watson).
(More on this can be found on our Creative research)
Understanding the Superformula:
The Superformula is a geometrical equation that can create many organic and natural shapes with 6 parameters: a, b, m, n1, n2, n3. It is made up of the Pythagoras theorem with exponent ‘n’ instead of the squared exponent. Below is our notes to understanding the formula on a low-level:
(More on background research: Technical Research)
We started off by working on the creatures, using Superformula3D by Kamen Dimitrov. In his code, the shape is a plain white mesh and does not move, therefore we needed to customise this. Having the GUI allowed us to adjust the values of the 6 parameters, making it much easier to understand how the formula and the mesh work together to create beautiful symmetrical shapes:
- Making the movement and polishing the appearance:
We wanted to create something similar to the pulsating movement in Cindermedusae, which we found very natural and mesmerising. To do this, we added sine and cosine to each of the 6 parameters, for each vertex of the mesh. We then used booleans as buttons to turn each parameter on and off, allowing us to see how adding oscillations to each parameter would affect the movement of shape as a whole:
(Note: n1value is not used because it defines number of angles (m))
Once we’re happy happy and satisfied with the movement, the next step is to turn one creature into a family of creatures, each having its own unique appearance and behaviour.
First, for each creature to have a unique behaviour, we randomised the oscillations of the 6 parameters (amount and speed of oscillation), direction and speed of rotation, number of vertices (or resolution) and size.
Second, we added the shading using ofLight (sets point light, ambient, diffuse and specular colour) and ofMaterial (sets the brightness, saturation and shininess) to create a glossy and polished look. All of these, together with the mesh colour, are randomised with ofRandom(). Doing this has made the creatures look more realistic and captivating!
THE FISH-LIKE LIFE FORM:
Initially, we wanted to create an object that looks and moves like a realistic fish. In the first version, we created the fish using many ofConePrimitive objects added together, which we were not satisfied with:
We realised it would require a lot of work to create a realistic 3D object for beginners like us, let alone creating the movement for it. At this point, we were suggested to look into Karl Sims‘ swimming behaviour in Evolving Virtual Creatures. This behaviour is achieved by ‘turning off gravity and adding the viscous water resistance effect,’ as explained by Karl in his paper. However, we did not wish to employ the same method, but rather used this as an inspiration for our life-form.
Since the life-form is controlled by the player, it needs to follow the mouse position. We therefore looked into Processing example follow3. It is a simple 2D snake that does not move but only rotates itself towards the mouse position.
Below video is the our recreation of follow3 in 3D:
This, combined with Karl Sims’ swimming behaviour, were exactly what we need. Finally, we managed to create a 3D life-form which follows the mouse and moves like Karl Sims-inspired behaviour.
We wanted the food to have a glowy / bloomy effect to make them stand out against the dark shader background, and thus making it easier for the player to look for the food. We stumbled upon a beautiful glowy mesh made by a digital artist called Reza Ali, which looks very close to what we wanted:
The final food object (below) is made up of many nodes, drawn in an orientation of a 3D sphere, where each node is texture mapped with a blurry white dot image to create the bloom effect:
THE ABSTRACT ENVIRONMENT:
This is created with little shiny particles floating freely around, on top of a dark shader background that smoothly changes colours as the player moves to create the illusion of movement:
- The background: The background simply uses the RGB values of the background color and assigns increments or decrements to them. Then once one of the values reaches the minimum or the maximum, give it a new random increment or decrement accordingly. The background was subsequently changed from using the entire RGB colour space, to a much darker tone, to add contrast with the glowing particles.
- Particles: Adapted from openFrameworks’ billboardExample, the particles are little floating nodes that move randomly using noise. Together with the smooth 360 camera rotation, they helped to create a fluid and floaty environment.
We loved the look and feel of the Superformula creatures and we thought that they could be the focus for the visual aesthetics of the game. Looking at them moving, we felt that there is an element of something rough and strong, set out by restrained forms of the skeletal mesh, yet at the same time is so free, expressive and magical in the way they look and move. Altogether, they have a harmonious balance between the two elements, and this has therefore helped us to decide the visual style of the game as futuristic and mystical.
MUSIC AND SOUND EFFECTS:
To accompany with the chosen art style, we chose an ambient background music and ambient musical notes as the sound effects. These further enhanced the gameplay experience by add a thrilling and adventurous atmosphere to the game.
Problems & How We Solved Them
- Adding movement and constraining sizes: We thought that creating movement would have the same approach to adding colours (that is, looping through each vertex and adding oscillation). However this was the wrong approach as this changed the positions of the vertices, and thus distorted the mesh. To solve this we added oscillation straight to the 6 parameters instead of each vertex, with booleans to turn each parameter on and off consecutively to see how oscillation of each parameter would affect the movement of the shape as a whole. This way, we are actually using the super formula parameters, not distorting the pre-made shape.
Doing this also allowed us to see exactly which parameter(s) makes the creatures big (since the values are random), which we noticed they are n3value and n4value, so we reduced the amplitude and vertical shift for these 2 parameters to prevent the creatures getting too big.
- Speed vs Resolution tradeoff: Setting a high resolution would make the creatures look better but would significantly slows down the game and causes lagging, therefore we had to reduce it a bit to ensure a smooth gameplay experience.
- Randomising mesh colour: Unfortunately, we were never able to randomise the colour of the creatures themselves when they go off screen. So at the moment, the 3 creatures change their shapes, movements and lighting colours, but their mesh colour stays the same.
- Lighting limit: In the game, each of the 3 creatures has a specular light and a point light. We wanted to have around 5-7 creatures to make the game more exciting, however, this caused the light on the life-form to crash, and we also got this error:
[error] ofLight: setup(): couldn't get active GL light, maximum number of 8 reached
According to this page, openGL introduced a fixed-function pipeline of only 8 lights allowed per scene. Also, removing the life-form light significantly reduces its appearance and does not match with the environment of the game. Therefore, to keep the player light, we could only have 3 creatures.
with lighting (creates nice scales)
- Moving vertices and bloom effect: To start off with, we used the same approach that created the creatures: create a sphere mesh, loop through all the vertices and add oscillation to each vertex. However we could not get all the vertices to move (due to how the sphere mesh was made) and also struggled to find a way to create the bloom effect.
At this point we found the openframeworks example called pointsAsTexture which is exactly what we needed. Therefore we decided to use this example and tweaked it our own way.
- Initially, before we had the glowing particles, we wanted to make the background more appealing, as well as give the player a more obvious sense of direction when its moving. So we thought of making these massive gradient circles, which could act as ‘zones’ which the player can move into, which are of a fixed colour, as opposed to the changing background. Our vision for this was strong, and we spent a long time working on it. Initially we used many ofCircles drawn from the inside to the outside, with decreasing opacity. However, this was extremely laggy, due to the numerous sine and cosine calculations that need to happen every frame.This led us to having to learn about openGL: we created the circles by using the triangle-circle method, which was much more efficient, and allowed for even further possibilities in modifying the colour and look in interesting ways.However, this seemed to cause problems with the super formula shapes, that no longer appeared 3D but turned ‘flat’.
This led us to decide to let go of those shaders, and we didn’t think they were absolutely necessary for the look of the game anyway.
We have managed to create a basic exploration game that works well without any bugs or lagging. Most of our goals were completed within schedule, with only the parallax background and random creatures’ mesh colours left out.
Our 2 focuses were also successfully implemented, especially the visual aesthetics, as we have worked hard to make it look as good as possible within our abilities.
On the other hand, we had to remove lots of ideas, such as using AI, genetic algorithm, voice input, parallax background as they are beyond the scope of our abilities. Also, we were unable to carry out user testing on kids as we do not know any, therefore we only tested on adults. However, this would have been very interesting, as we are quite convinced of its possible appeal to this audience.
- flOw: http://www.jenovachen.com/flowingames/flowing.htm
- Spore: https://youtu.be/WoP5thatpr4
- MURMUR by Reza Ali: https://www.instagram.com/p/BAueZPpneiT/?taken-by=syedrezaali
- Evolving Virtual Creatures by Karl Sims: http://www.karlsims.com/evolved-virtual-creatures.html
- https://youtu.be/JBgG_VSP7f8 (swimming behaviour at 0:26)
- Cindermedusae by Marcin Ignac: http://marcinignac.com/projects/cindermedusae/
- Creepy Watson: https://www.youtube.com/watch?v=13YlEPwOfmk
- Supershapes by Paul Bourke: http://paulbourke.net/geometry/supershape/
openFrameworks documentation and tutorials:
- ofEasyCam: http://openframeworks.cc/documentation/3d/ofEasyCam/
- ofMesh: http://openframeworks.cc/ofBook/chapters/generativemesh.html#basicsworkingwithofmesh
- ofShader: http://openframeworks.cc/ofBook/chapters/shaders.html
(All free for commercial use)
- Superformula3D by Kamen Dimitrov: http://www.kamend.com/2013/12/superformula-3d/
- openFrameworks and Processing examples:
- follow3 (Processing > Examples > Interaction)
- ELIXIA font: https://uispace.net/1069-elixia-font
- Sound effects by Akira
Gitlab Repo & Extras
All Youtube videos above can be found in this playlist.
Also, take a look at our Work Diary for weekly documentation and progress 🙂
Growth of Dependency
‘Growth of Dependency’ by Sapphira Abayomi.
Growing up I found my passion for art through painting and very palpable materials. As my art practice developed I transitioned toward experimenting with different digital mediums. However more recently my work has been driven by the notion of integrating the untouchable and tangible. ‘Growth of Dependency’ is a painting which uses a generative system inspired by organic movement to explore the relationship between the virtual and the tangible. The synthesis between both elements is what completes the piece. ‘Growth of Dependency’ is representative of a commensal symbiotic relationship whereby the physical brushstrokes do not need the virtual element to exist. However the same can not be said for the virtual growth forms, those growths can only begin their life and growth forms once they have a physical element that can act as their host. ‘Growth of Dependency’ is a collaboration between myself and the generative system, in which the system uses my outputs as its way to thrive.
From my personal experience I have encountered a naïve understanding of what my friends and family imagine to be computational art, they don’t quite grasp what it is but I also pin that down to their lack of exposure to this art form.I often get asked what exactly it is that I am studying and after announcing the long-winded title of my course the reply is often along the lines of “oh so you do a lot of graphic design” or “do you create screen savers”. Even after fully explaining what the course is people still cannot fully place a finger on what the art would emerge into. However, that is exactly the beauty of computational art it can manifest itself into any form you can imagine. What I want to achieve with this piece is to show that computational art does not have to be very mechanical in nature; it can manifest itself into any form the artist can dream of. For example there are numerous computational art pieces that are performance, sculptural, or instillations based, and the list goes on. What you can do with fine art you can achieve with computational art and more. I want the audience to look at ‘Growth of Dependency’ and be amazed with its beauty and to be intrigued by this work of art having been made by a collaboration between a computer system and an artist. I want people to see how digital art can fit in the contemporary art world.
My intended outcome for ‘Growth of dependency’ was for it to develop into a series of static physical paintings. This series was inspired by the notion of mixing the virtual and the tangible to produce visuals inspired by organic growths which would then be captured in a physical form. I had intended for the generative system to act as my artistic brain, for the system to decide exactly what was to be painted, the composition, colour etc. and myself to act as the vessel by which the physical painting was produced. Almost as if I was the computers body. I started by running some basic preliminary test programs which setup this type of relationship between myself and the generative system. After testing out this kind of dynamic, I discovered how unsatisfying and mundane this was for myself. I really felt like robot, just completing what I was told to do. The picture below is the outcome of this test.
This type of relationship wasn’t stimulating enough for me. I needed to find a way for the system and myself to have more of an equal collaborative role in the final outcome. This is where the current idea developed from. The way I decided to achieve this new dynamic between myself and the system was to have the system create it’s growth patterns according to my brushstrokes on the canvas. i.e when I paint thick brushstrokes on the canvas the generative system would then start it’s growth process originating from my brushstrokes.
Generative art is a frequently reoccurring topic within the digital art community, but what exactly does this term refer to? The two determining elements of generative art are firstly the artwork must make use of a type of system, generally a machine or computational system. Secondly that system has a certain degree of autonomy to it. A definition that I find quite clear when defining the art term is by Philip Galanter who explains that “Generative art refers to any art practice where the artist uses a system, such as a set of natural language rules, a computer program, a machine, or other procedural invention, which is set into motion with some degree of autonomy contributing to or resulting in a completed work of art”.
A generative artist has control over some element in the work but they do not have complete creative control. The autonomous system has a portion of the creative license in how the outcome of the work takes form. This is the beauty of generative art. The algorithms that the artists create use principles of cause and effect and numerous parameter spaces to determine the various aesthetic outcomes. I say ‘various outcomes’ because the outcome is always unique due to the computer having an element of control, therefore there are a variety of different outcomes create.
A generative artist can be understood as a musical conductor and the generative system is their orchestra. They have an influence and an input into the work but they don’t have control over the outcome of what is generated after their input has been made. The orchestra or generative system decides that outcome.
In my art practice, I want to look at using a generative system as a part of the process to create the final pieces, rather than the system itself being the final piece shown. Ideally I would like to take the various outcomes produced by the generative system and then based on them, create a tangible version: a painting or sculpture. Consequently the final product will be a material piece.
I have done some research around some artists who have implemented this style of working using generative art within their practice. Mauris Watz for example is a digital artist who works a lot with generative systems. Throughout his artistic practice he began to move away from using the screen as the default output. He began exploring ways to manifest his generative systems into tangible objects. His core interests in representations within his art are the intersections between organic and mechanical forms. He represents this by trying to create organic movements out of code, which is completely inorganic. He uses a lot of geometric structures to try and achieve this. The relationship his art has with the computer is very inter-reliant. His art relies heavily on the unpredictability created by the computer. He is very interested in finding the interaction between these algorithms he creates and parameter spaces he can use to create a system with this unpredictability. To be able to have these software’s that create things that humans could not fascinates him and drives his practice. Although computers play such an important role in his process, Watz grew tired of using computers to represent the entirety of the pieces. He explains that with generative art often the screen becomes your material; the outcomes that are produced are displayed on screens of all different shapes and size. However using screens to display work fosters so many problems on its own. The quality of the displayed work gets reduced by an incredible margin. This exact problem is what sparked Watz’s practice to manifest itself into tangible material. He shifted away from screens and tried to find ways that he could represent his work in a physical space.
Watz started exploring many different palpable ways to represent the intangible outputs of the software. He used 3D printing, light fixtures, laser drawings and even wall drawings. What Watz began to notice by materializing these productions was that the relationship between the art and the viewer would change as well. This relationship often became a lot more intense with some of the physical objects he portrayed. He especially noticed this with some of his first 3D printed objects. As virtual objects there was nothing special about the 3D models but when they materialized there was a transformation. They acquired more of a depth and allowed there to be a sort of interaction with the viewer examining all the details and angles etc. In some cases I feel like the virtual representation can alienate the audience. A material representation starts a very different dialog between the work and viewer, which Watz has personally discovered in his own practice.
A further example is the Generative Jigsaw, which is a collaboration project developed by a company called Nervous systems and artist Jonathan Mccabe. The final results were a series of unique jigsaw puzzles. Nervous System and Mccabe shared an equal interest around the different patterns that occur in nature. This translated seamlessly into the project, which was heavily influenced by complex growth patterns and fossilized cephalopod shells.
I think this project is particularly interesting because it uses two separate generative systems to produce the two different aspects of the project. One system was used to generate the puzzle piece pattern, the second system for the artwork that fits on top of the puzzles.
To produce the system for the puzzle pieces, the Nervous system reverse engineered crystal growth processes that they then translated into code. Through changing the parameter space they created a myriad of pattern variations.
For the artwork Mccabe used a generative system that married three processing in order to produce the artwork. The first process is a modified idea based on Alan Turing’s chemistry paper in 1952 on “ The chemical basis of morphogenesis.” Morphogenesis refers to the appearance of an organisms body plan, for example the patterns on a cheetah or zebra. The paper discussed the theory that the spontaneous pattern formations found on organisms are a result of a diffusion and reaction of various substances. They in turn then either inhibit or activate one another. Moreover the rate at which the substances move through the tissues also varies. The other process Mccabe used is fluid flow, which combines coloured dots, and strips together forming sharp edges, there is also a dispersal process that yields patterns of movement and colour in the fluid. The last process essentially averages the values of colour and movement around the circle with in each instance; the slowly changing colour field is then recorded within each instant. At this point the selection process of the most visually appealing begins. Both of the final products from the two generative systems are then put together to create the one of a kind generative jigsaw.
Much like Nervous System I am very interested in implementing elements of nature into the work, like I previously stated there will be an over all organic makeup to the series.
Design and Build Process
The above picture is when I started testing a simple generative system made on processing to try and develop my idea on the relationship between myself and the computer. By exploring this I discovered that I did not what the computer to completely control the outcome of the painting, I wanted there to be a mixture of input from myself and the computer to make the final painting.
I wanted to develop a way for the generative system to take what I had painted and generate it’s own painting based on that. I started testing the infrared (IR) camera on the Kinect to see if I could find a way for the generative system to tell the difference between what I have painted and what it was projecting. I searched long and hard for a IR reflective or absorbent material to achieve this, however that was considerably hard to find. I struggled to find a paint or fabric which fit my criteria. Eventually in my research I discovered that Titanium dioxide was on the IR spectrum, and that it was an ingredient in titanium white. I decided to test different kinds of paints which I already had ( as seen in the above picture) , to my surprise black and white worked very well with the IR lens. I really liked the idea of the colour palette for the paint being only black and which and the colour was only introduced by the projection. However as the project developed I no longer had a use for the IR camera to detect which was my brushstroke.
My next step was to start getting openCV up and running and work with the background subtracting and contour finding that I would need to detect when there has been a change to the canvas ( the above picture). Once I had this working I started working on the polylines that I would later need to calculate where the generative growths would appear from.
I now needed to work out how to calculate the normals of the polylines and rotate the growths according to the normal angle. I have often found it a lot easier to work in processing and develop what I need in a processing sketch first, which is what I did with the normals. However this was a mistake on my part, with some of the structural difference in openframeworks, porting the code became quite a nightmare. I think at this stage I was just confusing myself with trying to port the code. In the end because all the elements I needed were there but it was just a few syntax errors which were giving me the last few issues.
From the beginning I knew that I wanted the growths to have an organic movement to them. I wanted them to look like they were really growing organically. To do this I started playing around with noise to try and achieve that type of movement.
The above screen shot is taken when testing the growths starting point from the circle trail created by the mouse movement. These trails are a place holder for the future paintbrush strokes.
A test of the aesthetic without the mouse trail.
I now started to combine the growth aesthetic with the openCv code. I started running into some issues here. For the growths to be produced the background could not refresh itself, however for the for the openCv to detect when a new brush stroke had been created it needed to constantly update the background causing the growths to turn into floating dots. The solution for this was to organise the code into modes so that the different elements such as, background subtraction, polylines and growths would happen at the appropriate time and not all at once.
Making the modes made a great difference to the appearance of the program. I could finally draw different brushstrokes on a white board and have the growths generate according to that. Now that all the elements were slowly coming together I wanted to sort out the actual colour aesthetic of the growths. A cohesive colour palette is really important to me when I am actually painting and there was no reason that it should be any different for this type of painting. However picking colours that merge well together is very different on a computer than on a mixing plate, the colour spaces are perceptually different. The solution for this was to take an image which had a cohesive colour palette and have the generative system base it’s own colours off of that image. I also like this because it allowed me to change the colour palette very easily for different paintings if this ever developed into a series.
My next step was to try and calibrate the camera and the projector so that what was painted matched up perfectly with what was going to be projected. I started by testing it on a a big white board, is was the easiest option for testing without making any permanent marks on my canvas. This stage of the process was something that I greatly overlooked. As always you learn from experience and with no previous experience with a project like this I did not realise how much time I needed for this stage of calibration. At this stage I admit to being very naive with my expectations of how the projection and painting would combine together as one piece. The research that I was doing suggested that I could use projection mapping to calibrate everything. However I also had another problem where the kinect itself didn’t seem to be calibrated properly and it was causing the polylines to be displaced by quite a bit. This problem was rapidly eating into my exhibition set up time and the painting process was held back because of this issue. I had to think of a plan B, a plan that was sure to result in a finished piece because time was running out and at this rate I wouldn’t have a piece in the exhibition. My plan B was to create a diptych. I got an identical canvas to the one I already had. On one canvas would be the physical painting and on the second canvas would be the output produced by the generative system based on the painting. Plan B solved many issues I was having because It meant that I could definitely stay clear of running any OpenCV during the exhibition and everything could be completely finished and prerecorded before the exhibition with very little worry about the piece during the exhibition.
Meanwhile I had been sketching different compositions for the physical painting. I knew that I wanted it to be very shape and line based, however this is a very different style of painting than what I’m used to producing so I wasn’t sure how successful the style would be. I was drawing inspiration from both Kandinsky and Paul Klee and there use of lines and shapes.
This is how the painting turned out in the end as I painted it I recorded the the computers output so that I could create the projection for the second canvas. which you can see in the photo below. I had a couple of issues during the painting process the generative system could only pick up objects of a certain size, as a result during the recording process I had to change some of the things I had already painted. Eventually I got the footage that I wanted which I then turned into a video that looping. (You can see this video in the executables section bellow.) My reasoning for this was because it was imperative for me that the audience get a glimpse of the process that the computer output goes through. Once the growth lines occur is starts to obscure the identifiable relationship between the original painting and the generative process. The beginning of the video allows the audience to really grasp the relationship between the two canvases. Another reasoning for this is because I really wanted to encapsulate that growth process in the projection, I didn’t want to project a static image, the organic movement was one of the key elements I wanted to capture in this project and even with plan B, I still had to stay true to what I really wanted to achieve with this project.
In the above picture you can see how the diptych was set up in the gallery space. In my original planning I wanted to box off my section of the gallery with black material, I believed that this would add to my piece because it would isolate the view and really draw their focus into what was happening in this diptych, minus any other distractions happening in the room. It would also further emulate this boxed appearance that was occurring with the canvases. However due to a lack of material I was unable to achieve this presentation aspect. This however was not detrimental to the piece.
A video of the computers output, which was projected on the second canvas.
A video of how the diptych was set up in the gallery.
Over all I believe that the project was very successful. Although there was some bumps in the development, and there were times the problems I ran into clouded my vision for this project. However despite all of that I still achieved the essential elements of what I wanted to produce – organic growth and a painting which explored the relationship between the physical and the virtual . Even though I had to change my original plan of creating one painting with the projection occuring on top of the painting, I believe that my plan B actually worked better for this exhibition because it allowed the audience to really see the difference between my input and the computers output, and the growth process of the generative system. I would still like to further develop the project and test it with the original one canvas plan. However for now I am more than happy with the over all turn out of the piece.
Artist Project / Tree Drawings. (2007). Cabinet, [online] (ssue 28 Bones Winter 2007/08). Available at: http://cabinetmagazine.org/issues/28/knowles.php [Accessed 20 Dec. 2015].
Ball, P. (2012). Turning Patterns. [online] Chemistry World. Available at: http://www.rsc.org/chemistryworld/2012/05/turing-patterns [Accessed 3 Oct. 2015].
Bit forms Gallery, (2011). Recorded Delivery. [online] Available at: http://www.bitforms.com/exhibitions/tim-knowles-recorded-delivery/press-release [Accessed 21 Dec. 2015].
Bohnacker, H., Gross, B., Laub, J., Lazzeroni, C. and Frohling, M. (n.d.). Generative design
Budds, D. (2015). This Code-Generated Architecture Can Only Exist On Paper. [Blog] Co.Design. Available at: http://www.fastcodesign.com/3053214/this-code-generated-architecture-can-only-exist-on-paper [Accessed 23 Dec. 2015].
Doan, A. (2008). TREE DRAWINGS! Tim Knowles Arbor Interpretations. [Blog] Inhabitat. Available at: http://inhabitat.com/tree-drawings-arbor-interpretations-by-tim-knowles/ [Accessed 21 Dec. 2015].
Docs.opencv.org. (2016). Basic Thresholding Operations — OpenCV 22.214.171.124 documentation. [online] Available at: http://docs.opencv.org/2.4/doc/tutorials/imgproc/threshold/threshold.html [Accessed 29 Feb 2016].
Docs.opencv.org. (2016). OpenCV: Background Subtraction. [online] Available at: http://docs.opencv.org/master/db/d5c/tutorial_py_bg_subtraction.html#gsc.tab=0 [Accessed 29 Feb 2016].
Docs.opencv.org. (2016). OpenCV: How to Use Background Subtraction Methods. [online] Available at: http://docs.opencv.org/master/d1/dc5/tutorial_background_subtraction.html#gsc.tab=0 [Accessed 29 Feb 2016].
Docs.opencv.org. (2016). OpenCV: Image Thresholding. [online] Available at: http://docs.opencv.org/master/d7/d4d/tutorial_py_thresholding.html#gsc.tab=0 [Accessed 29 Feb 2016].
Dorin, A., McCabe, J., McCormack, J., Monro, G. and Whitelaw, M. (2012). A framework for understanding generative art. Digital Creativity, 23(3-4), pp.239-259.
DW, (2012). How old is Generative art?. [Blog] Graphic Dimensions. Available at: https://graphicdimensions.wordpress.com/2012/12/18/how-old-is-generative-art/ [Accessed 23 Dec. 2015].
Filippetti, J. (2012). generative jigsaw puzzles by nervous system. [online] designboom | architecture & design magazine. Available at: http://www.designboom.com/design/generative-jigsaw-puzzles-by-nervous-system/ [Accessed 13 Oct. 2015].
Galanter, P. (2004). Generative art is as old as art.
Galanter, P. (2012). Generative art after computers.. 1st ed. [ebook] XV Generative art conference. Available at: http://www.generativeart.com/GA2012/phil.pdf [Accessed 22 Dec. 2015].
Galanter, P. (n.d.). What is Generative Art? Complexity Theory as a Context for Art Theory. 1st ed. [ebook] Available at: http://www.philipgalanter.com/downloads/ga2003_paper.pdf [Accessed 13 Nov. 2015].
Galleries around the world are finally “starting to take digital art seriously”. (2015). [Blog] Dezeen. Available at: http://www.dezeen.com/2015/02/17/movie-galleries-around-world-starting-to-take-digital-art-seriously/ [Accessed 20 Dec. 2015].
Gupta, K. (2013). Color Tracking in OpenFramework. [online] KUNAL GUPTA. Available at: https://dirtydebiandevil.wordpress.com/2013/01/21/color-tracking-in-openframework/ [Accessed 4 Mar 2016].
Kaganskiy, J. (2010). Losing Control: Generative Art and the Art of Letting Go. [Blog] The Creators Project. Available at: http://thecreatorsproject.vice.com/blog/losing-control-generative-art-and-the-art-of-letting-go [Accessed 19 Dec. 2015].
Knowles, T. (n.d.). Tree Drawings. [Generative art, Pen on canvas].
Kurosu, M. (2015). Human-Computer Interaction: Interaction Technologies. Cham: Springer International Publishing.
Levin, P. (2015). Interactive Art & Computational Design.
Lieberman, Z. (2004). ofBook – Foreword. [online] Openframeworks.cc. Available at: http://openframeworks.cc/ofBook/chapters/foreword.html [Accessed 16 Feb. 2016].
MacCormack, J. (2013). Representation and Mimesis in Generative Art: Creating Fifty Sisters. 1st ed. [ebook] Italy: xCoAx. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.348.5670&rep=rep1&type=pdf [Accessed 18 Dec. 2015].
Mallick, S. (2015). OpenCV Threshold ( Python , C++ ) | Learn OpenCV. [online] Learnopencv.com. Available at: http://www.learnopencv.com/opencv-threshold-python-cpp/ [Accessed 4 Mar 2016].
Mccarthy, C. (2015). THE CODE-GENERATED ARCHITECTURAL DRAWINGS OF MIGUEL NÓBREGA. [Blog] Kill Screen. Available at: https://killscreen.com/articles/the-code-generated-architectural-drawings-of-miguel-nobrega/ [Accessed 23 Dec. 2015].
McCormack, J., Bown, O., Dorin, A., McCabe, J., Monro, G. and Whitelaw, M. (2014). Ten Questions Concerning Generative Computer Art. Leonardo, 47(2), pp.135-141.
McDonald, K. (2012). kylemcdonald/ofxCv. [online] GitHub. Available at: https://github.com/kylemcdonald/ofxCv/wiki/Intermediate-Computer-Vision-with-openFrameworks [Accessed 20 Feb 2016].
Mufson, B. (2015). MC Escher Meets Mondrian in Code-Generated Marker Drawings. [Blog] The creators project. Available at: http://thecreatorsproject.vice.com/blog/mc-escher-meets-mondrian-in-code-generated-marker-drawings [Accessed 23 Dec. 2015].
Nóbrega, M. (2015). Possible structures, Potential structures, Plausible spaces.. [Generative art, computer and Plotter machine] Superficie. Online gallery.
Nokami.com, (2016). Generative Art. [online] Available at: http://www.nokami.com/generative-art/ [Accessed 19 Dec. 2015].
Pearson, M. (2011). Generative art. Shelter Island, NY: Manning.
Reis, R. (2001). Careers in art and graphic design. Hauppauge, NY: Barron’s Educational Series.
Research.philips.com, (n.d.). Philips Research – Technologies – Generative Art: protoquadro. [online] Available at: http://www.research.philips.com/technologies/protoquadro/ [Accessed 28 Dec. 2015].
Sahin, O. (2012). Generative & Algorithmic Art LEA Call for Papers. Leonardo Electronic Almanac. [online] Available at: http://www.leoalmanac.org/generative-algorithmic-art-lea-call-for-papers/ [Accessed 21 Dec. 2015].
Scholz, A. (2012). Generative Jigsaw Puzzles – Nervous System toys with a classic enigma. [online] CreativeApplications.Net. Available at: http://www.creativeapplications.net/processing/generative-jigsaw-puzzles-nervous-system/ [Accessed 13 Oct. 2015].
Solaas, L., Watz, M. and Whitelaw, M. (2016). GENERATIVE PRACTICE. THE STATE OF THE ART.
Visnjic, F. (2015). Possible, Plausible, Potential – Drawings of architecture generated by code. [online] CreativeApplications.Net. Available at: http://www.creativeapplications.net/processing/possible-plausible-potential-drawings-of-architecture-generated-by-code/ [Accessed 23 Dec. 2015].
Wang, H. (2014). Painting with Code. How generative art reveals new possibilities in visual expression and design. [Blog] IDEO labs. Available at: https://labs.ideo.com/2014/06/04/painting-with-code/ [Accessed 19 Dec. 2015].
Wassilykandinsky.net. (2016). Wassily Kandinsky – “Composition VIII”, 1923. [online] Available at: http://www.wassilykandinsky.net/work-50.php [Accessed 20 Mar. 2016].
Watz, M. (2010). Closed systems: Generative art and Software Abstraction. 1st ed. [ebook] Available at: http://mariuswatz.com/wp-content/uploads/2012/03/201005-Marius-Watz-Closed-Systems.pdf [Accessed 17 Nov. 2015].
Watz, M. (n.d.). MARIUS WATZ: CODE, ART & AESTHETIC.
www.wikiart.org. (2016). Paul Klee, Late Works.. [online] Available at: http://www.wikiart.org/en/paul-klee [Accessed 20 Mar. 2016].
Lux – Dat & Kevin
We wanted to build a creative tool which encouraged users to explore visual aspects of light painting by adding their own creative flair to a blank digital environment.
We decided to follow through with an idea that surfaced as a result of our creative research albeit with some slight modifications. Our final project idea was to create a digital environment which acted as a canvas for the user to explore aspects of lighting artwork.
The user can navigate through the 3D world with the following controls:
- WASD keys: general movement in world
- QE keys: Move up and down the Y axis
- F key: move quicker when used with WASD & QE keys
- LMB: for drawing
- TAB key: this is used to switch from drawing mode to edit mode and vice versa when wanting interact with the GUI interface
- SPACEBAR: move selected light sphere to player position
Lux was created in openFrameworks and written in C++.
For the research part of the project, the brief outlined that we should concentrate on ‘aspects relating to design and aesthetics’ and to ‘avoid looking…at technical aspects [like] libraries [and] frameworks’. Owing to this, we decided to focus entirely on lighting as an outlet for art, paying particular attention on light painting and light graffiti due to the visual properties they possess.
After an in-depth research on the history of lighting artwork and its process, we highlighted two aspects which we felt could be improved on. The first was how dated the technique was. If we wanted to create a light painting piece right now, we would still be using the same procedure used in 1886 by the founding fathers of light painting, Georges Demeny and Étienne-Jules Marey. The process is still a ‘photographic technique in which exposures are made by moving a…light source while taking a long exposure photograph’.The second problem we found was the possible restrictions light painting holds for someone who wants to explore the art form for the first time. We personally felt that this restriction lay in the cost of the equipment that needed to be used. This was backed up by articles we read online which expressed the importance of using a DSLR camera owing to the full control you have over the shutter speed (which is the duration of exposure), the aperture (which is the opening of the lens which light passes through), and the ISO speed (which controls how much light enters). We felt that this was a costly investment for anyone if their motive was solely to experiment with the capabilities of light painting.
These problems and our interest in the aesthetics of lighting artwork spurred us into wanting to create a tool which allowed people to explore visual aspects of light painting. With this in mind, we decided that our application should offer a strong foundation for people who wanted to explore light painting as a creative outlet but may lack the knowledge or equipment to do so – this was our desired target audience. When accepting this as our target demographic, we understood that difficulties could arise as there was no specific age group or gender we wanted to focus on. This meant that when it came to designing the application, there had to be a medium, as a fourteen-year-old male would have to hold the same interest as a twenty-four-year-old female when using the program.
We were adamant in producing this project on openFrameworks without any additional physical components. By creating it purely through openFrameworks and C++, we felt that we could share it with anyone who had a computer and was interested in exploring the visual properties of lighting artwork in a digital world.
When it came down to our initial prototyping, we were very keen on the theme of light versus darkness. We felt that this would be suitable to the nature of light photography as the brightness of colours used always strike through, creating a huge contrast to the dark environment it is in.
Continuing with this thought process, we began thinking of potential objects we could add into our environment to really hone in on this theme. The first image which popped into our heads was a tree since it connotes the idea of growth and therefore positivity and light. However, a tree also has negative connotations when it is bare, as it alludes to the idea of death and darkness.
We looked into fractals and used Daniel Shiffman’s tutorial to get the foundation of the tree object and then developed it to fit our project needs.
However, after some testing we ultimately had to scrap this idea as the recursion technique used to draw the trees meant that our application became extremely slow when adding more than one fractal tree – this was because the recursion calculation was continuously looping as it was run in the draw section.
Nevertheless, we still intended to keep with this idea of light and darkness but to channel it in a different aspect. We opted instead to have multiple light sources which would light up the dark environment and the objects surrounding it.
This was also the stage where we decided to allow the user to move freely wherever they desired in the 3D world as opposed to a flat terrain with boundaries. We believed that by doing this, we would meet our personal criteria of creating a digital environment which had no restrictions for the user when drawing.
Moving on to the light drawing prototype, we had an idea of what we wanted the output to look like but we did not physically create any sketches. The reasoning behind this was because we found it difficult to draw the line strokes the way we wanted and there were already a lot of resources online that we could base our final design on – Yellowtail by Golan Levin and inkSpace by Zach Lieberman being some of these. Both of these applications had implemented a very natural brush stroke which enhanced the sense of control that the user had when drawing.
We split up the the build into two sections and aimed to complete the tasks in this order:
- The build of the environment and mouse picking
- The functionality of the drawing method
The 3D Environment and mouse picking
This part of the build was the most time consuming. We did expect this to be the case however, as both of us had never created a 3D environment before. Initially, we based our environment off a ofxBullet example as we had already researched into this as a result of our technical research. The benefits of using this addon was that all the collisions and mouse picking could have been easily implemented. However, the limitation to this was that you were restricted to only using ofxBullet objects; unfortunately, these were very simple shapes and would not have given us the flexibility we required for our light drawings. We spent some time implementing the ofxBullet world before realising our mistake but we did take something away which was useful. We discovered that ofxBullet adds a texture to its shapes which means that the usual implementation of lighting would not generate the results you would expect. From learning this, we were able to solve our own issues with lighting later on in our project.
After our blunder with the ofxBullet world, we began to look at building an environment from scratch again. The first week of starting this environment build was tricky for us as we were unsure of the correct terminology to search for to get the correct information to help us. Fortunately, we stumbled upon an article by Marco Alamia which explained in-depth the different spaces required to map a 3D world on a 2D screen. After many more articles and tutorials, we began to understand the logistics of creating a 3D environment but we were still unable to successfully implement theory into code. At this stage, we were becoming increasingly worried that we would remain behind on our personal schedule.
Fortunately, we were able to progress after some help from Simon who had constructed a 3D environment as a result of showing us mouse picking. This was extremely helpful as we had a copy of code which we could use as reference to understand the online tutorials that we had previously read. For instance, we were aware that we had to create a frustum which contained the near and far plane but we did not understand how to put this into code. We realised after looking at Simon’s code that this was a simple fix as ofCamera deals with all of this behind the scenes. There were many other instances where we overcomplicated the code when there was no need to. We felt that this was a recurring problem we faced throughout the project which affected our progression.
After successfully building the 3D environment, we moved on to populating our world with shapes like spheres and boxes. Our decision in doing this was because we felt the world was not as encouraging as we would have liked for the user to start drawing. This was also the stage where we attempted to implement the mouse picking code which Simon gave us. Instead of just copying Simon’s code without understanding the ray, we attempted our own implementation of mouse picking and were able to successfully re-create the ray from scratch without using the ofCamera and ofNode built in functions like screenToWorld(). This was our code:
Unfortunately, we were unsure with how to use the ray in mouse picking. We felt that the problem lay in the fact that we were comparing our world coordinates with our normalised ray values which were incompatible with each other. We spent a lot of time trying to work out what we were doing wrong but in the end, we had to move on from mouse picking and go with a simpler interaction.
The drawing methods and objects in our world
After the completion of the environment, we started to tackle the drawing methods in our application. We found a drawing tutorial to base off of from the openFrameworks page and followed the steps to create interesting graphics which could impersonate the lighting visuals we wanted.
These tutorials we followed used ofPolyline objects to draw points onto the screen; we fed in the world coordinates into the draw() parameter of the ofPolyline object which was how we were able to draw accurately in our 3D world.
We realised that this on its own was quite bland, so we implemented a GUI using ofxGui so the user could select different brushes to draw with. We also added lighting which the user could change to make the environment more vibrant and exciting.
Our original goal was to create a tool which allowed users to explore visual aspects of light painting. Keeping this in mind, we feel that we were able to meet this brief to a certain degree. For instance, our light cluster stroke certainly emulates a more modern take on light painting whereas our standard line stroke takes our users back to the beginning where a more primitive approach was used to reflect the first light painting piece by Demeny and Marey. However, we were unable to truly represent light painting in the way we wanted to. From the very beginning of the project, we were motivated to create a digital environment because we felt that it could offer a different perspective to light painting that the normal creation process would not be able to provide. The bubble stroke was one way of attempting this as we wanted to add a different and interesting perspective to the creative process.
One positive we can take from this is that our knowledge in working in a 3D environment has improved greatly. We now understand the process in creating a 3D environment from scratch, the differences between world coordinates and screen coordinates and the importance of understanding the distinction between the two, and also the reasoning for transformation matrices. Moreover, even though our attempt at mouse picking was unsuccessful, we were able to code the ray from scratch which highlights our understanding and development in this field.
Overall, we believe that the project as a light painting creative tool was partly successful. It does represent the visual aspect of light painting through the light cluster stroke and we have been able to produce a digital environment where the user can roam freely which were our original goals. However, we understand that we could not represent the full capabilities that light painting has to offer. We could have given a better representation of different aspects which could have then provided greater control for the user to create new and unique visuals. What made our progress stagnant was the amount of time we spent understanding and implementing perspective projection and mouse picking. These were challenging concepts to become familiar with but we feel that now, if a similar task was given to us in the future, we would be able to overcome the difficulties we faced this time around as we have a greater understanding of its implementation. This would then allow us to spend more time focusing on the visuals of our light drawing.
Images used from external sources:
Articles/Tutorials used for research:
Neuromersion – An Immersive Neurofeedback Experience
What if our technology was not merely for entertainment, or communication, or news? What if it played more of a role in our lives than facilitating work or study? What if our purpose in using technology extended beyond making us effective and efficient at work? Neuromersion is an audio-visual experience that aims to fill a role more than our technology currently plays.
Neuromersion uses neurofeedback, which is a form of biofeedback, that trains and conditions neural patterns within the brain. Explained at its simplest, the brain operates at different frequencies of neural oscillations, also known as brain waves. Each of the frequency bands correlate to a different state of cognition. For example, high levels of alpha brain waves, 8–13 Hz, often represent deep mental relaxation. Other states of mind, moods, and behaviours also have corresponding changes in brain waves.
If used regularly, neurofeedback could make our screen time far more beneficial and healthy. Neuromersion is a meditative neurofeedback experience, that allows the user to enter a state of deep meditation. We achieve this using a simple form of neurofeedback with an EEG headset. The headset gets the brainwave data from the user, which is given to the Neuromersion software. It is within the software that the feedback is created. The software monitors the user’s brainwaves, specifically the meditation levels, whilst they use the program. All the user has to do is look at the screen, and listen to the headphones. As the users meditation levels change, the audio-visuals change, for example, if the user becomes distracted, the brightness of the visuals could change. The subconscious brain realises that in order to increase the brightness of the screen, it must begin to increase certain brain wave frequencies, therefore increasing concentration.
When brainstorming ideas, we realised we both shared an interest in altered states of mind. There were many areas which we had originally discussed, including virtual reality, hypnosis, and Chaldni plates, amongst other things. Our initial exposure to Neurofeedback came by way of The Joe Rogan Experience podcast #629, with Andrew Hill, PhD. In this podcast, Andrew Hill discusses his work with neurotherapy, nootropics and his work at the Alternatives Brain Institute.
Hill explains his use of neurofeedback in treating addiction, ADHD, anxiety, depression and other mental illnesses. This immediately piqued our interest, as it offered an ability to control and modify the brain without use of medication, in a relatively easy, lasting and potentially cost effective way. Addiction and mental illnesses are both interesting subjects, with their treatments often also being controversial. Both affect a huge number of the population, with varying causes, though both are still generally considered medically incurable. The pharmaceutical industry offer treatments for various mental illnesses, including addiction, though these treatments are not intended to cure the problem, they merely prevent the symptoms at best, often not entirely effectively and at the expense of other side-effects. The effectiveness of the pharmaceutical industry is questionable, particularly when you consider their business model, namely that the demand for their product ends when the buyer is cured, though this topic quickly leads to conspiracy theories and moral questions. Regardless, neurofeedback would seem to be a promising practice for essentially curing or training the brain, which could be invaluable to societal and individual wellbeing. The podcast with Andrew Hill was very informative in explaining how neurofeedback works, and how patients are treated, and we also did additional research on how it is used . Not only did we find neurofeedback intriguing because of our interest in neurological change, but also because we felt morally inclined to pursue something that could hold long term benefits for many.
As this interest developed, an interesting connection between areas arose; mental illness could potentially be treated by technology, using neurofeedback, though mental illness can often be caused by technology and our relationship with technology and social media. My personal experiences with mental illness have been inseparably connected with my relationship with technology, specifically social media. I found that my levels of depression and anxiety were proportional with the time I spent using social media, to the extent that when I stopped using social media altogether for a short time, my mental well being improved drastically. We decided to look into addiction, because we felt that this was a significant part of most people’s relationships with technology, and also because this would be something we would want to help with neurofeedback.
This video extract from The Culture High features Dr Gabor Mate, who is an author and specialist on addiction. Although the documentary this clip is taken from is specifically about marijuana, the principles of addiction hold true for all areas.
One of the things we found most interesting about this video is where he asks, “What did it do for you?”. This is an interesting way of looking at substance or behavioural addiction, because we typically don’t want to acknowledge any sort of positives to the addiction, or appear to condone it. However, I think in order to understand even our mildly addictive behaviours, we have to understand why it is we enjoy that behaviour. We decided to research this by producing a small, five question survey about our relationships with technology. We asked 100 people ranging in age from 14 – 80.
The results were not entirely surprising, and even though it is a short survey, the results were very revealing. The first three questions show that a majority of people own smartphones and laptops, with 77% of people spending over 3 hours of screen time a day, and also that a large majority of people use Facebook. From these results we can tell that most people use social media, for several hours a day, using portable devices such as smartphones and laptops. One piece of feedback we consistently got was that people were surprised at how much time they spent on these devices when they really thought about it.
Questions four and five are much more interesting, and deal with why people use technology and how it affects them. The top 3 needs for using technology were for fun, to stay in touch with friends, and for work. These first two reasons could generally be considered recreational, and not entirely necessary. The most common thing that people felt from technology was connected. More than half felt informed, as well as distracted. Just less than half feel happy using technology, with more people feeling addicted, anxious and depressed, than focussed and fulfilled. There is something fascinating and strange about the way we currently interact with our technology. One thing we immediately noticed was that people felt connected, which is something that comes up frequently in discussions about addiction. Gabor Mate talks about addictions as ways that people try to solve or remedy the true and deeper problem, with the addiction being a symptom of the problem, not the problem itself. What if our dependency on technology is just a symptom of our deeper problems? We seek technological platforms that allow us to feel connected. If this truly is the case, then our current model is not solving the problem, it’s just creating a supply and demand, where the demand can never fully be satisfied.
What if neurofeedback could be implemented in our devices, allowing us to train and condition our brains whilst we use them, no matter what we’re doing? Results show that most people have devices which could use neurofeedback, and we are clearly spending enough time using them for the treatment to be effective. Imagine a time when EEG headsets are as widely available, and perhaps implemented in our headphones, and where we are constantly improving our brain health when using our phones, or our laptops. Rather than wasting time on our phones checking Facebook during the commute work, we would be conditioning our brains, while we connect with those we love. We could be training our brain to be more focussed, attentive and creative, whilst we write school work, or quarterly reports. This has the potential to negate a lot of the negatives we are currently experiencing with technology. Not only would this make our screen time more effective, it would most likely reduce our need for screen time in general.
This is clearly a big idea, without a doubt larger than the scope of this project. This has fairly dramatic implications, though it could be considered idealistic, unrealistic and impractical at this point, especially for us. For this reason, we decided to scale this idea down, using this project as somewhat of an experiment in the way of testing consumer, recreational neurofeedback. Based on the fact that most people said that they used technology for fun, we decided we wanted to make a simple and fun experience that would allow people to feel the effects of neurofeedback.
Considering this project is the first step towards a much bigger idea, we did not want a limited target audience. If we were creating our utopian neurofeedback device, the target market would be all users of consumer technology, with no specific age, gender, ethnicity, or occupation. We decided that a more artistic approach would be the most consumer accessible option. Although we are not aiming our project at a specific demographic, the sort of people that would be interested in a recreational, artistic, neurofeedback experience are most likely people similar to ourselves, namely, students, academics or people involved in experimental technological innovation. This shouldn’t affect our design process or final outcome though, as we still want this product to be widely usable by all people, and above all, enjoyable and beneficial.
Our design process started out relatively simple because our idea didn’t seem too complicated. The two main parts of this project are; the brain data from the EEG headset, and the feedback, in the form of visuals and audio. As Andrew Hill talks about at 56 minutes into the podcast, the neurofeedback can be performed with essentially any stimulus or feedback, the important thing is that the stimulus is yoked to the changing parameter in the brain. Therefore, the goal for our first prototype was to get meaningful data from an EEG, and map the data to a simple image. Our basic milestone plan was:
- Get data from an EEG
- Map data to an image
- Develop visuals
- Develop audio
- Optimize data
- Beta testing
- Final design
We began the build part of this project by acquiring an EEG headset. We first got a hold of a Neurosky Mindset, which is a low cost bluetooth EEG headset. We didn’t have very good experiences with this headset. This headset is more intended for use with their own software, and even though they do have developers tool available, it is not up to date, and far from self-explanatory. We knew we would using OpenFrameworks for this project, so we decided to try to find ofxaddons that could interface with the headset. As it turns out, Neurosky products use a processor chip called ThinkGear, which is where the output values are calculated.
Ideally, we should have been able to connect the headset to the computer with bluetooth, and get the brainwave data with an add-on. Unfortunately, the most recent add-ons for the ThinkGear chip were 3 years old, which meant that there were compatibility issues with OpenFrameworks. We spent far too long trying to get addons to work. There were two addons that we spent a large amount of time working on:
There were so many issues that we tried to overcome with these addons, but no matter which problem we solved, there was always something else. We weren’t being very effective with our time, because essentially we were trying to solve a problem that was beyond our skill level, and not doing anything else.
After weeks of getting nowhere with the ThinkGear addons, we began working on the audio-visuals. Our original visual research led us to geometric patterns, such as Chladni plates and kaleidoscopic patterns. We developed some basic geometric pattern sketches in Processing. We worked with ofxMaxim for the audio, synthesising two pure sine waves and system to modify them according to some parameters, such as brain data, attention, meditation and high alpha wave, along with a defined binaural beats frequency.
Finally, we got to make things to work, after having found a way to fix and update probably the oldest, yet most advanced, addon by Trent Brooks: https://github.com/trentbrooks/BrainWaveOSC. It gives the possibility to establish a connection with the MindWave Headset both using the bundle driver by Neurosky and “manually” parsing the serial data stream of the dongle. Since the bundle we could find isn’t for 64-bit architecture, we settled with the other communication method.
Link to the repository: http://gitlab.doc.gold.ac.uk/eheat001/Neuromersion.git
Link to the OS X executable: http://igor.gold.ac.uk/~dmanc001/NeuromersionApp.zip
After some tests, we tried to tweak the feedback system to make it more reactive and fine tuned to achieve our aims. Then we asked some friends of ours to try it for at least half an hour, that seems to be the minimum time for the brain to respond to the binaural beats and visual neurofeedback stimuli. The comments we received were considerably satisfying and testers were amazed at experiencing a positive change in their mood they weren’t expecting. Some further work can be surely done in order to improve the immersive aspect of our application. For example we would like to implement visuals using an Oculus Rift or similar technologies. Nevertheless, we are rather excited about the results we could obtain and happy that our work was well appreciated by who could try it.
 http://www.tandfonline.com/doi/abs/10.1080/10874208.2010.545760  http://eeg.sagepub.com/content/40/3/180.short
Wobble – By Johan & Cormac
We have created an environmental synthesizer which scans its location and interprets light and topographical information to produce sound. The device, named Wobble, is designed with the intent to not only help children understand sound and music making on an abstract level but to also give them the opportunity to modify the device for further experimentation. Wobble is built on the linux platform running on a Raspberry Pi with an easy to use and understand breakout breadboard. Wobble is powered by a rechargeable battery and rechargeable speaker. With all the hardware built into the device it is totally wireless and can be used in any environment to make a wide array of synthesized sound.
The software running on the Raspberry Pi is all written in C++ utilising three libraries which form the bedrock on which the architecture of the program is built on. The first and most important of these libraries is WiringPi, which allows access to the GPIO pins of a Raspberry Pi with C++. The second is Maximillian, a C++ audio and Digital Signal Processing (DSP) library which is the linchpin in the synthesisation of the audio. The third and final library is a small user-made library to control the TSL2561 Lux sensor from Adafruit.
We also use pulse width modulation to control the speed of the motor spinning the sensors on the device to produce a multitude of effects.
Our Intended Audience
Our audience is 6 to 10 year old children who are just about to start or are already learning an instrument. Wobble is used for interactive play with music, sound and lights to get children interested in technology and music through interaction. Our intended outcome would be to build something that would engage them musically and technically. They would be able to create music in any environment they wanted on an abstract level. The older children would be able to then continue on their experimentation by modifying the open source code and easy to use breadboard setup to create their own sounds and music.
Our background research had us looking into artists using their pieces to change the environment like Andy Goldsworthy, Javier Riera, Jim Sanborn and Olafur Eliasson. Biomimicry, particularly sonar and ultrasonic mapping of environments played a large part in our research. Wobble fundamentally grew from the idea of reinterpreting an environment like the bat and how they navigate through environments quite unlike any other mammal.
We had a pretty good idea about how we wanted Wobble to sound. We wanted to have a device that had a very unique synthesized sound, and we wanted the sound to be both interesting and fun for kids. The main way we tested our sounds was to generate pure sine waves and then add envelopes and frequency modulation. To find the sound that we wanted, we used ofxOsc for Openframeworks and OSCpack on our Raspberry Pi. Through OSC messages we could now communicate between our two devices and play around with parameters regarding the sound output. By moving the mouse in our Openframeworks program the mapped position of the mouse changed the sound parameters inside our running code on our Raspberry Pi. Through this we could interact and change the sound in real time.
Throughout the project we have experimented with the Maximilian library to find the right sound. We started making some simple tests, such as playing specific frequencies when you press certain keys on the keyboard to change the keyboard input to be replaced by our sensors. Then we moved on and added more functionalities from the library such as envelopes (ADSR), filters (Hi-res and Lo-res), delay and compression such as noise gate. We also made our own filters such as a foldback distortion and a Moog filter to make the sound even more interesting and fun.
To start we created the paper prototype of the device for some real world context and to test whether our hardware would fit into it. Below is the first 3D model.
We chose to have the shape in two parts as it was both aesthetically pleasing and economic in regards to space. We wanted the top half to spin while the bottom remained still as a base. The two sketches below show some variations of mechanics proposed. Eventually we decided on a electronics housing that sat on a lazy susan bearing that would be driven in a circular motion with an inverted cog and gear system (see below for demonstration).
We then made a cardboard prototype of the housing to once again test the sizing and aesthetics of the device. Using the Pepakura suite we broke down the 3D model, created in Maya, into a 2D mesh that was printed and glued to the cardboard, cut folded and glued into shape.
The finished cardboard prototype.
The prototype with testing rig of electronic.
Now that we had the basis for our housing complete we started to assemble the 3D models for the housing and the internal drive system. Below is the final representation of the internal assembly. The small inner cog will be held with a DC motor attached to the inner electronics housing placed in the centre.
We started 3D printing the housing with clear ABS plastic to make the final product opaque so that light can shine through from inside.
The whole printing process took 44 hours for the final product, not including the failed attempts which could easily be over 15 hours. The printing was spread over three weeks of iteration and testing and then final printing. The cogs and electronics housing was printed in white as we were doubtful of getting a full print from the clear plastic we had left.
Some of the pieces were too large for the printer bed so the incidentally had to be sectioned. The pieces were then welded back together with a plastic adhesive.
Once the two bottom sections were printed we could start assembling. Below is the lazy susan bearing placed at the bottom of the device.
Below is the first iteration of the ultrasonic sensor we are going to use only with a Arduino instead of a Raspberry Pi. The approach for the Pi however was the same, we knew we wanted to use the breadboard so that any user could modify their own Wobble to suite their project.
Below is all of our sensors and our motor wired into the breadboards for testing. As you can see on the right breadboard we are using a Pi-Cobbler to extend the Pi’s GPIO pins onto the breadboard.
Here is the current working model of the breadboard with the essential circuitry wired in with short wires to be flush with the boards. A needed improvement to the above circuitry.
The first version of the center piece. This one is much smaller in diameter and has the small and not so powerful motor mounted of the outside of the center piece.
The Video below is the new center piece with the which is much larger and the new larger motor is mounted on the inside through a hole in the center piece.
Here is Wobble with the top on.
Here is the all the components setup for testing.
Here is the full setup placed inside the center piece.
Below is an image of the first test with the LED lights and the Arduino.
Links to executables
Link to video of the first test with the new motor: https://youtu.be/yrZ8t-VzUTo
Test with lights and motor: https://youtu.be/M77j2RyFHrw
Final test: https://youtu.be/xQt04OLUIRA
Our Building Process
When approaching the build of Wobble we knew that we wanted to reach a point of minimum viable product, or at least a working prototype, that could be used as a showcase of our skills. Because of this we leapt headfirst into building methodically with the intent to reach our weekly milestones each week.
Even though one of our main features was to have Wobble spin we couldn’t actually test the spinning mechanism until very near the end of the build, once we had finished our final 3D printed parts and sourced all of our hardware. Knowing this, we had to make a leap of faith by waiting until the end of the process for this vital piece of hardware. By planning for this delay in testing we were able to keep it in mind when designing the other features of the device, making sure that the end result would be able to be powered by the motor.
One of the first issues we encountered was getting the initial data from the sensors we were using. Both the ultrasonic and lux sensors were extensively documented in both Python and the Arduino IDE. Because of this there were no libraries built for our intended use. After much research, we found two user made libraries for both sensors that we modified to suit our needs by extracting only the data that we were going to use and using some of the code to make classes. The real saving grace to this issue was by way of WiringPi, the GPIO access library in C++. When installed onto the Pi we could use any pin as if we were developing in Python.
A major issue then sprung up with the ultrasonic sensors. The distance data that was being calculated would be received and printed in our console for testing. After an indeterminate amount of time – from five to twenty seconds – the console would throw a number unlike any other, usually far larger than expected, and then nothing else would be printed. What was happening was the distance gathering function was being called so often, more than once every couple milliseconds, that the sensor was being physically overloaded trying to measure the many signals it was receiving. To combat this we implemented a timer function that measured the seconds elapsed and an if statement that only let the sensor update its distance function every 0.2 seconds. This time is arbitrary, it can be as fast or slow as you like up till the point where it overloads itself.
Another issue we came across near the end of our build was that when everything was mounted together we realized first that our DC motor was much too weak to pull all the weight of our components. We also found out that our first centerpiece that would fit all the components was also too small. If we had kept this print we would have great issues with fitting everything inside of it. So, we decided to do one more print where the centerpiece would have a much larger diameter, this would give us much more space for all of our components.
Our project was semi-successful. It turned into a strong first prototype that demonstrates the ideas and the technology used to great success. The 3D printed housing came out fully and as we intended with very few flaws. The construction is solid and the look is very aesthetically pleasing. The sound of the motor and the bearing spinning is relatively low and doesn’t detract from the overall experience.
The sound can be manipulated very easily and intuitively even on first use. The fast response and visible sensors allow the user to understand how Wobble works at first spin. The wirelessness of Wobble is by far one of the best selling points of the device; being able to take Wobble with you wherever you go is a boon when testing it out in unique environments. We would have limited our users greatly if we had not gotten to this point of completion.
Some of our success was marred by running out of time to finish some of our hardware features, the first of which was not fully integrating lighting in the device. We were trying to install programmable LED lights that would change with the audio being produced by Wobble. Instead of this,we connected them to an Arduino. This was a quick implementation so that we could show the aesthetics of the lights even without the functionality.
The second feature to not make the final prototype was to allow the user to control the speed of the motor. This was in part due to the original motor used for testing not being powerful enough, and also in part the pulse width modulation causing some motors to be not powerful enough when running at slower speeds. To combat this we have sourced a physically geared down motor to run at a steady 10rpm, however, we’ve had to wire it into a power source to make it run constantly.
All in all it was a successful first try at the device and there are clear improvements to be made. We could redesign the assembly to make the device smaller and we would like to fix the issues with the lights and motor so that they are both controlled by the Pi. We could even give Wobble more life by letting it record the data it takes in from the environment to create a pseudo memory. This memory could then be mimicked by other users to hear particular environments.
Reference & Bibliography
WiringPi – http://wiringpi.com
Maximillian – https://github.com/micknoise/Maximilian
Pepakura – http://www.tamasoft.co.jp/pepakura-en/
Autodesk 123D – http://www.123dapp.com/design
Digital Signal Processing – http://musicdsp.org/archive.php