The project was an audiovisual application that combined hardware and software to generate sounds. Overall, it intended to explored how visual stimuli, colours and different forms of sound can be combined as well as show another approach to music and one that would be easily understood by the audience.
The projects software is a program that draws six lines in a circular motion, setting each of them to a different and random HSB colour. The hue of each of the lines is interpreted by the program and turned into an integer that lies between 0 and 360. All the lines hues are then averaged and depending on where the final output value lies between all the 60 degree ranges, the program then plays a specific note (if larger than 330 and smaller than 30, the colour is red and the program plays C1 and so on). The program lies on chance of which is the dominating colour of the lines and then sends the instructions to the Arduino.
The hardware part is thus triggered by the averaged colour variable and depending on which one of the six colours is true the corresponding floppy drive plays a note. Each floppy drive plays only a certain note and each note is associated with one of the six(yellow, red, cyan, magenta, green and blue) HSB main colour hues.
The notes on the floppy drives are configured to be played using the the speed of the stepper motor that moves the pedal inside it. By changing the speed of the motor to a certain amount a specific note can be generated. As an end result the idea was to attempt to represent colours through sound, as well as explore the variety of how audio can be represented.
The project started out wanting to experiment with sound and the variety of ways that sounds and notes can be created. The intended audience thus, from the beginning were people with not the best of knowledge and depth into musical theory or even computing for that matter. I wanted to give the viewer an opportunity to create sounds of his/her liking simply through experimentation and through easily comprehended outputs. From the beginning these sound outputs were intended to be ticks that would differ in notes depending on what the material the dials were ticking against. I created prototypes for this using Processing to see how this would visually look and as well as I was trying to find a way where this output of sound could be independent, not changing due to the user’s input, but due to some other instance or obstacle.
I then tried concentrating rather on just one type of rotating output or input and experimented with just one stepper motor to see what would be the sound that it made when ticking. The code was at this stage done just in the Arduino IDE.
After experimenting with the rotation of the motor and the Arduino I thought about using that movement of the circle and experimented with sketches that would use circular motion to create data visualisations. I looked at different ways to use circles and how they have been used in music and I came across a Spectogram that output frequencies using the distance of dots from the centre and output amplitude as the brightness of these dots. The higher the amplitude the higher the point and the lower the amplitude the darker the point. I used that to create a sketch that would draw randomised points across a circle to see how the output would look, nevertheless without the analysis of the frequency. However, I still wanted an audio output within the project and one that would to an extent have a system of some sort. As a result with the intention to experiment with colour I created circular lines drawn in different randomised colours deciding to use them as the triggers for the audio outputs. For aesthetic and simplicity purposes I used the HSB colour model as it displayed a more pleasing set of colours as well as it would be easier to get the average colour by just detecting where between 0 and 360 it is in the hue.
From then on I wrote my program in C++ using Openframeworks and continued experimenting with the physical audio outputs. The stepper motor was not enough and I came across Sam1Am Moppy Software’s Github page that apparently was the beginning of a still growing community which uses floppy drives to output certain notes and thus even play songs (preferably ones which are within the range that can be played on the drive). I used the instructions along with his software to see what the actual output would sound like.
I experimented with the Moppy program and used the code as reference to understand how the speed of the motor within the drives is changed to generate a specific note and then tried getting those notes within my program. However, I could not get the right notes as I encountered many problems with moving the floppy drive backwards and forwards. I started with moving one, and whilst that worked from the Arduino IDE I could not manage to trigger all the floppy drives realtime from the program that was running in Openframeworks.
As an end result the floppy drives did not trigger correctly depending on the colours and instead either lagged or played all at once without the correct delays. Thus from the viewers perspective it seemed that not only the colours were generated randomly on the screen, but also the physical audio output was chaotic as well.
Comments on the build
I did not come across with many major problems when dealing with the code for the circular color line. Some smaller and more tedious ones where getting the hue of each of the members of the rings class after they had been changed. Essentially from the beginning the randomness of the colour of each of the rings was defined in setup, thus when it got changed within the member functions I could not pickup the new changed hue. For that I used a pointer to each of the members of the class to access the new hue of each of them to get their overall average.
Another problem when using Openframeworks and C++ I encountered was connecting to the Arduino. As when previously using Processing the connection as well as the documentation for it was more straightforward, whilst at the time it was hard to find not outdated Openframeworks connection examples with the Arduino as they varied depending on the version Openframeworks and it libraries. However, fortunately within the newest version of Openframeworks there is already an inbuilt Arduino class that can be used to connect to it, taking into account that the Arduino board has the StandardFirmata sketch uploaded unto it from the examples folder. However, even after sorting out the connection I still came across various problems when triggering the function for the floppy drives from Openframeworks.
Whilst the movement of the motors worked as intended from the Arduino IDE, when porting them to C++, I couldn’t manage to achieve the right result. The main struggle was not only triggering them to move the right way, but also to play the right note. I did manage to achieve a similar version of just the movement I had written in the Arduino IDE, however there still remains space for much improvement regarding the outputs of the floppy drives.
After going through all the prototyping and exploration of different approaches and triggers of sound I was satisfied with the end concept of the project. The interface that showed the randomised colours seemed clear and simplistic enough. However, due to the difficulties that I went through to get the right notes playing on the floppy drives it eventually came to sound like what I feared most – just randomised sound. Thus the end result did not seem as the easily understood application that it should have been.
Apart from that, when getting the right notes, I wished to build upon the application to also use the saturation of the colours to trigger the seventh note. Further on I want to find a more systemised generation of the colours so that the notes in some ways would evolve into repetition that would allow it to seem as a track itself.
Link to gitlab repository that contains the final project as well as documentation files:
To me, To you
By Tom Pinsent
‘To me, To you’ is a collaboration between me and my computer. Inspired by a bombardment of screens, wires, beeps and boops alongside a lack of traditional, more physical forms of work within the digital art world, the electronic part of my piece is kept solely within the production. This leaves the final product independent of digital technology, un-reliant upon electricity to be experienced, although having been heavily influenced by it.
The paintings are created through a step-by-step process. It starts with a simple shape or form painted on to a canvas. My program detects this through a webcam and compares the shape to a pre-trained set of shapes, with the most similar projected back on to the canvas. I add these shapes to the painting, and the process repeats until the picture is done.
Audience and Intentions
Through my piece I wanted to convey an interplay between contemporary artistic practice and the
traditional, through the counterposition of modern technology with the conventional, seemingly
dated, medium of paint. Furthermore, I’d hoped to suggest a dialogue that is the collaboration between an artificial, digital artist, consisting of both hardware and software, and
myself. I hoped to show viewers that contemporary art with a heavy use of technology does not
have to be all “beeps” and “boops”, and can in fact exist within the real world (as opposed to the ephemeral cyber-space realm), in a form that is well known and recognised throughout art history. My work might attract those with either professional or personal interests within art or computer science fields, as well as hopefully providing insight to members of the general public who may not have been exposed to these kind of ideas before.
When starting with this project, I looked at a range of artists and works that dealt with both contemporary and traditional methods of creative practice, where neither aspect holds a greater importance than the other. New approaches are combined seamlessly with the conventional, bringing both styles into a new context, therefore producing art that feels fresh yet familiar.
Names include Yannick Jacquet, Joanie Lemercier and more.
I also conducted further technical research, by looking at computer vision, machine learning and neural network techniques.
Udacity Deep Learning course (mainly convolutional neural network section)
Self Organising Map tutorial
I initially thought this would be useful since a Kohonen Self Organising Map is a unsupervised machine learning algorithm widely used for grouping together images (and many other types of data) through similarities.
Design Process and Commentary
I began with an initial list of things my program needed to do:
1. Scan canvas and detect shapes
2. Analyse detected shapes and composition
3. Create new shapes and forms based on how well detected forms fit a desired aesthetic
4. Display new forms on to canvas
I then began exploring methods and techniques used in computer vision, machine learning and neural networks, since these are areas that are used within similar applications, such as image classification/face detection.
When scanning and detecting shapes, I wanted my setup to work like this, which would be the simplest and easiest during the actual use of the program (the painting of the canvases), and meant it would be suitable for live scenarios:
However, I found this was impractical for a few reasons: The camera should not detect anything apart from the desired shapes on the canvas – even with cropping, the distance combined with the noise from a low quality webcam makes for very bad shape detection. At this stage I did not know how computationally intensive the detection and generation of shapes was going to be, so I though committing to a setup and build geared for quick and easy use may have led to problems later on, towards the end of completion, and a live scenario might not have been viable. Also, there was an issue of getting the same perspective between camera and projector, for accurate representation of shapes from both my point of view and my program’s.
Instead, I ended up holding the webcam up to the canvas and pointing it at the desired shape. This worked well, since with the web cam closer up, there is greater detail in the image and hence more accurate contour detection. I also found that a ‘1-to-1’ representation of forms on the canvas and displayed shapes was not necessary at all, which I will come to later.
In terms of the shape detection itself, I started with ofxOpenCV. I used a variety of test images (as the webcam was unavailable at this stage) and results were OK, but contained a lot of noise – rough, jagged edges were picked up all over contour detection. I later implemented the use of smoothing and resampling functions within the ofPolyLine class in openFrameworks, which made some improvements. In the end I switched to the ofxCv add-on, a package with greater functionality than ofxOpenCv, closer to the OpenCv library. Contours detected with this library were much better.
2. and 3.
So I wanted to find the similarity between a painted shape and a set of test shapes, then generate shapes from there. Initially my test set was going to be a collection of images, textures and forms that fit some sort of aesthetic of my choosing. I thought the use of genetic algorithms, such as L-systems, cellular automata or other rule based systems, to generate my test set would be interesting, further increasing the influence of my computer on the work. I assumed this data would be in the form of .jpgs, so implemented various functions for loading and analysing data from image files. Yet as experimentation continued, I was informed about SuperFormula, an algorithm for the generation of shapes, most of which resemble forms throughout nature. The beauty of this algorithm is that such a range of shapes can be created just from the alteration of four individual variables. I also realised that my training set not longer needed to be in the form of imgs, and i can just take vertex positions and feed that straight into my program. I edited and modified a version of superformula in processing, allowing me to create and save a training set.
Finding a value for the similarity, or difference, of two shapes was very tricky, since there are many factors to take in to account, such as size, orientation, location… After much research and testing, I settled on a method using a turning function, as described on page 5 (section4.5) here:
Using this website as a reference along the way:
This function allows you to map the change of angle from vertices along the perimeter of a shape onto a graph. It is then (relatively) easy to calculate a distance, or similarity, between a two or more sets of points (for example, shape A’s first angle change is compared to that of shape B’s and saved, with the same happening for every other vertex. A mean average of similarity is found at the end, and therefore a similarity between two / more shapes).
So now I have a range of base training shapes, each with a similarity to an initially drawn shape. I then looked into clustering, so i could group together similar similarities. However, after more research I found it was quite inefficient to implement most clustering algorithms on 1 dimensional data, my single similarity value, and instead looked to the Jenks Natural Breaks algorithm. This is used for splitting up a set of data into sub groups with similar values ( so shapes with similarities 1,7,15 would be in a different group to those with 150,120 or 110 for example). After my data had been split into sub groups, the most similar group, so that containing the lowest values, were used to generate images.
However, for the actual generation, this was all – the initially generated superformula shapes in the training set were just re-presented. I tried to use the values fed into superformula to generate the most similar shapes as a starting point, and slightly tweak the values and regenerate new shapes. But I found after even small tweaks, variation was too great and any detection of similarity was lost.
I would have liked to include some sort of composition calculation too, yet I found within the time I had this was not possible. Instead, for the paintings produced I tried different methods of composition. One was my own choice of placement of generated shapes, the next was a layering of shapes on top of one another, from most similar to least, and the last I tried distorting the projection itself. As I was painting, I noticed how the projection distorted along with the distortion of the surface itself. This made me think about how my program had influence over the piece through the code, the software, but I am also painting over a projection. How can the hardware and positioning of this affect the work?
Overall, I have learnt a lot from this project. With initially very little understand of machine learning or computer vision techniques, I had to read about and jump over many small hurdles and problems.
I would have liked to have a more accurate measure of similarity, since currently my turning function does not work well with changes in rotation or lots of noise on the contours.
The generation could have been better too, since in the end it is almost just recycling the initial forms giving to it (however, parallels can be seen between this and the age old mystery of what true creation really is, if it really exists, or if we ourselves just recycle and reproduce ideas presented to us). I would also like to implement some sort of colour and composition training and generation. My code is also quite messy, with many functions included that weren’t used in the final painting. This could have been avoided by better planning of exactly what functions and methods I was going to use, with a clear idea of exactly what I needed.
Furthermore, I believe the majority of interest within my project lies in the production, in the technology, leaving the traditional values that I wanted to place emphasis on severely lacking. I succeeded in my plans to push technology out of the limelight, yet it left the public had little understanding of the process of its creation. For similar projects in the future, I would need to spend at least as much time upon the physical, non-electronic side of things as I did on the electronic if I would like them to hold the same importance, as well as finding a way to communicate the process clearer through the work itself (in other ways than a short video documentation on the process).
Cycle is a series of technology-assisted performances, incorporating the use of robotics and sound. It was inspired by the interrelating concepts from Graphic Notation and East Asian Calligraphy/Ink Wash Paintings. In each unique recurrence, Cycle explores the theme of spontaneity and individuality transpired within a structured framework as the performers present their own interpretation of a set of instructions.
Each performance lasts approximately three minutes, give or take a minute; the performers end it at their own discretion. During the performance, a sole performer walks around the ‘Ink Stick Rotation Machine’ (ISRM) in a seemingly undefined way. The ISRM grinds an ink stick on an ink stone according to how the performer walks. Ambient sounds and vibrations generated from the constant moving contact of the ink stick and ink stone are amplified by speakers through a microphone located on the sides of the ink stone in real time.
In the performer’s interpretation from a set of rules constructed by the graphic score’s composer, control over the manner of performance is removed from the composer’s authority which alludes to a spontaneous creation of the performance by leaving it to ‘chance’. Unlike music represented in traditional notation, different performances of one graphic score do not have the same melody yet still articulate similar notions expressed in the score. In the cases of Ink Wash painting, the rules in posture, way of holding the brush, and practiced strokes, the results cannot be fully controlled by the painter and are still unpredictable due to human error and the nature of ink and water – their interaction take on a life of its own.
The audience sees and listens – nothing really comes out of watching the performance. Yet, even if the audience does not understand the concepts implicated in this work due to requiring some background knowledge about the act of grinding an ink stick, to experience Cycle, they merely have to practice being in a state of calmness and ambiguity. Just like when a painter or calligrapher prepares ink by manually grinding the ink stick, it is to ebb their flow of thoughts, momentarily forget about the things that are happening outside of the performance and just watch and listen. The performance would be both like a ‘performance’ and a non-religious ‘ritual’ at the same time. The feeling that one would sense is like when one is a non-Buddhist listening to the chants of Buddhist monks. Strangely calming, yet it could get annoying when one listens to an ununderstandable language for too long.
For the performers, I would hope that they would be in a world of their own without minding the presence of the audience and focus on their body walking in a circular path, yet I can imagine that they would perhaps be nervous in front of an audience, especially if they are performing for the first time. As a recurring theme in my work, ‘walking’ is a simple movement that can be of disinterest and a distraction all the same. It not only refers to the bodily action of moving your legs as a mode of transport but also signifies the act of repetition, which is structural, and the mundane. As the performer walks after a few times, the performer may build up a personal routine or choose to walk a different manner each time.
After my research on Graphic Notation and East Asian Ink Wash Paintings, I have drawn connections between these two distinctively different genres in art and show their overlapping characteristics in which my artwork attempts to embody conceptually. I likened graphic notation to instructions that were rather open-ended yet specific in certain ways, hence, I decided on creating a performance that Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink.
Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink. With the addition of the sound of the motor, I thought that the sound would be a nice hybrid between the organic and inorganic materials.
In the late 1950s and the first half of the 1960s, many prominent international avant-garde composers such as Roman Haubenstock-Ramati, Mauricio Kagel, and Karlheinz Stockhausen, as well as experimental composers such as John Cage, Morton Feldman, and Christian Wolff started to produce graphic scores that used new forms of notation and recorded them on sheets that were very divergent from traditional music notation in size, shape, and colour. This new way to convey ideas about music alters the relationship of music/sound to the composer and musician. “In contrast to scores with traditional notation, graphic notation emphasized concepts and actions to be carried out in the performance itself, resulting in unexpected sounds and unpredictable actions that may not even include the use of musical instruments.” (Kaneda, 2014)
Here, I focus on how graphical notation evolved from John Cage’s musical practice and then on Treatise, one of the greatest graphical scores, by Cornelius Cardew.
Influence of Zen Buddhism in Cage’s Work
In Cage’s manifesto on music, his connection with Zen becomes clear: “nothing is accomplished by writing a piece of music; nothing is accomplished by hearing a piece of music; nothing is accomplished by playing a piece of music” (Cage, 1961).
This reads as if a quote from a Zen Master: “in the last resort nothing gained.” (Yu-lan, 1952). Cage studied Zen with Daisetz Suzuki when the Zen master was lecturing at Columbia University in New York. Zen teaches that enlightenment is achieved through the profound realization that one is already an enlightened being (Department of Asian Art, 2000). Thus we see that Cage has consciously applied principles of Zen to his musical practice: he does not try to superimpose his will in the form of structure or predetermination in any form (Lieberman, 1997).
Cage created a method of composition from Zen aesthetics which was originally a synthetic method, deriving inspiration from elements of Zen art: the swift brush strokes of Sesshū Tōyō (a prominent Japanese master of ink and wash painting) and the Sumi-e (more on this in the next section) painters which leave happenstance ink blots and stray scratches in their wake, the unpredictable glaze patterns of the Japanese tea ceremony cups and the eternal quality of the rock gardens. Then, isolating the element of chance as vital to artistic creation which is to remain in harmony with the universe, he selected the oracular I Ching (Classic of Changes, an ancient Chinese book) as a means of providing random information which he translated into musical notations. (Lieberman, 1997)
Later, he moved away from the I Ching to more abstract methods of indeterminate composition: scores based on star maps, and scores entirely silent, or with long spaces of silence, which the only sounds are supplied by nature or by the uncomfortable audience in order to “let sounds be themselves rather than vehicles for man-made theories or expressions.” (Lieberman, 1997)
John Cage: Atlas Eclipticalis, 1961-62
Atlas Eclipticalis is for orchestra with more than eighty individual instrumental parts. In the 1950s, astronomers and physicists believed that the universe was random. Cage composed each part by overlaying transparent sheets of paper over the ‘Atlas Eclipticalis’ star map and copied the stars, using them as a source of randomness to give him note heads. (Lucier, 2012)
In Atlas, the players watch the conductor simply to be appraised of the passage of time. Each part has arrows that correspond to 0, 15, 30, 45, and 60 seconds on the clock face. Each part has four pages which have five systems each. Horizontal space equals time. Vertical space equals frequency (pitch). The players’ parts consist of notated pitches connected by lines. The sizes of note heads determine the loudness of the sound. All of the sounds are produced in a normal manner. There are certain rules about playing notes separately, not making intermittent sounds (since stars don’t occur in repetitive patterns), and making changes in sound quality.
Cornelius Cardew: Treatise, 1963-67
After working as Stockhausen’s assistant, Cornelius Cardew began work on a massive graphic score, which he titled Treatise; the piece consisting of 193 pages of highly abstract scores. Instead of trying to find a notation for sounds that he hears, Cardew expresses his ideas in this form of graphical notation, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated (Cardew, 1971). The scores were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes. (Tilbury, 2008)
As John Tilbury writes in Cornelius Cardew: A Life Unfinished (2008), ” The instructions were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes.”
“A Composer who hears sounds will try to find a notation for sounds. One who has ideas will find one that expresses his ideas, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated.” – Cornelius Cardew
In the Treatise Handbook which guides the performer on the articulation of the score, Cardew writes that in Treatise, “a line or dot is certainly an immediate orientation as much as the thread in the fog” and for performers to “remember that space does not correspond literally to time.” (A Young Persons Guide to Treatise, 2009)
East Asian Ink Wash Painting
The Enso, or Zen circle, is one of the most appealing themes in Zen art. The Enso itself is a universal symbol of wholeness and completion, and the cyclical nature of existence, as well as a visual manifestation of the Heart Sutra, “form is void and void is form.” (Zen Circle of Illumination)
Despite there being many specific technicalities in Cage’s work, these are all qualitative instructions which are open-ended, ultimately leaving it up to the performer’s or conductor’s judgement on how they would play the piece as implied by Cardew’s ideas. In a sense, the individuality of each performance of the graphic score by different performers emerges. This is mirrored in appropriating the creation of the Enso in Cycle by the performer. Every painter draws a circle but every circle is different. Bodily and mindfully engaged in drawing the circle, the circle becomes an allegory of the individual.
The performer not only becomes both the painter and the medium in creating the circle, the performer is also a musician with the indirect control of the device that grinds ink – the instrument with a naturalistic sound created from the contact between the ink stick and the ink stone. To quote Cage’s approach to what defines music, it is the “the difference between noise and music is in the approach of the audience” (Lieberman, 1997).
The act of grinding the ink stick becomes the juxtaposition between the ritualistic and the improvised. Also, ink that is produced after each performance are of different quality each time as no two performances will last the exact same time nor will the performers be able to replicate their performance exactly.
Communication between the phone and the computer is through OSC. The ISRM is made up of an Arduino Uno, which controls a stepper motor, which is directly connected to the computer with a USB cable. The speed and direction of the performer would be measured by the built-in sensors in a phone on the performer. Data from the orientation sensor and accelerometer of the phone is computed in a C++ program on the computer which maps the speed and direction of the performer to that of the ISRM.
Controlling the Stepper Motor with C++
The Arduino part was pretty straightforward as there was the Firmata library for the Arduino that enabled serial communication with a C++ program. However, there was no stepper library in C++, so I translated the Arduino stepper library to C++. Working through the technical details of the stepper motor that I had with some trial and error, this was the circuit that I used to test controlling the stepper motor through a C++ program.
Here’s me testing the program out:
To hold the ink stone, ink stick, and the stepper into a single functional entity, I started off with a preliminary design of a 3D model in Blender, which eventually I was going to 3D print.
I got the idea of the rotation wheel and axis from the driving wheels of steam locomotives, but I was not satisfied with the motions of the rotating mechanism in the first prototype. It caused the ink stick to rotate in a rather awkward manner that did not keep the ink stick facing the same direction. I also removed the water tank as I felt that it was visually obstructive and had no better purpose than to provide the ink stick with water, which I did not manage to figure out a fail-safe method of channeling the water into the ink stone. I thought of using a wick to transfer water from the tank to the ink stone, but water transfer was too slow, or a small hole with a pipe dripping water to the ink stone, but the rate of dripping will change when the water in the tank decreases due to decrease in pressure. Also, it would damage the ink stick if I let it touch the water for too long periods of time, hence I scraped the water tank from then on and decided to manually add water before every performance.
There were many difficulties trying to get the holder for the ink stick to fit. I realised that it was never going to fit perfectly as the dimensions of the ink stick itself was not uniform; one end of the stick could be slightly larger than the other end, which made it either too loose or too tight when I tried to pass through the entire length of the stick through the holder. I resolved this by making the holder slightly larger and added sponge padding on the inside of the holder so that it would hold the ink stick firmly no matter the slight difference in widths. The ink stick was shaky when it rotated so I increased the height of the holder to make it more stable. I also added a ledge on each side of the holder for rubber bands such that the rubber bands could be used to push the ink stick downwards as it gets shorter during grinding.
Before arriving at the final design, there were just wheels that were only connected to each other through the rod. The rotation did not work like expected of a locomotive wheel and I realised that it was because the wheel not connected to the motor had no driving force that ensured it spun in the right direction. Therefore, I changed the wheels to gears.
The printed parts did not fit perfectly and that was not because of the wrong measurements as there was a factor of unpredictability in the quality of 3D printing. I tried using acetone vapour on the parts that need to move independently of each other to smooth the surface, but the acetone vapour also managed to increase the size of the plastic. The plastic became more malleable so I easily shaved them down with a penknife.
This process was too slow and I ended up using a brush to brush on the acetone directly to the plastic parts and waited for a few seconds for it to soften before using a penknife. Super glue was then used to hold parts that were not supposed to move together. The completed ISRM:
I used electret microphones that were connected to a mic amp breakout, then connected to a mixer for the performance. I got an electret microphone capsule to use with the Arduino but I did not know that the Arduino was not meant to be used for such purposes and the microphone was not meant for the Arduino.
So, I got another kind which could directly connect to output as I did not want to use the regular large microphone which would look quite ostentatious with the small ISRM.
Trying to amplify the sound of making ink (sound is very soft because I only had earphones at that time, and I was trying to get the phone to record sound from the earphones):
Sensor Data & Stepper Motor Controls
I initially thought of creating an android application to send data to the C++ program via Bluetooth, but there was the issue of bad Bluetooth connectivity, especially the range and speed of communication. Hence, I switched to using OSC to communicate the data. Before finally deciding on using an OSC app, oscHook, I made an HTML5 web application with Node.js to send sensor data. It worked well except for speed issues as there was a buffer between moving the phone and getting the corresponding data that made it rather not ‘real-time’, and it also sent NaN values quite often which would crash the program if there were no exception handlers.
For controlling the speed of the stepper motor, I mapped the average difference of the acceleration of the y-axis (up and down when the phone is perpendicular to the ground) within the last X values directly to the speed of the motor. Prior to this, I looked at various ways to get the speed and direction of walking, from pedometer apps to compass apps. As different people had different sensor values with the phone, I created a calibration system that adjusted the values of the mean acceleration when the performer is not moving and when the performer was moving at full speed. This ensured that the stepper will be able to run at all speeds for all performers.
Link to Git Repo.
Performance & Installation
Videos of performances were playing on the screen for the second day of Symbiosis. The TV was covered with white cloth on the first day. The ISRM was placed on a white paper table cover with the microphone next to it.
Instructions for Performers
Besides having to run a calibration before their performances, I requested the performers to wear “normal clothes in darker colours” to make a contrast with the white room walls. I decided not to specifically ask for black as it was too formal and intimidating. Although the performance exudes the sense of a ‘ritual’, it was not meant to be solemn or grievous, as was such cultural connotations of fully black clothes in a rather ritualistic setting.
During the performance, the performers were to heed these instructions:
- Walk around the room.
- When you stop, stop until you hear the sound indicates that the motor is at its lowest speed.
- End the performance when it is three minutes since the start.
Prior to completing the program that controls the stepper motor, I wanted to attach the phone to a belt and hide it under the clothes of the performers such that they would be walking hands-free. I realised that it was quite abrupt to merely end the performance with the performer standing still as there was no indication if the performer was pausing or stopping entirely to the audience. Hence, after realising that by placing the phone parallel to the ground caused the motor (and in turn the sound) to stop in an elegant manner, I decided that the performer would hold the phone (which I covered in white paper to remove the image a phone) in their hand and have them place it on the ground to signify the end of the performance.
There was a total of eight performances by three people, Yun Teng, Leah, and Haein. These are videos* of the performances by each of them on the Symbiosis opening night and their thoughts on their experience of performing:
*The lights in the room were off during the day, hence videos of the earlier performances look quite dark. If you do not hear any sound from the video, please turn up the volume.
“Being asked to perform for this piece was an interesting experience. For me, it was seeing how (even on a conceptual level, as it turned out) that my physical movement can be translated through electronics and code into the physical movement of the machine and the audio heard. Initially, although we were given simple instructions to follow and even, to some extent, encouraged to push these instructions, I was at a loss to how to interpret them, and just walked in a circular fashion. I tried to vary the pace, speed and rhythm of my walking in order to create variation, but ultimately fell back into similar rhythms of fast, slow, and fast again. It would have been interesting to perhaps push this even further if the machine was more sensitive to height changes, or arm movements – as a dancer who is used to choreography, this was a challenge for improvisation and exploration. In addition, due to the size of the room, the space was limited and hence the walking could only take place in certain patterns.” – Yun Teng
“At first, the walker was uncertain, distracted and anxious. She explored the link between sound and her unchoreographed strides and expected the connection to be instantaneous and obvious. However, it was not. There were delays and inconsistencies; the electronic and mechanic could not accurately reflect the organic. A slight panic arose from the dilemma of illustrating the artist’s concept to the audience and accepting its discrepancies as part of the performance. Slowly she started to play around with the delay, stopping suddenly to hear the spinning sound trailing on, still at high speed, and waited for it to slow down. Rather than a single-sided mechanical reaction to movement, the relationship between the walker and the machine becomes visible and reciprocal. Rather than just walking, now she also had to listen, to wait, and by doing so interact with the machine on a more complicated level. Through listening, she felt the shadow of her movements played back to her by the machine. The observation sparked contemplation on the walker’s organic presence versus the machine’s man-made existence and the latter’s distorted yet interesting reflection of the former.” – Leah
“The whole practice first was received as confusing and aimless as there was too much freedom for one to explore. It was challenging to perform the same act (walking/running) for more than two minutes. At first, I performed more than four minutes, unable to grasp the appropriate time, but it decreased as I repeated the practice. This repetitive performance was quite meditative and physically interactive with the work that caused me to wonder about the close relationship between myself and sound piece (which changes according to my walking speed). The most pleasant part of the performance was that I got to control the active aspect of the work and directly interact with it.” – Haein
The audience was very quiet, probably so that they could hear the sound that was very soft even at its loudest. When they first came in, they did not know what to do as there was no visible sitting area (so I directed them to sit at places that allowed the performer to roam most of the room). It was a huge contrast to the audience that interacted with my previous work as only the performer gets to have a direct interaction with the ISRM. Even then, the ISRM was visibly moving during the performances.
Just hours before the opening night, the ISRM broke at (fig. A & B). It was a mistake on my part as I was reapplying super glue (fig. B) to the base as it had somehow loosened from the previous application of super glue. In hindsight, I did not make extra parts (I did print extras of certain, not all, parts but they of no use when I did not bring them on site, nor were they ‘acetoned’ to fit together.), could not manage to salvage the parts, and I knew that I would not be able to reprint the parts in time. In the end, I slightly altered my work as the ISRM could no longer function as intended. Instead of having the microphones stuck to the sides of the ink stone, I stuck them to the stepper motor instead. Although the sound no longer had an organic element from the ink stick and ink stone, it was completely mechanical now.
After undertaking this project, I have learnt not to limit myself by my tools, but to explore different methods and tools before limiting myself in the creation of the work. I had a misconception that 3D printing was the most efficient way. In some ways, it was because it was the printer that was doing the hard work, not me and I did want to try 3D printing. Despite that, I should not have limited myself by my lack of consideration in using other materials to build the ISRM, such as the traditional way of putting together wood and gears. On the other hand, I do not regret my attempts to build an android app (which I quickly decided was not worth my time for the simple thing I was trying to accomplish) and a web application for sending the sensor data from the phone with Node.js as it is something new that I learnt even though I did not use it in my final work.
Fortunately, I managed to finish the design of the ISRM and print it out in time, but I felt that I should have focused more on the ISRM instead of coding in the earlier phase of the project timeline. 3D printing takes a lot of time, as I have experienced through this project, and any botched prints needed to be printed again as they are rarely salvageable even after being in print for hours. It is also tricky to get the settings right (i.e. infill) such that the printing time is minimised without compromising the quality.
Apart from the many technical things, I also learnt how to organise a performance art (this is my first performance art) and through making this artwork, there many more implications and questions that arise from what I created. For the performance, there were many things to keep track of, such as rehearsing with the performers beforehand, the attire of performers, the schedule of performances, getting the camera to film for documentation and managing the audience. In conclusion, despite being unable to carry out the performances as I have originally planned, I am glad that I have managed to still put together what is left of the entire work even when the ISRM failed to work correctly and the original intentions behind the artwork are still largely intact.
References & Bibliography
Works Cited in Background Research
A Young Persons Guide to Treatise. (12 December, 2009). Retrieved 2 November, 2015, from http://www.spiralcage.com/improvMeeting/treatise.html
Asian Brushpainter. (2012). Ink and Wash / Sumi-e Technique and Learning – The Main Aesthetic Concepts. Retrieved 2 November, 2015, from Asian Brushpainter: http://www.asianbrushpainter.com/blog/knowledgebase/the-aesthetics-of-ink-and-wash-painting/
Cage, J. (1961). Silence: Lectures and Writings. Middletown, Connecticut: Wesleyan University Press.
Cardew, C. (1971). Treatise Handbook. Ed. Peters; Cop. Henrichsen Edition Limited.
Department of Asian Art. (2000). Zen Buddhism. Retrieved 11 December, 2015, from Heilbrunn Timeline of Art History. New York: The Metropolitan Museum of Art: http://www.metmuseum.org/toah/hd/zen/hd_zen.htm
Kaneda, M. (13 May, 2014). Graphic Scores: Tokyo, 1962. Retrieved 2 November, 2015, from Post: Notes on Modern & Contemporary Art Around the Globe: http://post.at.moma.org/content_items/452-graphic-scores-tokyo-1962
Lieberman, F. (24 June, 1997). Zen Buddhism And Its Relationship to Elements of Eastern And Western Arts. Retrieved 10 December, 2015, from UCSC: http://artsites.ucsc.edu/faculty/lieberman/zen.html
Lucier, A. (2012). Music 109: Notes on Experimental Music. Wesleyan University Press.
Tilbury, J. (2008). Cornelius Cardew (1936-1981): A Life Unfinished. Copula.
What Ink Stick Should You Choose For Japanese Calligraphy? (2015). Retrieved 3 December, 2015, from Japanese Calligraphy: Modern Japanese Calligraphy inspired in Buddhism and Zen: http://www.theartofcalligraphy.com/ink-stick
Williams, M. L. (1981). Chinese Painting – An Escape from the “Dusty” World. Cleveland Museum of Art.
Yu-lan, F. (1952). A History of Chinese Philosophy. Princeton, New Jersey: Princeton University Press.
Code References & Software
Duppy Tree is an installation based on the concept of the Iroko bottle tree. The Iroko bottle tree can be found in West Africa and the Caribbean and its purpose is to keep bad spirits or bad vibes away. With what we have learned throughout the year, we were intrigued to see how these skills could alter and enhance the bottle tree concept, essentially creating an immersive experience for the user. We decided that we wanted to play with light so that as the user walks closer to the tree, the light gets brighter and as the user walks further away from the tree, the light gets dimmer. To achieve this, we used an ultrasonic proximity sensor via openFramework with the Arduino.
Although this installation appeals to artists and Interior designers, we believe that the tools we used for our project peak the interest of professionals amongst a vast range of fields ranging from: musicians, artists , interior designers, education purposes and set designers.
In this instance; the concept is an interpretation of the Iroko spirit bottle tree which is present in West African and Caribbean cultures. Therefore upon our artistic interpretation of the tree, it is neither a part of the past, on time or coterminous with European avant-garde modernist art, which follows the trajectory of a succession of styles (such as Fauvism, Cubism, Expressionism, Surrealism, Abstract Expressionism, etc.). This opens the possibility for a new artistic language.
- In the first week, Alabo attended “Sound System Culture London Exhibition” and shared various ideas that linked to sound and culture. Next, upon playing around with different signal paths and add-ons on OpenFrameworks and Max-msp, we thought that it would be interesting to use sound alongside the light and proximity sensor so that now, as the user walks closer to the tree, the light gets brighter and a sound is triggered.
- The following week, we started doing more research and decided that the best option for this project was a Raspberry Pi because we wanted it to be a stand-alone object. The Arduino was a good option but we thought the Pi was a better option due to its portability.
- For the project we agreed to equally split the costs, retaining a maximum budget of £500 for equipment. However, we managed to spend less. So we pretty much had the structure of Alabo focusing predominantly on sound, whereas Eddie was focusing on the audio reactive visuals.
- At this point our idea was to create an interactive speaker
- By week 4 we pretty much had the majority of our equipment, which enabled us to experiment with the Raspberry Pi. The Raspberry Pi was quite a challenging prospect. Ideally, we wanted to spend at most 3 weeks on getting the sound to work with proximity. If we managed to get something working, then we wanted to go out and get user feedback. This ultimately would have allowed us to know if we had a lot of work to do or not. We went beyond 3 weeks struggling with getting the raspberry Pi setup.
- The first thing we struggled with was getting the raspberry Pi to work on a Raspbain image. This OS allows openFramework to run on it. So we had 2 Raspberry Pi’s, Raspberry Pi Number 1 and 2. Now we both struggled on installing the image onto an SD card. Eddie managed to install it onto number 2 using a tutorial and it worked really well. Alabo had issues with the SD card he had, but one of us at least managed to install something, (after a while his SD card also had a Raspbian image.) The next step was to install OpenFrameworks onto the Pi which proved to be a big challenge, as we have never done anything like this before. Both of us started looking through the OpenFrameworks website and learned that we must install Linux arm6 onto the Pi so we tried. For some reason everything we tried kept failing so Eddie decided to add a blog post on the openFrameworks website and we got a response from Arturo. He suggested installing the nightly build and openFrameworks started compiling projects. We even added a camera and did simple codes just to check if it worked or not and it did.
- By this time Alabo had bought the LiFX lights and had begun trying to get them to communicate with OpenFrameworks but to no avail, they proved to be difficult to work with between home and university, due to needing to sign into a Wifi connection to access the lights.
- So the next step was to link the Pi with the ultrasonic sensor we bought using a breadboard, two resistors, and various wires. We checked online for many ways to set up which was a struggle to find. We had to ensure that the ultrasonic sensor outputs 3.3 volts or else the Raspberry Pi dies. The hardest and most challenging part was to compile a project that executes the sensor so that we could detect the distance. This was a real struggle. We tried and searched for various solutions, however, we couldn’t find solutions so we spent weeks on trying to get the sensor working with the Pi but ended up having to give up on it and deciding to use the Arduino instead.
- Alabo then purchased the Philips hue lights, to prototype on max-msp and TouchOSC leading us to see that this idea to merge sound with light was easily achievable with other platforms.
Touch OSC Test
- Trying to get the ultrasonic sensor to work proved a struggle, and after searching through forums and various tutorials we came across Cormac and Johan’s technical research which helped us set up the ultrasonic sensor with openFrameworks.
Philips Hue Light Test
Problems with our Build
The main problems we encountered with our OpenFrameworks code were to do with sending a request to communicate with the LiFX lights. Despite trying different addons, getting assistance from our lecturers and searching through the forums, we still couldn’t find a solution. Upon discovering that python could also send a request to the lights, we decided to approach the task by using python and communicating between the two programs with OSC messages.
However, still unsatisfied with the approach, Alabo bought the Philips hue lights due to there being a lot more online support and even an ofx addon that enabled you to change parameters from within OpenFrameworks. Now comfortable with the Philips Hue lights and aware of the similarities between the Philips Hue and LiFX’s API, Alabo was able to implement the same principles with the LiFX in OpenFrameworks with success. The solution lied in having to add a Syscommand.cpp file to the project, which allows you to then call the cURL request to the light.
Now, the only problem left was that the lights were not responding to the proximity sensor due to the fact that the sensor was sending data too quick for the network to respond. To produce the results we were after, we had to increase the delay number in the Arduino sketch and add it after ‘Serial.write’
Reflecting back on the project we realised that what we were actually trying to create wasn’t one product, but several that could be interconnected products like the Apple HomeKit. The sound speaker idea was a good idea however we just fell into many potholes approaching the deadline. If we did manage to get the sensor working on time then we would have had the trouble of playing the music wirelessly and working with the Raspberry Pi was a tedious process . So the decision to switch to Arduino saved the project because there wasn’t any time. In regards to functionality, we noticed several improvements that could be made. Firstly, our results for the prototype were very restricted and linear. To improve the project, we would like to use more proximity sensors and chain them in a circle around the tree to get readings from all angles. To conclude, in the future, we can look into developing our own Voice activation feature and creating an Intelligent Assistant similar to the likes of: Siri, Amazon Echo, Google voice control and Lifx Jasper (we can even create a way to implement these ready-made systems as well).
Raspberry Pi ultrasonic sensor – https://ninedof.wordpress.com/2013/07/16/rpi-hc-sr04-ultrasonic-sensor-mini-project/
Raspberry Pi Motion Sensor Tutorial – https://www.pubnub.com/blog/2015-06-16-building-a-raspberry-pi-motion-sensor-with-realtime-alerts/
Lifx API – http://api.developer.lifx.com/docs/set-state
Philips hue max msp OSC – https://www.youtube.com/watch?v=0D3tL4Wv9aQ
Arduino Ultrasonic Proximity Sensor HC-Sr04 – http://www.instructables.com/id/Arduino-HC-SR04-Ultrasonic-Sensor/
cURL Request in OpenFrameworks Code – https://github.com/wouterverweirder/devineClassOf2016/blob/master/src/SysCommand.h
By George Sullivan and Leon Fedden
Our project is an interactive installation using gesture and expression to explore sound scapes. We wanted to design an immersive environment where sound is manipulated through the users presence and kinetic movement of ‘nodes’. We have created a sound scape for the user to adapt and explore through intuitive ways which, we hope, sounds and looks pleasing.
By using AR computer vision techniques we have build a system that gives us a positional data stream for each node present on our table. Using this data we have created different ways of interacting with Reason  to manipulate sounds and textures present in our sonic landscape.
We built our own box in which to house the equipment and covered it with a transparent lid. Placing ‘nodes’ on top with AR code graphics facing down into the table, we are able to track the X and Y position of each node, as well as the angle it is to the camera, giving us a constant data stream of position to manipulate. In our code we have used multiple libraries to make this possible, and rely on several different input streams to be passed through our code and into Reason to create and shape the sounds heard around the room.
Target Audience & Intended Outcomes
Our original aim was to create an installation which offered someone of any musical background a platform to experiment with sound design. To make this interesting we needed our system to be responsive to the user’s gestures and thus inspire them to interact intuitively with the space. This offers an experience that is not exclusive, but instead, sound design to those with or without any previous knowledge. The control of each sound must be obvious when heard but implemented in creative ways, we created different methods of interaction for different nodes and sounds thus making an interesting piece which encourages experimentation. Although initially we aimed to create synthesis techniques with coherent actions, after research and experimentation of our system we decided that in order to create an interesting piece which sounded rewarding. We would have to give the user control over higher level audio processes rather than the gritty DSP side.
We spent a lot of time considering the control methods used in our project. After researching into embodiment we were inspired by the idea of creating an interface which provided a seamless connection between the computer system, the user, and the environment surrounding them. Consequently, we wanted our project to conceal all technology within the box we built, and allow our users to interact with physical objects within the space around it. However to succeed in creating this connection not only a tangible interface was necessary, but thoughtful and natural output from our program.
We researched other projects to see what methods were successful in similar pieces. Most notable was the Polyphonic Playground , which George visited at the start of term. After talking about it, we decided that although it was an impressive exhibit, the output sound from the system was not coherent with the interaction from the audience, especially when so many people were using it. The way the sounds work together was where we thought it fell short, and so in order for our project to be sonically impressive we decided that we would need to put in a lot of thought into the sound materials used. Another problem we saw with it was the users interaction was limited essentially to multiple switches. Consequently we decided that our nodes should exhibit differences in interaction to remain interesting.
It is, of course, worth noting Reactable , a recent project that is not too dissimilar with our idea. Believe it or not, we were unaware of this when we initially started thinking of our design. But since discovering its parallels with our ideas, it absolutely shaped some of our decisions for our design. A recurring criticism of Reactable is that has a feature set which can be difficult to learn, especially without a background in music. Understandably Reactable has a different audience, budget and time constraints, however it highlighted a few importances in our build.
Leon went to a guest lecture in the Whitehead building by Dr. Nicholas Ward . The lecture was how movement should be considered into the design of Musical Instruments and partially shaped how we mapped our nodes to our sound output. After researching further into some of his work we felt it really reinforced what George took from the Polyphonic Playground instillation. For example, In a paper written for a NIME conference ( New Interfaces for Musical Expression ), Ward explains the design process for his own ‘musical interface’ (The Twister) and discusses the importance of gesture design and sensible mapping which was becoming a recurring theme in our ideas. “The number of useful gestures discovered represents the starting point for the subsequent development of a musical gesture vocabulary” 
Design Process and Build Commentary
We had our users or target audience integrated into the plan from around the time of the project proposal; it was then we elected to build a system that was high level enough for anyone – interested in sound design or not – to be able to make interesting sounds. After some conversations with one-another we settled on a physical and digital form for the project. Having a shared vision we set out to draw components and how we envisaged users using them and how the system would be manipulated to achieve our desired output.
Having the ideas on paper really helped to ensure that there were no discrepancies in our expectations of the project. The next task was to reductively identify the key components of the project. Once we had a list of components we subsequently allocated time to build mini-projects to create each item on the list.
This was the fun bit; we were still green to openFrameworks and C++ (and still are in many ways) and we were opened up to the rather large ecosystem of addons and libraries that we might want to use. Aside from building the components as lego-bricks that plug together into a bigger cohesive model, the mini-projects served a second function; they were an objective method of rating the libraries effectiveness in providing the horsepower in some of the more complex computations. We also spent some time making classes or simple models which verified the sanity of our plans; checking and predicting approximate system dynamics for example by making a ‘table-top’ sketch controlled via mouse and keyboard that outputs midi much like the final model would.
Next is a roughly chronologically ordered list of mini-projects and what they achieved. The names reflect the titles of the projects submitted in GitLab:
- ofxArucoExample: This was one of the first things we did as the whole project hinged on being able to track objects of some description in the real, physical world. Initially there was a fruitless folly of a foray into ofxARToolkit. The library the addon wrapped at the time was most unfortunately not maintained and compilation was futile. However the omnipresent Arturo was soon on deck to help with a little library he wrote called ofxAruco. This project was (or is) his example that we used ensuring marker detection worked to a decent standard and it was worth proceeding with. It is worth mentioning that as of very recently, ARToolkit is back in development with a new API and subsequently a new addon Github page  has been made – something worth keeping an eye on perhaps.
- MarkerTracking: This was the class that took the example and wrapped it up into a “deceptively simple” (Theodoros Papatheodorou) class that kept track of the AR markers, their position, rotation and whether they could be seen or not. Weeks of struggling with an inaccurate reading of rotation and quaternions and matrices led to us waiving the white flag and bringing out the gaffertape with a cheap, hacky solution – sometimes it is easy to get caught in the finer details and forget the overarching picture, so we pressed onwards.
A small project to model the tabletop and markers – essentially circles that could be dragged around using the mouse. This was then turned into a class to interface with the forthcoming midi-library explorations.
Here we worked through an example to explore ofxMidi  and get familiar with the API. The documentation was straight forward and the examples provided gave us enough to implement our own basic midi messages.
We then took that knowledge of ofxMidi and spent time working out how to route midi signals on a Mac from our program to Reason. As basic note-on and note-off messages had been covered, we now wanted to create our own software interface to allow control over parameters inside of reason. This was where the documentation fell short and we spent a long time figuring out how (and verifying it was possible to) to send these controller change messages. After researching into how midi itself works, and digging deeper into ofxMidi, we were able to create our own midi channels and send controller data from our software.
- About ten assorted ofxMaxim projects: A lot of these projects initially stemmed or were from Leon’s tutorials for his technical research. They all fall under the bracket of fairly low level sound design so have been grouped together here. The real take away from these was that Maximillion was too low level for our users and means. The proceeding files are; Saw wave with pitch, Amplitude & Ring Modulation , Stereo Output , Frequency Modulation, and finally a combination of these processes.
- PS3EyeGrabber: This project took the openFrameworks add-on’s  functionality and wrapped it into a class that allowed for straightforward interfacing with the MarkerTracking class API. Essentially it copies the openFrameworks base class VideoGrabber API so the same functions can be called in the same place.
These mini-projects were key to refining our ideas. Often the usage of an add-on unearthed new possibilities and pathways for the project’s journey; one of the major realisations during this project was that Dr Mick Grearson’s Maximillion library  was too low level for what we had in mind sonically and by extension our target audience. It was here we decided to do the ofxMidi route into a Digital Audio Workstation (DAW). We went with reason because Leon had the most experience with that.
After we felt we had reasonable examples of projects that covered most components necessary for the final build we began to piece them together. From here on we iteratively refined the final code base to improve performance, remove bugs and to add new features like selecting modes, function pointer arrays to simplify modes (each mode could then have its own function in an efficient manner) and other changes.
The second stage of our design process was the physical build of the box and nodes. For this we worked alongside Henry Clements, who is a 2nd year Design student at Goldsmiths. We brainstormed initial design ideas, focussing on building something that allowed us to optimise space for interaction, and not limit the processes we wish to get out of our system. We also wanted to create something that was easily transportable and could be set up or taken apart easily, as well as allow space for the webcam, computer/raspberry pie, and any other wires or parts required in the future. Together we built a prototype design and created some initial blue prints.
Once we settled on a final design, we experimented with the range of the webcam to figure out how large our build would need to be. It was important to provide enough space for the user, and furthermore a large range of pixel data. We also took into account the size of the nodes, and the amount we would be using. In conclusion we decided to build the box to be 90 x 90 x 90 cm, giving us enough depth to get a large surface to play with and space inside for the webcam (and anything else that needed to be hidden), while retaining a design that was still portable.
You can see in the images above; blue print sketches for each piece of our box (center & left), a screen grab from the CAD software we used to get the wood cut using a laser cutter (right). We booked into a woodwork studio to have it cut, before measuring up all the dimensions for the lid.
To finish off, we added a stylish gold tint to our box and were ready to go. After spending a little time calibrating the camera and AR codes, we pieced all our code together and began designing sounds and interaction methods.
Regarding problems and dually our solutions, we were lucky not to have any major show stopping issues over the duration of this project. It would not be true to say it went without a hitch however; we have compiled a list of reflections that we have learnt from. Note largely this list is for the reader who is not our senior; we are sure they are beyond such mistakes. For all others have a good laugh at our errors if you don’t fancy learning from them!
- Sometimes the code in the library just doesn’t work. Or more frustratingly, it does work but it doesn’t give the expected amount. Even more frustratingly, it might feel like one is a maths class away from actually being able to re-write the damn thing. The issue in question was obtaining the z-axis rotational value of any given marker. The code as standard returned approximately twenty to three-hundred and forty degrees when a full rotation happened. We tried to get help on the forums and by talking to anyone who’d listen at University – start there, but in the end we had to settle for a (relatively) ugly hack, mapping the returned range to the desired range. It’s in our opinion better to keep moving rather than get caught up in the minor details.
- Midi was also an issue for us – we had a lot of trouble routing Midi through Mac OSX to our desired DAW. Here we can only recommend ‘Google-fu’, and if you’re working in C++, reading the header files! That eventually got us through.
- Sometimes libraries don’t compile, and are broken or outdated beyond repair or changing a few pre-processing directives. Don’t be afraid to throw in the towel and find a new alternative or better yet, depending on the scale, writing the solution yourselves.
- If working with hardware, paint or glue specifically, then ensure you test a corner before committing to the whole sheet. We eagerly tinted our Acrylic sheet gold before realising the tint was too dark for effective AR tracking. The solution was subsequently spending hours removing glue residue and it has never got back to one-hundred percent the transparency it used to have.
- A major, really stupid mistake I – Leon – made borders arrogance. Do yourself a favour and never make a major change to operating systems mid-way through an important computing project; having to change development ecosystems can really stifle progress whilst you have to wrap your head around new ways of doing things. I in particular spent time talking to different Tutors and friends to weigh up whether I wanted a new Mac computer or Linux system. Whilst I don’t regret the change ultimately; I have enjoyed the rather arcane art of some of the more masochistic Linux operating systems, I regret the timing and wish I had waited until the end of term.
- A more abstract error on our part was perhaps too much ambition. A personal view of ours is that it’s always worth being ambitious because the results are usually larger in a project the more optimistic you begin, but perhaps we expected we would be able to deliver a completely professional project on such a small budget and the time we set for ourselves. Time management of course played a part and there is always room for improvement on that end, but I never thought we were being lazy. The solution to this is not to stifle ambition, but perhaps be realistic in terms of the expected final project and remember you are only human!
Conclusion & Future Proposals
We are pleased with what we have achieved within this project. As we are using reason we can change and alter the source sounds appropriately.We managed to implement data mapping not just using X and Y positions but also from each nodes; speed, distance between them, how many are within a short distance, and rotation. It is possible to add new AR codes into our system providing you can identify them (use drawData function in markerDetection.cpp). We believe that if it was an exhibition piece, it would be of importance to be able to create the output for particular occasion, hence why we designed our software to be flexible. The video above is a short performance to demonstrate what is possible using our system. On a final note, we would like to add that it is difficult to get a feel for what is capable from our installation without spending some time experimenting with it.
Here speaking as myself, Leon, I can talk about what I took from the project, and more pertinently what it has inspired me to go on to do. After being introduced to the world of AR, I have taken an interest into the mechanics behind it and the applications of it.
Largely I would like to partake in more adventures in the blend of the digital and real world – mixed reality. Hololens  from Microsoft is a great real world example. Unfortunately however there are two issues here, firstly it’s Microsoft and I have little interest working in that eco-system and secondly the level of polish of their project is likely beyond my scope of possible achievement. However I like the idea of three-dimensional data representation and visualisation in the real world and would be interested in exploring avenues along those lines.
- Propellerheads.se. (2016). Create more music, record and produce with Reason | Propellerhead. [online] Available at: https://www.propellerheads.se/reason [Accessed 21 Apr. 2016].
- Studiopsk.com. (2016). Polyphonic Playground. [online] Available at: http://www.studiopsk.com/polyphonicplayground.html [Accessed 21 Apr. 2016].
- Technology, R. (2016). Reactable. [online] – Music Knowledge Technology. Available at: http://reactable.com/ [Accessed 21 Apr. 2016].
- DMARC | Digital Media and Arts Research Centre. (2016). Dr. Nicholas Ward. [online] Available at: http://www.dmarc.ie/people/academic-staff/nicholas-ward/ [Accessed 21 Apr. 2016].
- Ward, N. and Torre, G. (2014). Constraining Movement as a Basis for DMI Design and Performance. [online] NIME. Available at: http://www.nime.org/proceedings/2014/nime2014_404.pdf [Accessed 18 Apr. 2016].
- GitHub. (2016). naus3a/ofxArtool5. [online] Available at: https://github.com/naus3a/ofxArtool5 [Accessed 21 Apr. 2016].
- GitHub. (2016). danomatika/ofxMidi. [online] Available at: https://github.com/danomatika/ofxMidi [Accessed 21 Apr. 2016].
- GitHub. (2016). bakercp/ofxPS3EyeGrabber. [online] Available at: https://github.com/bakercp/ofxPS3EyeGrabber [Accessed 21 Apr. 2016].
- GitHub. (2016). micknoise/Maximilian. [online] Available at: https://github.com/micknoise/Maximilian [Accessed 21 Apr. 2016].
- Microsoft HoloLens. (2016). Microsoft HoloLens. [online] Available at: https://www.microsoft.com/microsoft-hololens/en-us [Accessed 21 Apr. 2016].