The project was an audiovisual application that combined hardware and software to generate sounds. Overall, it intended to explored how visual stimuli, colours and different forms of sound can be combined as well as show another approach to music and one that would be easily understood by the audience.
The projects software is a program that draws six lines in a circular motion, setting each of them to a different and random HSB colour. The hue of each of the lines is interpreted by the program and turned into an integer that lies between 0 and 360. All the lines hues are then averaged and depending on where the final output value lies between all the 60 degree ranges, the program then plays a specific note (if larger than 330 and smaller than 30, the colour is red and the program plays C1 and so on). The program lies on chance of which is the dominating colour of the lines and then sends the instructions to the Arduino.
The hardware part is thus triggered by the averaged colour variable and depending on which one of the six colours is true the corresponding floppy drive plays a note. Each floppy drive plays only a certain note and each note is associated with one of the six(yellow, red, cyan, magenta, green and blue) HSB main colour hues.
The notes on the floppy drives are configured to be played using the the speed of the stepper motor that moves the pedal inside it. By changing the speed of the motor to a certain amount a specific note can be generated. As an end result the idea was to attempt to represent colours through sound, as well as explore the variety of how audio can be represented.
The project started out wanting to experiment with sound and the variety of ways that sounds and notes can be created. The intended audience thus, from the beginning were people with not the best of knowledge and depth into musical theory or even computing for that matter. I wanted to give the viewer an opportunity to create sounds of his/her liking simply through experimentation and through easily comprehended outputs. From the beginning these sound outputs were intended to be ticks that would differ in notes depending on what the material the dials were ticking against. I created prototypes for this using Processing to see how this would visually look and as well as I was trying to find a way where this output of sound could be independent, not changing due to the user’s input, but due to some other instance or obstacle.
I then tried concentrating rather on just one type of rotating output or input and experimented with just one stepper motor to see what would be the sound that it made when ticking. The code was at this stage done just in the Arduino IDE.
After experimenting with the rotation of the motor and the Arduino I thought about using that movement of the circle and experimented with sketches that would use circular motion to create data visualisations. I looked at different ways to use circles and how they have been used in music and I came across a Spectogram that output frequencies using the distance of dots from the centre and output amplitude as the brightness of these dots. The higher the amplitude the higher the point and the lower the amplitude the darker the point. I used that to create a sketch that would draw randomised points across a circle to see how the output would look, nevertheless without the analysis of the frequency. However, I still wanted an audio output within the project and one that would to an extent have a system of some sort. As a result with the intention to experiment with colour I created circular lines drawn in different randomised colours deciding to use them as the triggers for the audio outputs. For aesthetic and simplicity purposes I used the HSB colour model as it displayed a more pleasing set of colours as well as it would be easier to get the average colour by just detecting where between 0 and 360 it is in the hue.
From then on I wrote my program in C++ using Openframeworks and continued experimenting with the physical audio outputs. The stepper motor was not enough and I came across Sam1Am Moppy Software’s Github page that apparently was the beginning of a still growing community which uses floppy drives to output certain notes and thus even play songs (preferably ones which are within the range that can be played on the drive). I used the instructions along with his software to see what the actual output would sound like.
I experimented with the Moppy program and used the code as reference to understand how the speed of the motor within the drives is changed to generate a specific note and then tried getting those notes within my program. However, I could not get the right notes as I encountered many problems with moving the floppy drive backwards and forwards. I started with moving one, and whilst that worked from the Arduino IDE I could not manage to trigger all the floppy drives realtime from the program that was running in Openframeworks.
As an end result the floppy drives did not trigger correctly depending on the colours and instead either lagged or played all at once without the correct delays. Thus from the viewers perspective it seemed that not only the colours were generated randomly on the screen, but also the physical audio output was chaotic as well.
Comments on the build
I did not come across with many major problems when dealing with the code for the circular color line. Some smaller and more tedious ones where getting the hue of each of the members of the rings class after they had been changed. Essentially from the beginning the randomness of the colour of each of the rings was defined in setup, thus when it got changed within the member functions I could not pickup the new changed hue. For that I used a pointer to each of the members of the class to access the new hue of each of them to get their overall average.
Another problem when using Openframeworks and C++ I encountered was connecting to the Arduino. As when previously using Processing the connection as well as the documentation for it was more straightforward, whilst at the time it was hard to find not outdated Openframeworks connection examples with the Arduino as they varied depending on the version Openframeworks and it libraries. However, fortunately within the newest version of Openframeworks there is already an inbuilt Arduino class that can be used to connect to it, taking into account that the Arduino board has the StandardFirmata sketch uploaded unto it from the examples folder. However, even after sorting out the connection I still came across various problems when triggering the function for the floppy drives from Openframeworks.
Whilst the movement of the motors worked as intended from the Arduino IDE, when porting them to C++, I couldn’t manage to achieve the right result. The main struggle was not only triggering them to move the right way, but also to play the right note. I did manage to achieve a similar version of just the movement I had written in the Arduino IDE, however there still remains space for much improvement regarding the outputs of the floppy drives.
After going through all the prototyping and exploration of different approaches and triggers of sound I was satisfied with the end concept of the project. The interface that showed the randomised colours seemed clear and simplistic enough. However, due to the difficulties that I went through to get the right notes playing on the floppy drives it eventually came to sound like what I feared most – just randomised sound. Thus the end result did not seem as the easily understood application that it should have been.
Apart from that, when getting the right notes, I wished to build upon the application to also use the saturation of the colours to trigger the seventh note. Further on I want to find a more systemised generation of the colours so that the notes in some ways would evolve into repetition that would allow it to seem as a track itself.
Link to gitlab repository that contains the final project as well as documentation files:
To me, To you
By Tom Pinsent
‘To me, To you’ is a collaboration between me and my computer. Inspired by a bombardment of screens, wires, beeps and boops alongside a lack of traditional, more physical forms of work within the digital art world, the electronic part of my piece is kept solely within the production. This leaves the final product independent of digital technology, un-reliant upon electricity to be experienced, although having been heavily influenced by it.
The paintings are created through a step-by-step process. It starts with a simple shape or form painted on to a canvas. My program detects this through a webcam and compares the shape to a pre-trained set of shapes, with the most similar projected back on to the canvas. I add these shapes to the painting, and the process repeats until the picture is done.
Audience and Intentions
Through my piece I wanted to convey an interplay between contemporary artistic practice and the
traditional, through the counterposition of modern technology with the conventional, seemingly
dated, medium of paint. Furthermore, I’d hoped to suggest a dialogue that is the collaboration between an artificial, digital artist, consisting of both hardware and software, and
myself. I hoped to show viewers that contemporary art with a heavy use of technology does not
have to be all “beeps” and “boops”, and can in fact exist within the real world (as opposed to the ephemeral cyber-space realm), in a form that is well known and recognised throughout art history. My work might attract those with either professional or personal interests within art or computer science fields, as well as hopefully providing insight to members of the general public who may not have been exposed to these kind of ideas before.
When starting with this project, I looked at a range of artists and works that dealt with both contemporary and traditional methods of creative practice, where neither aspect holds a greater importance than the other. New approaches are combined seamlessly with the conventional, bringing both styles into a new context, therefore producing art that feels fresh yet familiar.
Names include Yannick Jacquet, Joanie Lemercier and more.
I also conducted further technical research, by looking at computer vision, machine learning and neural network techniques.
Udacity Deep Learning course (mainly convolutional neural network section)
Self Organising Map tutorial
I initially thought this would be useful since a Kohonen Self Organising Map is a unsupervised machine learning algorithm widely used for grouping together images (and many other types of data) through similarities.
Design Process and Commentary
I began with an initial list of things my program needed to do:
1. Scan canvas and detect shapes
2. Analyse detected shapes and composition
3. Create new shapes and forms based on how well detected forms fit a desired aesthetic
4. Display new forms on to canvas
I then began exploring methods and techniques used in computer vision, machine learning and neural networks, since these are areas that are used within similar applications, such as image classification/face detection.
When scanning and detecting shapes, I wanted my setup to work like this, which would be the simplest and easiest during the actual use of the program (the painting of the canvases), and meant it would be suitable for live scenarios:
However, I found this was impractical for a few reasons: The camera should not detect anything apart from the desired shapes on the canvas – even with cropping, the distance combined with the noise from a low quality webcam makes for very bad shape detection. At this stage I did not know how computationally intensive the detection and generation of shapes was going to be, so I though committing to a setup and build geared for quick and easy use may have led to problems later on, towards the end of completion, and a live scenario might not have been viable. Also, there was an issue of getting the same perspective between camera and projector, for accurate representation of shapes from both my point of view and my program’s.
Instead, I ended up holding the webcam up to the canvas and pointing it at the desired shape. This worked well, since with the web cam closer up, there is greater detail in the image and hence more accurate contour detection. I also found that a ‘1-to-1’ representation of forms on the canvas and displayed shapes was not necessary at all, which I will come to later.
In terms of the shape detection itself, I started with ofxOpenCV. I used a variety of test images (as the webcam was unavailable at this stage) and results were OK, but contained a lot of noise – rough, jagged edges were picked up all over contour detection. I later implemented the use of smoothing and resampling functions within the ofPolyLine class in openFrameworks, which made some improvements. In the end I switched to the ofxCv add-on, a package with greater functionality than ofxOpenCv, closer to the OpenCv library. Contours detected with this library were much better.
2. and 3.
So I wanted to find the similarity between a painted shape and a set of test shapes, then generate shapes from there. Initially my test set was going to be a collection of images, textures and forms that fit some sort of aesthetic of my choosing. I thought the use of genetic algorithms, such as L-systems, cellular automata or other rule based systems, to generate my test set would be interesting, further increasing the influence of my computer on the work. I assumed this data would be in the form of .jpgs, so implemented various functions for loading and analysing data from image files. Yet as experimentation continued, I was informed about SuperFormula, an algorithm for the generation of shapes, most of which resemble forms throughout nature. The beauty of this algorithm is that such a range of shapes can be created just from the alteration of four individual variables. I also realised that my training set not longer needed to be in the form of imgs, and i can just take vertex positions and feed that straight into my program. I edited and modified a version of superformula in processing, allowing me to create and save a training set.
Finding a value for the similarity, or difference, of two shapes was very tricky, since there are many factors to take in to account, such as size, orientation, location… After much research and testing, I settled on a method using a turning function, as described on page 5 (section4.5) here:
Using this website as a reference along the way:
This function allows you to map the change of angle from vertices along the perimeter of a shape onto a graph. It is then (relatively) easy to calculate a distance, or similarity, between a two or more sets of points (for example, shape A’s first angle change is compared to that of shape B’s and saved, with the same happening for every other vertex. A mean average of similarity is found at the end, and therefore a similarity between two / more shapes).
So now I have a range of base training shapes, each with a similarity to an initially drawn shape. I then looked into clustering, so i could group together similar similarities. However, after more research I found it was quite inefficient to implement most clustering algorithms on 1 dimensional data, my single similarity value, and instead looked to the Jenks Natural Breaks algorithm. This is used for splitting up a set of data into sub groups with similar values ( so shapes with similarities 1,7,15 would be in a different group to those with 150,120 or 110 for example). After my data had been split into sub groups, the most similar group, so that containing the lowest values, were used to generate images.
However, for the actual generation, this was all – the initially generated superformula shapes in the training set were just re-presented. I tried to use the values fed into superformula to generate the most similar shapes as a starting point, and slightly tweak the values and regenerate new shapes. But I found after even small tweaks, variation was too great and any detection of similarity was lost.
I would have liked to include some sort of composition calculation too, yet I found within the time I had this was not possible. Instead, for the paintings produced I tried different methods of composition. One was my own choice of placement of generated shapes, the next was a layering of shapes on top of one another, from most similar to least, and the last I tried distorting the projection itself. As I was painting, I noticed how the projection distorted along with the distortion of the surface itself. This made me think about how my program had influence over the piece through the code, the software, but I am also painting over a projection. How can the hardware and positioning of this affect the work?
Overall, I have learnt a lot from this project. With initially very little understand of machine learning or computer vision techniques, I had to read about and jump over many small hurdles and problems.
I would have liked to have a more accurate measure of similarity, since currently my turning function does not work well with changes in rotation or lots of noise on the contours.
The generation could have been better too, since in the end it is almost just recycling the initial forms giving to it (however, parallels can be seen between this and the age old mystery of what true creation really is, if it really exists, or if we ourselves just recycle and reproduce ideas presented to us). I would also like to implement some sort of colour and composition training and generation. My code is also quite messy, with many functions included that weren’t used in the final painting. This could have been avoided by better planning of exactly what functions and methods I was going to use, with a clear idea of exactly what I needed.
Furthermore, I believe the majority of interest within my project lies in the production, in the technology, leaving the traditional values that I wanted to place emphasis on severely lacking. I succeeded in my plans to push technology out of the limelight, yet it left the public had little understanding of the process of its creation. For similar projects in the future, I would need to spend at least as much time upon the physical, non-electronic side of things as I did on the electronic if I would like them to hold the same importance, as well as finding a way to communicate the process clearer through the work itself (in other ways than a short video documentation on the process).
The Space that Brought Us Here
The space that Brought Us Here is a piece that challenges audiences perception of their surroundings in a more intimate way, though the physical engagement with the piece itself. Screens are used to show sections of the same space. The viewer is able to then reorientate the screens, through movement, creating new-found compositions.
The idea behind ‘The Space that Brought Us Here’ was to challenge a viewer’s understanding of the space around them. As someone walks around a space they form an understanding about how they are operating within their surroundings and how other people are also interacting with the Space. Individuals form their own perspectives and compositions of the space. The piece had 6 tablets suspended on metal steel wire within a wooden frame. A viewer was able to move the tablets up and down the wire, rotate the tablets on the wire. The tablets took video from there rear camera and displayed the video on the screens. However, as all the tablets were connected to the same local area network they could share the video between one and another. A viewer could tap on the screen of the tablet in order to change the video shown to one of the other 5 tablets. This resulted in mismatched video, creating confusion but intrigued about how that perspective had come about.
Intended outcomes & background research
The piece was based on a lot of background research I had done. Particularly with the work of Olafur Eliasson. I was intrigued with him talking about public space with ‘New York City Waterfalls’. Here the waterfalls acted as an intervention in public space. This allowed people to evaluate the space around them. Following on from this Eliasson’s work on the glass façade in the Harpa building also demonstrates this by it’s “three-dimensional quasi-brick structure” that creates “fivefold symmetry” . The sections create shifts in their appearance and colour according to people in the building and the environment. Looking through the facade distorts the view, creating new found compositions. Also with Jeppe Hein’s work, ‘Please Touch The Art’, where he created a mirror labyrinth made out of a spiral of mirrored stainless steel planks set against views of lower Manhattan. The posts are arranged in various arcs that distort the surrounding park and city. I found this piece intriguing with how the mirrors create distortions which created extra perceptions of the space.
I wanted to take this further to explore how individuals could form different understanding of a space, in terms of perspective and how other viewers have an impact on the space that is viewed. Just prior to this I had worked with perspective and perception with Focal Grid which split the same view in to multiple different focus points. Interaction controlled the different focus points that were shown, from a tablet with virtual buttons. From the exhibition of this piece it was clear that the interaction was not always obvious and the viewer didn’t feel connected with the scene that was shown.
I wanted to make the viewer more of a part of the piece through physical engagement with the piece. In the reorientation of the tablets and being able to be part of the video shown. This removes conventional boundaries with how the piece can be interacted with. There is a sense of playful fun, moving different objects independently of one another like a jigsaw, allowing a viewer to see different possibilities with how the objects can be positioned.
I wanted the video shown on the tablets from the outset to be familiar to the viewer. This was in order to augment and intervene with a space they were familiar with in order to challenge there understandings of the space. I achieved this by using live streamed video of the space from the tablets, for the desire to create engagement within the viewers surroundings.
I had decided upon using the Amazon Kindle fire for my project due to it’s relative low cost and running android which openFrameworks could compile on to. I started initial tests by trying to set up of the openFrameworks android examples on a Kindle Fire. This took a lot longer than expected. I found that using Android Studio was a better way of compiling on to the tablets.
I worked on some rotation tests with the accelerometer in the tablet to help me with the rotations of video that would be displayed. I had issues with being able to get the values required to allow continued rotation of the tablet because of the lack of a gyroscope within the tablet. But this still meant that I could impact the rotation of the image displayed based on if the tablet had been rotated left or right.
Following on from this I was not able to get the camera examples for openFrameworks Android compiling on to an Android device. This resulted in the program crashing very quickly after compiling on to the device. I tried this with 2 devices running new and older versions of Android. The solution for this was to use an IP Cam that ran in the background of the tablet which would then stream the video to my openFrameworks app. I used IP WebCam. In order to get the video into my oF app I used the oF addon ofxIpVideoGrabber this took in a list of IP cams via an XML file to display them on screen.
From this solution being used it allowed me to share the video between all of the tablets which worked very well. I got some good feedback from people I showed this too.
I needed to build a frame that would allow the tablets to be suspended within the frame and also, allow them to move up and down and rotate on the wire all with keeping the tablets fixed within the frame. It became clear after talking with Nicky Donald, one of the technical support team, that the frame and the attachments for the tablets would be easiest custom made as we were not able to find anything commercially available to create this. As a result, the frame would be made out of wood and metal steel wire would be used to attach the tablets to the frame.
The attachments would need to be 3D printed to allow them to grip on to the wire and rotate the tablet at that position. I set about looking for 3D designs on thingiverse.com to find the basis of the parts. From this I started modifying the designs, going through tens of iterations of design and print tests.
I was unable to find a suitable case for the for the kindle fires that I was using with my project. I came to the conclusion that it would be best to print cases for the tablets. This came with the advantage of being able to have the attachment plate printed on to the case, saving having to glue resulting in a messy look. However, there were issues with the printing of the cases. It was very difficult to perfect. The cases took around 7 hours each to print meaning that it took a while to be able to make modifications to the design and get all printed at a high enough quality. I also had issues with the rafts and supports that the 3D printer uses to support the print as it is being built. The supports were not consistent in the print due to the complexity of the design, this resulted in weaker prints. I started using a program called MeshMixer, this allowed me to adjust my cased in real world measurements to fine tune the cases. It was also used make the cases optimum for 3D printing. However this still did not solve the issues with the supports for the printing and as the 3D printers that I had access to didn’t always print correctly.
I had issues with the main rotation part snapping when the cases were assembled. This damaged some of the cases meaning that modifications had to be made to them so some were not able to rotate on the steel wire.
I had initially been considering a square wooden frame for my piece. However, through thinking about how my piece would sit in the gallery space I thought I would opt for a portrait frame. This was for several reasons including, creating the look of a window or portate that would frame the tablets to allow for this change of perspective. Also to match the aspect ratio of the tablets within the frame.
The first frame that I build wouldn’t have been strong enough to support the tension from the metal steal wire. Nicky Donald advised me to use structural wood used for building to support the tension from the wire. As a result, I purchased more wood and added metal braces as well as wooden mitred braces for each corner. This resulted in a very strong frame that was capable of supporting the tensioned wire.
For the steel wire, I used wire grippers and Gripples, which allow steel wire to be gripped but adjusted for tensioning of the wire. The wire grippers were put at the top of the frame and the Gripples at the bottom. Pliers were used to pull the wire through the Gripple to tension the wire.
I was really pleased to see people’s intrigue at the opening night. This intrigue initially was from the video being very often missmatched which drew viewers in to interact further with the piece. Mainly viewers rotated and moved the tablets up and down the wire, however usually quite tentative as most seemed to think the tablets look a bit precarious. One interaction I had not been expecting was people to twist the tablets left or right on the wire to see around. This surprised some people as it became very clear then that the tablet that they were moving didn’t always respond to their movements. This was due to the feed shown on the tablet being another tablet within the frame. This discovery lead to alot of viewers interacting with the exhibition further. However, for some the mismatched video wasn’t enough for them to physically engage with the piece.
There were a couple of tablets that were a bit precarious on the frame. This resulted in 2 of them falling off. This did cause issues with viewers wanting to interact with the piece due to these issues from the 3D printed attachments.
The Space that brought us here was a successful piece. I felt like my piece lived up to the original concept of challenging a viewers perception of their surroundings. I had ideas of using the rotation to control the rotation of the image and using an image buffer to create a delay. Although I did not explore these ideas fully, it became clear that this would add too much complexity to my piece and confusion in physical interactivity. Through my solution of using an IP Cam to capture the video instead of using the cameras locally allowed me the ability to share the video between all of the tablets. This created a very subtle shift in perspective but was obvious enough from just looking that the perspective had been altered. I feel that this was better than adding a time dimension or unexpected rotation of the video as I wanted to make the intentions of the piece clear. However, due to the latency of the video stream, there was sometimes a small delay that was noticeable when someone walked in front of the tablets. This was useful to illustrate how the tablets were creating a different perception of the surroundings and allowed a viewer to think about the various operations and alternative compositions occurring within the space.
The issues with 3D printing did hamper the viewers ability to interact with the piece. This was due to my difficulties with the complexity of the printing. It also resulted in some of the attachments looking quite rough. If I was to do that component again I would be hesitant to 3D print the components due to the complexities of it and breakages that are hard to prevent. Using proper cases for a tablet and metal or molded plastic for the rotation and vertical movement would be better. However considering I haven’t seen someone else suspend tablets and have the ability to move them vertically and rotate on steel wire I think my solution was good. More time should have been put towards the mechanics of the structure in order for this to work better.
Overall I am happy with my piece. I negotiated a lot of challenges with the 3D printing as well as issues with compiling software for android with openFrameworks. I feel that my piece striking and allowed viewers to explore the space differently through the intervention of the frame.
References and bibliography
evoNatural is a collage of animations and still images inspired by the forms and patterns found in nature. It shows the patterns and animations evolving over time and branching into the screen.
I started off by looking into stochastic processes. Stochastic is a word used to describe something that becomes unpredictable, to a certain degree, due to a random variables influence. A simple example of a stochastic process would be throwing a dice or flipping a coin multiple times. The outcomes can be guessed but not predicted precisely which as it there are not many outputs in comparison to a stochastic process with a high degree of randomness, as the probability of getting the same result can be near impossible.This then led to me look at the artists Pery Burge, Yoshiyuki Abe and Harold Cohen.
Pery Burge is a British artist who uses paint and ink in water then photographs and videos the process. She used to be a painter but due to a strain injury she couldn’t continue and she then stumbled across the idea of using ink and water as an medium for her art. She believes that ‘nature is the best artist’ and her aim is to ‘reveal its unsurpassed beauty and variety.' At first, she worked at home with a bowl of water and ink and paint. This got her started on her ‘inksplosions’. She first started by trying to get ink moving without having to intervene by using it within water which let the ink flow naturally. Then she became a resident artist at Exeter University where she used the specialised equipment such as a tank to create her work. Later on, she started experimenting with natural light going through glass, calling these ‘Lightscapes’, and taking pictures up close of the movement. She has also looked at the patterns found on soap film and taken pictures as the pattern develops. The inspiration for her work is natural processes and she tries to create pictures and videos that reflect this.
Harold Cohen is a British artist who created AARON, a software cyberartist designed to create its own paintings and art. The program used the rules of objects and body elements (pattern recognition), that were taught through lists, which contained the fundamental rules and relationships between them, to learn how to draw and images were never shown to the program for reference.
The project is split into many different smaller sketches, each sketch making up one element of the final piece.
I started with recreating fractal and recursive patterns such as koch curves and l-systems and then went into creating my own patterns such as the rotating petal pattern found at the endow my piece.
While I was creating the different sketches there were a few problems that occurred. When I was creating the growing tree pattern it was hard to get it to look and work exactly how I wanted it to and it was hard to convert the code from processing to c++. This resulted in the project taking longer than expected. Another problem was how to put all the bits together, it was originally going to have all the animations running in one sketch but this was not possible as this would make the sketch unable to run as the program would be unable to render every animation at the same time. So I decided to video all the different animated elements into videos and put them all together into one video
I used blender to create and put together the final video and it took a while to learn as I had never used the video editor in blender before.
When in the gallery, I got comments about the influences of my work, it was thought that it may have some Japanese influence. Although, this was not the case, I could see how the colours and look of the work could show this. I also got comments saying how the work was quite hypnotic, I would think this was maybe due to the way the rotational patterns move. This was something that drew the audience into the work, along with the bright colours.
Overall it did not turn out exactly as planned, it was made a bit more simple than it was originally going to be. I would have liked, if I had more time to create more videos or make the one video more varied. I could of done better if I maybe made more sketches and created more than one video.
Git Repository: http://gitlab.doc.gold.ac.uk/ecros052/CreativeProject.git
Cycle is a series of technology-assisted performances, incorporating the use of robotics and sound. It was inspired by the interrelating concepts from Graphic Notation and East Asian Calligraphy/Ink Wash Paintings. In each unique recurrence, Cycle explores the theme of spontaneity and individuality transpired within a structured framework as the performers present their own interpretation of a set of instructions.
Each performance lasts approximately three minutes, give or take a minute; the performers end it at their own discretion. During the performance, a sole performer walks around the ‘Ink Stick Rotation Machine’ (ISRM) in a seemingly undefined way. The ISRM grinds an ink stick on an ink stone according to how the performer walks. Ambient sounds and vibrations generated from the constant moving contact of the ink stick and ink stone are amplified by speakers through a microphone located on the sides of the ink stone in real time.
In the performer’s interpretation from a set of rules constructed by the graphic score’s composer, control over the manner of performance is removed from the composer’s authority which alludes to a spontaneous creation of the performance by leaving it to ‘chance’. Unlike music represented in traditional notation, different performances of one graphic score do not have the same melody yet still articulate similar notions expressed in the score. In the cases of Ink Wash painting, the rules in posture, way of holding the brush, and practiced strokes, the results cannot be fully controlled by the painter and are still unpredictable due to human error and the nature of ink and water – their interaction take on a life of its own.
The audience sees and listens – nothing really comes out of watching the performance. Yet, even if the audience does not understand the concepts implicated in this work due to requiring some background knowledge about the act of grinding an ink stick, to experience Cycle, they merely have to practice being in a state of calmness and ambiguity. Just like when a painter or calligrapher prepares ink by manually grinding the ink stick, it is to ebb their flow of thoughts, momentarily forget about the things that are happening outside of the performance and just watch and listen. The performance would be both like a ‘performance’ and a non-religious ‘ritual’ at the same time. The feeling that one would sense is like when one is a non-Buddhist listening to the chants of Buddhist monks. Strangely calming, yet it could get annoying when one listens to an ununderstandable language for too long.
For the performers, I would hope that they would be in a world of their own without minding the presence of the audience and focus on their body walking in a circular path, yet I can imagine that they would perhaps be nervous in front of an audience, especially if they are performing for the first time. As a recurring theme in my work, ‘walking’ is a simple movement that can be of disinterest and a distraction all the same. It not only refers to the bodily action of moving your legs as a mode of transport but also signifies the act of repetition, which is structural, and the mundane. As the performer walks after a few times, the performer may build up a personal routine or choose to walk a different manner each time.
After my research on Graphic Notation and East Asian Ink Wash Paintings, I have drawn connections between these two distinctively different genres in art and show their overlapping characteristics in which my artwork attempts to embody conceptually. I likened graphic notation to instructions that were rather open-ended yet specific in certain ways, hence, I decided on creating a performance that Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink.
Borrowing the motif of ink grinding, which is in itself the stage that happens before the actual painting is executed, and combining it with the imagined sound that graphic notation alludes to, I made the ISRM a framework for the performers. The performers actions are translated to 26 rotations speeds and merely two rotating directions on the ISRM. Within the structure of the ISRM itself, I also found it ironic to have a physically mechanical device replace the mechanical and repeated motions of ink stick grinding. I was unsure of the exact sound that would be produced at the beginning as the sound that is amplified would be quite different from the tiny scratching noise that I am familiar with when grinding ink. With the addition of the sound of the motor, I thought that the sound would be a nice hybrid between the organic and inorganic materials.
In the late 1950s and the first half of the 1960s, many prominent international avant-garde composers such as Roman Haubenstock-Ramati, Mauricio Kagel, and Karlheinz Stockhausen, as well as experimental composers such as John Cage, Morton Feldman, and Christian Wolff started to produce graphic scores that used new forms of notation and recorded them on sheets that were very divergent from traditional music notation in size, shape, and colour. This new way to convey ideas about music alters the relationship of music/sound to the composer and musician. “In contrast to scores with traditional notation, graphic notation emphasized concepts and actions to be carried out in the performance itself, resulting in unexpected sounds and unpredictable actions that may not even include the use of musical instruments.” (Kaneda, 2014)
Here, I focus on how graphical notation evolved from John Cage’s musical practice and then on Treatise, one of the greatest graphical scores, by Cornelius Cardew.
Influence of Zen Buddhism in Cage’s Work
In Cage’s manifesto on music, his connection with Zen becomes clear: “nothing is accomplished by writing a piece of music; nothing is accomplished by hearing a piece of music; nothing is accomplished by playing a piece of music” (Cage, 1961).
This reads as if a quote from a Zen Master: “in the last resort nothing gained.” (Yu-lan, 1952). Cage studied Zen with Daisetz Suzuki when the Zen master was lecturing at Columbia University in New York. Zen teaches that enlightenment is achieved through the profound realization that one is already an enlightened being (Department of Asian Art, 2000). Thus we see that Cage has consciously applied principles of Zen to his musical practice: he does not try to superimpose his will in the form of structure or predetermination in any form (Lieberman, 1997).
Cage created a method of composition from Zen aesthetics which was originally a synthetic method, deriving inspiration from elements of Zen art: the swift brush strokes of Sesshū Tōyō (a prominent Japanese master of ink and wash painting) and the Sumi-e (more on this in the next section) painters which leave happenstance ink blots and stray scratches in their wake, the unpredictable glaze patterns of the Japanese tea ceremony cups and the eternal quality of the rock gardens. Then, isolating the element of chance as vital to artistic creation which is to remain in harmony with the universe, he selected the oracular I Ching (Classic of Changes, an ancient Chinese book) as a means of providing random information which he translated into musical notations. (Lieberman, 1997)
Later, he moved away from the I Ching to more abstract methods of indeterminate composition: scores based on star maps, and scores entirely silent, or with long spaces of silence, which the only sounds are supplied by nature or by the uncomfortable audience in order to “let sounds be themselves rather than vehicles for man-made theories or expressions.” (Lieberman, 1997)
John Cage: Atlas Eclipticalis, 1961-62
Atlas Eclipticalis is for orchestra with more than eighty individual instrumental parts. In the 1950s, astronomers and physicists believed that the universe was random. Cage composed each part by overlaying transparent sheets of paper over the ‘Atlas Eclipticalis’ star map and copied the stars, using them as a source of randomness to give him note heads. (Lucier, 2012)
In Atlas, the players watch the conductor simply to be appraised of the passage of time. Each part has arrows that correspond to 0, 15, 30, 45, and 60 seconds on the clock face. Each part has four pages which have five systems each. Horizontal space equals time. Vertical space equals frequency (pitch). The players’ parts consist of notated pitches connected by lines. The sizes of note heads determine the loudness of the sound. All of the sounds are produced in a normal manner. There are certain rules about playing notes separately, not making intermittent sounds (since stars don’t occur in repetitive patterns), and making changes in sound quality.
Cornelius Cardew: Treatise, 1963-67
After working as Stockhausen’s assistant, Cornelius Cardew began work on a massive graphic score, which he titled Treatise; the piece consisting of 193 pages of highly abstract scores. Instead of trying to find a notation for sounds that he hears, Cardew expresses his ideas in this form of graphical notation, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated (Cardew, 1971). The scores were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes. (Tilbury, 2008)
As John Tilbury writes in Cornelius Cardew: A Life Unfinished (2008), ” The instructions were a guide which focused each individual’s creative instinct on a problem to be solved – how to interpret a particular system of notation using one’s own musical background and attitudes.”
“A Composer who hears sounds will try to find a notation for sounds. One who has ideas will find one that expresses his ideas, leaving their interpretation free, in confidence that his ideas have been accurately and concisely notated.” – Cornelius Cardew
In the Treatise Handbook which guides the performer on the articulation of the score, Cardew writes that in Treatise, “a line or dot is certainly an immediate orientation as much as the thread in the fog” and for performers to “remember that space does not correspond literally to time.” (A Young Persons Guide to Treatise, 2009)
East Asian Ink Wash Painting
The Enso, or Zen circle, is one of the most appealing themes in Zen art. The Enso itself is a universal symbol of wholeness and completion, and the cyclical nature of existence, as well as a visual manifestation of the Heart Sutra, “form is void and void is form.” (Zen Circle of Illumination)
Despite there being many specific technicalities in Cage’s work, these are all qualitative instructions which are open-ended, ultimately leaving it up to the performer’s or conductor’s judgement on how they would play the piece as implied by Cardew’s ideas. In a sense, the individuality of each performance of the graphic score by different performers emerges. This is mirrored in appropriating the creation of the Enso in Cycle by the performer. Every painter draws a circle but every circle is different. Bodily and mindfully engaged in drawing the circle, the circle becomes an allegory of the individual.
The performer not only becomes both the painter and the medium in creating the circle, the performer is also a musician with the indirect control of the device that grinds ink – the instrument with a naturalistic sound created from the contact between the ink stick and the ink stone. To quote Cage’s approach to what defines music, it is the “the difference between noise and music is in the approach of the audience” (Lieberman, 1997).
The act of grinding the ink stick becomes the juxtaposition between the ritualistic and the improvised. Also, ink that is produced after each performance are of different quality each time as no two performances will last the exact same time nor will the performers be able to replicate their performance exactly.
Communication between the phone and the computer is through OSC. The ISRM is made up of an Arduino Uno, which controls a stepper motor, which is directly connected to the computer with a USB cable. The speed and direction of the performer would be measured by the built-in sensors in a phone on the performer. Data from the orientation sensor and accelerometer of the phone is computed in a C++ program on the computer which maps the speed and direction of the performer to that of the ISRM.
Controlling the Stepper Motor with C++
The Arduino part was pretty straightforward as there was the Firmata library for the Arduino that enabled serial communication with a C++ program. However, there was no stepper library in C++, so I translated the Arduino stepper library to C++. Working through the technical details of the stepper motor that I had with some trial and error, this was the circuit that I used to test controlling the stepper motor through a C++ program.
Here’s me testing the program out:
To hold the ink stone, ink stick, and the stepper into a single functional entity, I started off with a preliminary design of a 3D model in Blender, which eventually I was going to 3D print.
I got the idea of the rotation wheel and axis from the driving wheels of steam locomotives, but I was not satisfied with the motions of the rotating mechanism in the first prototype. It caused the ink stick to rotate in a rather awkward manner that did not keep the ink stick facing the same direction. I also removed the water tank as I felt that it was visually obstructive and had no better purpose than to provide the ink stick with water, which I did not manage to figure out a fail-safe method of channeling the water into the ink stone. I thought of using a wick to transfer water from the tank to the ink stone, but water transfer was too slow, or a small hole with a pipe dripping water to the ink stone, but the rate of dripping will change when the water in the tank decreases due to decrease in pressure. Also, it would damage the ink stick if I let it touch the water for too long periods of time, hence I scraped the water tank from then on and decided to manually add water before every performance.
There were many difficulties trying to get the holder for the ink stick to fit. I realised that it was never going to fit perfectly as the dimensions of the ink stick itself was not uniform; one end of the stick could be slightly larger than the other end, which made it either too loose or too tight when I tried to pass through the entire length of the stick through the holder. I resolved this by making the holder slightly larger and added sponge padding on the inside of the holder so that it would hold the ink stick firmly no matter the slight difference in widths. The ink stick was shaky when it rotated so I increased the height of the holder to make it more stable. I also added a ledge on each side of the holder for rubber bands such that the rubber bands could be used to push the ink stick downwards as it gets shorter during grinding.
Before arriving at the final design, there were just wheels that were only connected to each other through the rod. The rotation did not work like expected of a locomotive wheel and I realised that it was because the wheel not connected to the motor had no driving force that ensured it spun in the right direction. Therefore, I changed the wheels to gears.
The printed parts did not fit perfectly and that was not because of the wrong measurements as there was a factor of unpredictability in the quality of 3D printing. I tried using acetone vapour on the parts that need to move independently of each other to smooth the surface, but the acetone vapour also managed to increase the size of the plastic. The plastic became more malleable so I easily shaved them down with a penknife.
This process was too slow and I ended up using a brush to brush on the acetone directly to the plastic parts and waited for a few seconds for it to soften before using a penknife. Super glue was then used to hold parts that were not supposed to move together. The completed ISRM:
I used electret microphones that were connected to a mic amp breakout, then connected to a mixer for the performance. I got an electret microphone capsule to use with the Arduino but I did not know that the Arduino was not meant to be used for such purposes and the microphone was not meant for the Arduino.
So, I got another kind which could directly connect to output as I did not want to use the regular large microphone which would look quite ostentatious with the small ISRM.
Trying to amplify the sound of making ink (sound is very soft because I only had earphones at that time, and I was trying to get the phone to record sound from the earphones):
Sensor Data & Stepper Motor Controls
I initially thought of creating an android application to send data to the C++ program via Bluetooth, but there was the issue of bad Bluetooth connectivity, especially the range and speed of communication. Hence, I switched to using OSC to communicate the data. Before finally deciding on using an OSC app, oscHook, I made an HTML5 web application with Node.js to send sensor data. It worked well except for speed issues as there was a buffer between moving the phone and getting the corresponding data that made it rather not ‘real-time’, and it also sent NaN values quite often which would crash the program if there were no exception handlers.
For controlling the speed of the stepper motor, I mapped the average difference of the acceleration of the y-axis (up and down when the phone is perpendicular to the ground) within the last X values directly to the speed of the motor. Prior to this, I looked at various ways to get the speed and direction of walking, from pedometer apps to compass apps. As different people had different sensor values with the phone, I created a calibration system that adjusted the values of the mean acceleration when the performer is not moving and when the performer was moving at full speed. This ensured that the stepper will be able to run at all speeds for all performers.
Link to Git Repo.
Performance & Installation
Videos of performances were playing on the screen for the second day of Symbiosis. The TV was covered with white cloth on the first day. The ISRM was placed on a white paper table cover with the microphone next to it.
Instructions for Performers
Besides having to run a calibration before their performances, I requested the performers to wear “normal clothes in darker colours” to make a contrast with the white room walls. I decided not to specifically ask for black as it was too formal and intimidating. Although the performance exudes the sense of a ‘ritual’, it was not meant to be solemn or grievous, as was such cultural connotations of fully black clothes in a rather ritualistic setting.
During the performance, the performers were to heed these instructions:
- Walk around the room.
- When you stop, stop until you hear the sound indicates that the motor is at its lowest speed.
- End the performance when it is three minutes since the start.
Prior to completing the program that controls the stepper motor, I wanted to attach the phone to a belt and hide it under the clothes of the performers such that they would be walking hands-free. I realised that it was quite abrupt to merely end the performance with the performer standing still as there was no indication if the performer was pausing or stopping entirely to the audience. Hence, after realising that by placing the phone parallel to the ground caused the motor (and in turn the sound) to stop in an elegant manner, I decided that the performer would hold the phone (which I covered in white paper to remove the image a phone) in their hand and have them place it on the ground to signify the end of the performance.
There was a total of eight performances by three people, Yun Teng, Leah, and Haein. These are videos* of the performances by each of them on the Symbiosis opening night and their thoughts on their experience of performing:
*The lights in the room were off during the day, hence videos of the earlier performances look quite dark. If you do not hear any sound from the video, please turn up the volume.
“Being asked to perform for this piece was an interesting experience. For me, it was seeing how (even on a conceptual level, as it turned out) that my physical movement can be translated through electronics and code into the physical movement of the machine and the audio heard. Initially, although we were given simple instructions to follow and even, to some extent, encouraged to push these instructions, I was at a loss to how to interpret them, and just walked in a circular fashion. I tried to vary the pace, speed and rhythm of my walking in order to create variation, but ultimately fell back into similar rhythms of fast, slow, and fast again. It would have been interesting to perhaps push this even further if the machine was more sensitive to height changes, or arm movements – as a dancer who is used to choreography, this was a challenge for improvisation and exploration. In addition, due to the size of the room, the space was limited and hence the walking could only take place in certain patterns.” – Yun Teng
“At first, the walker was uncertain, distracted and anxious. She explored the link between sound and her unchoreographed strides and expected the connection to be instantaneous and obvious. However, it was not. There were delays and inconsistencies; the electronic and mechanic could not accurately reflect the organic. A slight panic arose from the dilemma of illustrating the artist’s concept to the audience and accepting its discrepancies as part of the performance. Slowly she started to play around with the delay, stopping suddenly to hear the spinning sound trailing on, still at high speed, and waited for it to slow down. Rather than a single-sided mechanical reaction to movement, the relationship between the walker and the machine becomes visible and reciprocal. Rather than just walking, now she also had to listen, to wait, and by doing so interact with the machine on a more complicated level. Through listening, she felt the shadow of her movements played back to her by the machine. The observation sparked contemplation on the walker’s organic presence versus the machine’s man-made existence and the latter’s distorted yet interesting reflection of the former.” – Leah
“The whole practice first was received as confusing and aimless as there was too much freedom for one to explore. It was challenging to perform the same act (walking/running) for more than two minutes. At first, I performed more than four minutes, unable to grasp the appropriate time, but it decreased as I repeated the practice. This repetitive performance was quite meditative and physically interactive with the work that caused me to wonder about the close relationship between myself and sound piece (which changes according to my walking speed). The most pleasant part of the performance was that I got to control the active aspect of the work and directly interact with it.” – Haein
The audience was very quiet, probably so that they could hear the sound that was very soft even at its loudest. When they first came in, they did not know what to do as there was no visible sitting area (so I directed them to sit at places that allowed the performer to roam most of the room). It was a huge contrast to the audience that interacted with my previous work as only the performer gets to have a direct interaction with the ISRM. Even then, the ISRM was visibly moving during the performances.
Just hours before the opening night, the ISRM broke at (fig. A & B). It was a mistake on my part as I was reapplying super glue (fig. B) to the base as it had somehow loosened from the previous application of super glue. In hindsight, I did not make extra parts (I did print extras of certain, not all, parts but they of no use when I did not bring them on site, nor were they ‘acetoned’ to fit together.), could not manage to salvage the parts, and I knew that I would not be able to reprint the parts in time. In the end, I slightly altered my work as the ISRM could no longer function as intended. Instead of having the microphones stuck to the sides of the ink stone, I stuck them to the stepper motor instead. Although the sound no longer had an organic element from the ink stick and ink stone, it was completely mechanical now.
After undertaking this project, I have learnt not to limit myself by my tools, but to explore different methods and tools before limiting myself in the creation of the work. I had a misconception that 3D printing was the most efficient way. In some ways, it was because it was the printer that was doing the hard work, not me and I did want to try 3D printing. Despite that, I should not have limited myself by my lack of consideration in using other materials to build the ISRM, such as the traditional way of putting together wood and gears. On the other hand, I do not regret my attempts to build an android app (which I quickly decided was not worth my time for the simple thing I was trying to accomplish) and a web application for sending the sensor data from the phone with Node.js as it is something new that I learnt even though I did not use it in my final work.
Fortunately, I managed to finish the design of the ISRM and print it out in time, but I felt that I should have focused more on the ISRM instead of coding in the earlier phase of the project timeline. 3D printing takes a lot of time, as I have experienced through this project, and any botched prints needed to be printed again as they are rarely salvageable even after being in print for hours. It is also tricky to get the settings right (i.e. infill) such that the printing time is minimised without compromising the quality.
Apart from the many technical things, I also learnt how to organise a performance art (this is my first performance art) and through making this artwork, there many more implications and questions that arise from what I created. For the performance, there were many things to keep track of, such as rehearsing with the performers beforehand, the attire of performers, the schedule of performances, getting the camera to film for documentation and managing the audience. In conclusion, despite being unable to carry out the performances as I have originally planned, I am glad that I have managed to still put together what is left of the entire work even when the ISRM failed to work correctly and the original intentions behind the artwork are still largely intact.
References & Bibliography
Works Cited in Background Research
A Young Persons Guide to Treatise. (12 December, 2009). Retrieved 2 November, 2015, from http://www.spiralcage.com/improvMeeting/treatise.html
Asian Brushpainter. (2012). Ink and Wash / Sumi-e Technique and Learning – The Main Aesthetic Concepts. Retrieved 2 November, 2015, from Asian Brushpainter: http://www.asianbrushpainter.com/blog/knowledgebase/the-aesthetics-of-ink-and-wash-painting/
Cage, J. (1961). Silence: Lectures and Writings. Middletown, Connecticut: Wesleyan University Press.
Cardew, C. (1971). Treatise Handbook. Ed. Peters; Cop. Henrichsen Edition Limited.
Department of Asian Art. (2000). Zen Buddhism. Retrieved 11 December, 2015, from Heilbrunn Timeline of Art History. New York: The Metropolitan Museum of Art: http://www.metmuseum.org/toah/hd/zen/hd_zen.htm
Kaneda, M. (13 May, 2014). Graphic Scores: Tokyo, 1962. Retrieved 2 November, 2015, from Post: Notes on Modern & Contemporary Art Around the Globe: http://post.at.moma.org/content_items/452-graphic-scores-tokyo-1962
Lieberman, F. (24 June, 1997). Zen Buddhism And Its Relationship to Elements of Eastern And Western Arts. Retrieved 10 December, 2015, from UCSC: http://artsites.ucsc.edu/faculty/lieberman/zen.html
Lucier, A. (2012). Music 109: Notes on Experimental Music. Wesleyan University Press.
Tilbury, J. (2008). Cornelius Cardew (1936-1981): A Life Unfinished. Copula.
What Ink Stick Should You Choose For Japanese Calligraphy? (2015). Retrieved 3 December, 2015, from Japanese Calligraphy: Modern Japanese Calligraphy inspired in Buddhism and Zen: http://www.theartofcalligraphy.com/ink-stick
Williams, M. L. (1981). Chinese Painting – An Escape from the “Dusty” World. Cleveland Museum of Art.
Yu-lan, F. (1952). A History of Chinese Philosophy. Princeton, New Jersey: Princeton University Press.
Code References & Software
Door To Door
Door To Door
Door to Door is an interactive explorable three-dimensional environment that is experienced through a virtual reality headset in order to fully immerse the audience in various eerie locations featuring strange and uncanny themes.
The environment is controlled using the handprints on the table which switch between the scenes inside the oculus headset. This is done through the use of conductive paint which sends signals to an Arduino which in turn sends a signal to the unity game engine, displaying it on the Oculus headset.
Door To Door was created with the intention of being an installation piece that would be displayed in a digital arts gallery. With this in mind the content of the environment would have to be suitable for any members of the public and artists who would be likely to enter a digital arts exhibition. This could range from around mid teens to thirty years of age. Viewers in the exhibition would also be drawn to the piece because of the use of virtual reality which, even with its recent surge in popularity, is still a niche product. Nonetheless, the work was built with a more mature audience in mind, who would be able to appreciate the themes and structure of the environment in comparison to the conventional fast-paced video game action.
Initial Ideas And Intended Outcomes
Before doing extensive research into level and design structure, the piece was designed to be a fully explorable world with a controllable first person camera. Movement and character placement would affect the structure of the world around you such as in video games like DMC: Devil May Cry:
The effects and actions would be based around the themes of distortion and discord, as if the environment was in an unstable state with the user being directly responsible. Creating a composition alike to this would have taken excessive amounts of time to produce, especially creating something that would be as effective and well designed as the video above. I feel the piece would also have no objective and be labelled as just another game rather than something that has a deeper meaning.
However I still went on to created a virtual environment with eerie themes but the controls in the final product were minimal and bound to only head movement and teleporting scenes. This is so that the user would not be too distracted by the interaction of the piece and could focus more on the themes and elements of the places displayed. With this in mind I still wanted the position and view that the camera displayed to affect the objects displayed on screen. I wanted to do this through ray casting and making certain elements disappear or reappear using this method.
The idea was to recreate the room in such a way that it would be both similar and different to the original, with the actual room being a bit more bland than the virtual space. The wooden table would be in both the virtual and the real world, acting as the Door. The only difference would be that there would be people sitting on the chairs inside the virtual space. This adds an ominous feeling that there are people there with you but simultaneously there isn’t anyone there in reality. Along with a séance like theme the piece would put people in a state of unease through subtle small changes. Although it is not clear in final product, the people sitting round the table are all staring at one another including yourself in different scenes. This was to add to the creepy atmosphere. Originally there was a sequence between the levels but this idea was scraped.
My background research focused on different artists who had themes of surrealism and disruption in their works. I began researching Surrealist artists such as Max Ernst’s landscape pieces and Vladimir Kush, taking on board the symbolism and general integration between two separate themes in the pieces. After researching these surrealist artists I discovered photo manipulations and further studied artists such as Nancy Fouts and Erik Johansson, trying to find ways of incorporating their ideas into my work.
Later on in my research I also studied the Uncanny Valley. This is a concept in which an image or object has a slight resemblance to that of a human, because of this the viewer both recognises and partially understands the image but at the same time is repulsed and experiences unease making the image or object weird or creepy. Examples of this would be zombies of dolls, they are partially recognisable as human but are distorted in looks. I kept this concept in mind throughout the design process of the scenes, as I wanted to create an eerie atmosphere without using the cliché horror techniques such as jump scares and monsters. I based these areas around the AI used in the game Sherlock Holmes Nemesis in which the NPC Watson will teleport in front of you when he moves off screen. As displayed in this video, alongside a dramatic soundtrack, which was also incorporated in my piece, we can see that there is a unnerving feeling every time he appears around you.
Another area of interest that helped me shape my ideas were the warped environments created in video games such as the ones designed by Alessandro Taini in Enslaved: Odyssey to the West, DMC:Devil May Cry and Heavenly Sword. From games like these I briefly looked into video game level design, looking at the methods artists use to coerce the gamer into going in specific directions. An example of this would be leaving a open space thats more brightly lit than other paths so the player instinctively moves down that one. A more basic example would be making items glow so you know to interact with them.
Building and Design Process
Even with the idea in mind the process of creating this piece took on many forms and many ideas were scrapped. As mentioned above, the piece was originally meant to be a fully explorable setting with controls and a place that was procedurally generated. The world would have been designed and created using openFrameworks with libraries such as ofxRules and AssimpLoader alongside uses of shaders and a first person camera library.
Before the use of the Oculus the composition was originally planned to be developed on the more affordable Google Cardboard. I went on to create a working prototype using the Unity game engine. This allowed me to create my first testable Google Cardboard environment on an android mobile. However the piece was basic and not procedurally generated.
After I understood the scope of the project I realised that the piece was a bit too ambitious with the time constraints and attempted to rework the concept. Without abandoning the idea I went on to narrow it down, removing features such as player control and the procedural generation to focus on more important aspects of the art piece such as the principles of disorder and strangeness and how these would be presented and interpreted by the viewer.
During the development period I did not consider using Unity as the main system because of the lack of C++ support. My original plan was to create an environment using OpenFrameworks and export the file to a mobile device.The problem however was the issue of exporting an Xcode file as a executable for android or iPhone that would work with Google cardboard. Creating an android application this way would require a wide range of programs such as Eclipse and additional plugins and coding. The other programming options at the time were Unreal Engine and Unity. Unlike Unity, Unreal Engine had supported use of C++ while Unity ran on C# scripts.
Because of the lack of C++ support I began creating my project on Unreal Engine. Using the engine I was able to create a realistic model of the exhibition room using photos as sources. I made sure to get the dimensions correct and the positioning of the camera in the location in which the user would experience the piece. When recreating the room I had to keep in mind different textures and objects that were both available and essential in creation. When people see something that they recognise they are drawn to small distinctive features. I had to pinpoint specific features around the room to make it recognisable to the viewer. Even though this was the main point of the table other features also needed to be present. Objects such as the wiring poles along the walls, the computer desk and the TV were also included as static but distinguishable objects in the real world. Further detail was added such as the Goldsmiths logo and the same clock.
Exhibition Room Panorama
Early UE4 Room Structure
During the process of creating the basic prototype in Unreal Engine I had decided at this point that using the Oculus Rift was the most ideal way of immersing the viewer into my environment. At this point I had already scoured the internet for ways in which I could develop a Google Cardboard application from Unreal Engine. Unfortunately after much research and posting on forums I realised that it was not possible and I was more likely to have luck with the Oculus Rift. And even if I was able to develop my own UE Google Cardboard library there would more that likely be bugs and other problem present, alongside the fact that I also had to create my world.
Original Physical Design (before exhibition room)
The design concepts for Door To Door included the uses of a pressure pad or some kind of touch motion on the table. Using an Arduino was the next step in order to add physical interaction to piece. After researching many methods in Unreal Engine plugins I struggled to find a suitable way of conveying information between the Arduino and Unreal Engine. I then tried to send OSC signals between the two programs, trying to use it as a link.
Along with the steep learning curve of using a new game engine, I was also unable to successfully find a means of displaying the scenes created in an Oculus Rift. With only a few weeks remaining and the project at a standstill I decided to, once again, switch to a new program. This time I wanted to use Unity. Although it does not support C++, Unity is able to run C# scripts and is known to be easily compatible with the Oculus Rift, fixing both my problems together.
The room created in Unreal Engine was firstly exported a .FBX file so that it could then be imported into Unity as an asset. The model needed to be scaled down and textures had to be applied. After creating a suitable terrain to place the model in I then used photos taken inside the room to re-create it.
Unity Room Recreation
Using the Oculus Rift for test runs was also a success. This led me to create more scenes with different variations, some not included in the final piece.
Another essential step was to include interactivity into the piece. Instead of using a pressure pad for the hand controls I decided to use Conductive paint. I built the conductive paint into a simple Arduino circuit, every time the paint was touched the resistance in the circuit would increase and the output would be displayed on the program. using an if statement I used this to send a number to Unity using serial ports. This would be received using a C# script in Unity. The video below shows me using the conductive paint to move an object in the room.
Using the same code from above I then modified it so that instead of moving the cube a new level would be loaded. The shape of the ‘buttons’ on the table were also important. Instead of having a regular button or small area to touch with one finger hand prints were used instead. This way the user feels more connected to the characters in the scene, as if they really are taking part in a séance and not just playing another video game where joysticks control their movement. I believe that this is one of the aspects that separates my piece from being another video game experience.
To add to the atmosphere of the setting, audio was an important aspect of the piece. I began experimenting with cymbals being played backwards and low humming sounds as well as suitable transition sounds like a camera shutter, nothing too abstract or distracting. With very limited knowledge in creating audio I used GarageBand to manipulate and edit the sounds.
For the background music the soundscape synthesizer as used. I changed instrument used depending on the scene but kept the notes that were playing the same so there was a low humming noise in the background throughout. The only scene without the background music was the original empty recreation of the exhibition room. I felt that this needed to be silent as it gave the scene its own sense of unease, as if the user is waiting for something to happen.
The scene transition sounds, although it may sound like a camera it is actually based off my work from last year. My piece Recursive Flaws featured various typing sounds, rerecorded again and again. I used one of these sound files and reversed the sound and shortened it to create an eerie mixture of backwards typing as well as a shutter like sound.
Setting up the exhibition was not too difficult. The only challenged were hiding the laptop and the Arduino away from public view. The laptop was placed inside a plinth in the corner of the room and the Arduino was glued and placed inside a container under the table. The Conductive paint was applied directly onto the table.
Arduino Placed Under The Table
Conductive Paint Applied To The Table
Door To Door
- The first issue was the constant change in different programs, first being OpenFrameworks and the Unreal Engine. This would have been a bigger problem but swift action and converting the work (such as the models) over to Unity made the process a success. Even though the lack of coding in the final piece was also a problem, I had still been coding throughout the project even if it was not present in the final product.
- The conductivity of the paint varied depending on the material that it was placed on. Originally there was a mirror plate covered in the conductive paint for the hand pads. The plate did not work properly and caused more errors. This was amended by putting the paint directly on the table as wood seemed to be the most successful material.
- Problems with the serial ports and the information sent and received from the Arduino caused more errors than needed. They caused the scenes to run through continuously. At first I thought that there was a problem with the coding and Unity receiving too much information. I spent a lot of time working on a script that had a delay time between receiving signals however the problem persisted. By the time I realised that the problem was not only with the conductive paint and mirror plate but also the wiring it was too late to adapt the code. I used my C# failsafe which was not perfect but fixed the problem of the Oculus running through all the scenes consistently.
- The next problem branches off from the last one. The loading times between scenes caused confusion for the viewers. They thought the program had crashed. The sensitivity of the conductive paint also affected the interactivity of the piece. The fact that people could not hold there hands down even though there were hand prints on the table left them confused. I had to manually explain to them to touch the paint only once.
- The calibration in the program was slightly off. This caused some scenes to be slightly off angle. Although this was fixable manually many of the viewers enjoyed this happy accident and said that it added to the sense of confusion and disorientation.
Although the exhibition piece was not exactly how I imagined it to be when I first came up with the concepts I believe that it is was a success but not without its weaknesses and areas of improvement. Even from the second overhaul of the idea, there was a lot missing such as the ray-casting techniques and animated moving AI. On the other hand with the three changes in the programs used and the time constraints, I think that being able to incorporate these elements would have been highly unlikely with my amateur skills in using these programs. The original concepts that I developed were also a bit too over ambitious. Creating a whole gaming world itself is a lot of work, in both designing it as well as working on it, not to mention the original plan of also decorating the physical surrounding to a new room inside the exhibition room. In the future I will consider the time we are allocated to produce the work as well as the level of skill that I have in certain programs when designing new pieces.
The amount of coding in the final product was also of a small amount. To avoid a situation like this I must do more thorough research into both my ideas and the programs I am planning to use before diving head first into the project. The errors and problems with the interaction were one of the major weaknesses in my project. Although some of them turned out for the better I need to be able to find and correct these problems before it is too late. Whether that is more viewer feedback and tests or setting up earlier. Problems such as the loading times could potentially be fixed (such as putting all the scenes in one instead of loading them individually) with experimentation and time.
On the other hand, the audience were mainly impressed with the piece and were fascinated by the evolution and realness of the program, saying that they almost forgot that they were sitting in a room in Goldsmiths. This was a pleasant outcome as taking the viewer away from the current reality was one of my goals. People were also intrigued by the themes and structure of the piece. Many of them asked if there was a relevance between the order in which the scenes appeared and who or what was happening. They seemed to enjoy trying to decipher a meaning or find an understanding. The success was reflected by the queues of people waiting to experience it. The piece also gave people an opportunity to experience virtual reality which, without a lot of money or enthusiasm, they would not normally have encountered. The experience for them in itself was enough to satisfy. It has also shown me the importance of interactivity in a gallery art piece. People are more likely to be engaged if they feel a part of the piece and not just controlling someone else like in a video game.
Overall I feel that Door To Door was a success. It definitely has its faults and has much room for improvement but it has taught me valuable skills in not only programming and coding but also of the audience in galleries. There were many challenges throughout the project but the reactions of the spectators showed its strength as a piece.
Used in Research:
Addons in Research:
Videos and Images:
Sherlock holmes Video:
DMC: Devil May Cry:
Psychological ‘disintegration’ is a subject that fascinates me, on both a personal and conceptual level. The loss of mental cohesion, or emotional deterioration can be brought on by many experiences, substances or neurological abnormalities. An area that was of particular interest to me was the loss of sense of self, through experiences such as depersonalisation, body dysphoria / dysmorphia or prolonged mental illness. Whilst researching these psychological phenomena and reflecting on the part they have played in my own life, I began to define my concept and ideas.
Looking for Inspiration
At this point in my process, I looked for artists that have explored the sense of self and identity in a way that resonated with my personal experience. The photography of Francesca Woodman really struck a chord with me; her evocative self portraits use the manipulation of shadows to distort the human body in a way that is uncomfortably beautiful. The candid way in which she places herself within the frame allows the viewer a glimpse into her mindset at the time, whether showing her to be seductive, energetic or calm, and contain a knowing sense of humour.
In addition to this, a music video directed by Walter Stern and Jeff Richter for the Nine Inch Nails song ‘Into The Void’ caught my attention due to it’s use of macrophotographic shots of body parts and it’s non narrative way of communicating the emotions contained within the song (“Tried to save myself but myself keeps slipping away”). The symbolism of body parts taken out of context is, to me, a powerful way of expressing the loss of connection to reality; our physical selves are the parts of us which interact with the real world, whereas our mind can wander and imagine. As the body is taken apart and used simply as abstract shapes and colours, the only thing ‘grounding’ us is elevated. The video visually connects these shots of body parts to ‘losing yourself’ and the eventual destruction scene mirrors the effects slow disconnection from our identities can pour into the breakdown of our psychological state.
Above; Some of the images I took whilst documenting my mindset and sense of self throughout the process of creating this piece.
Development of my Concept
Visiting the space in which I would be displaying my work furthered my initial concept, and widened my research significantly. Above the wall in which I was intending to hang my prints stood an imposing symbol of religious faith; a large stained glass window depicting Christ on the cross, surrounded by saints. I began to think of the stories contained within religious texts, and the recurring themes of protection, possession and surrender to forces outside of your physical self. Many of these could be interpreted as ancient, literary metaphors of mental illness.
Throughout history, symbols of religion have survived longer than many of the texts and paintings; often being re-appropriated by many different religions. You could argue that this is because they are simple and easy to recreate, yet striking and intimidating. They are commanding to those who know their meanings, but still remain beautiful to those who don’t. To fill the space that I had been given, I knew that I would have to create something that is not easily overshadowed by the window and that also makes good use of the large amount of blank wall space. I developed an idea; create my own set of symbols that start as powerful patterns and devolve into simple, fragile and uneven shapes as they approach the floor. On the floor below would be a pile of the photographs I had previously taken, discarded at the bottom of a fading symbol of protection.
Above; Sketches from my research into religious iconography and meanings, and the shapes used in the architecture of churches.
Creating My Symbols
Using Processing, I started creating programs that built generative patterns and then destroyed them. My first few attempts were unsuccessful; the patterns that were created would not translate well onto the wall, and were not simple enough to be recreated in a more ‘manual’ way. At this point, I had decided that I would be using the most basic materials possible to build my installation – nails, a hammer and rope – to contrast against the use of technology in their creation and to make it apparent that manual labour had been involved in their construction. The materials were also a reference to the historic iconography that sat above them.
Above; An early attempt at making a pattern that builds and then reverses, using Processing.
I gradually, through trial and error, simplified my designs. Taking reference from symbols that I had previously looked at, such as the Pentacle / Pentagram, Star of David and the Nine Pointed Star, I created a program that built symbols based around star, triangular and rectangular shapes using one continuous line. Within this program, the line moves up and down the screen in an unstable manner as it churns out various patterns, until finally it goes to nothing. For the sake of capturing the patterns that best represented my theme, I lowered the frame rate and took screenshots of possible choices for my final piece.
Creating my Final Piece
Above; The series of symbols I selected from my Processing sketch. Using Adobe Photoshop, I arranged the symbols into a 3 x 3 grid, marking out all of the points in which the nails would go and numbering them. I was aware of the positioning, trying to ensure that the concept of the grid works in rows from left to right, and also in columns.
Above; Using the template I had created, I transferred one symbol onto a piece of material, using pins and thread. Here, I wanted to see how I could expect the pattern to work visually, using different colour threads to experiment. I eventually decided on red, as it created the greatest amount of contrast against both the background and the nails. It also connotes impending danger, whilst being the most ‘human’ of the primary colours.
Above; My first attempt at creating one of the symbols on a wall using rope and nails. Testing different types of rope, choosing the one that created the best visual effect and was durable enough to hold up without fraying. Tying off the ends of the rope without disrupting the pattern proved a challenge, but with time, I found a technique that worked.
Above; The start of the build. Using a projector to display the grid of symbols on the wall allowed me to mark out the corners and position everything correctly. Here, I had to achieve a balance between size, impact and practicality, and make sure that everything was spaced evenly.
Above; In addition to my original photographs, I took photographs of parts of the rope installation to connect the two parts of the piece. In these images, I used similar techniques (unconventional angles, close ups, black and white film effect) to decimate the rope in the same way in which I had decimated my own self portraits.
The End Result
The finished installation was comprised of a 3 x 3 grid of patterns, measuring 3 square metres in total, built with rope and nails. The photographs were arranged in a messy, horizontal pile underneath the piece, only becoming fully visible as you stepped closer to the wall.
Overall, I was extremely satisfied with the way my piece looked in the gallery space, I felt that the scale of the patterns used up enough of the space to draw attention. I am happy with my decision to create an installation instead of just displaying photographic prints. My initial worry of being overshadowed by the window was also eased; the two things complemented each other, rather than competing for the viewer’s attention.
My choice not to have a description on the wall, only the title, meant that I was given honest feedback on the public’s interpretation of the piece. This was a great experience for me, and an immediate way to get feedback on how successful my ideas were. On the other hand, I think the complexities may have been conveyed better with a minimal description of my ideas / methods.
Looking back on my process, I think I could have been more methodical in creating my piece. I struggled with the programming aspects as I did not have a complete clear idea of my aims when I began to use Processing. A large part of the work that I did was experimental and not used in the final product, meaning that I had less time to refine my concepts. Although I had quite a few strands of ideas intertwined in the installation, many of which I am sure didn’t come across to the audience, I was not too concerned about conveying my exact intended meaning. I think the piece had enough initial visual impact, and connection to familiar imagery, for the viewer to interpret the piece in their own way.
Link to my code; http://doc.gold.ac.uk/~rdunn012/uploads/Disintegration.zip
Book; ‘Dissociation and the Dissociative Disorders: DSM-V and Beyond‘ by Paul F. Dell, John A. O’Neil, 2009
Book; ‘Madness Explained; Psychosis and Human Nature‘ by Richard Bentall, 2004
Francesca Woodman; http://www.tate.org.uk/art/artists/francesca-woodman-10512
‘Into The Void’ directed by Walter Stern and Jeff Richter; https://vimeo.com/61997644
Pentagram Meaning; http://www.witchvox.com/va/dt_va.html?a=usma&c=basics&id=2875
Book; ‘Learning Processing‘ by Daniel Shiffman, 2015
Book; ‘Cultural Sociology of Mental Illness: An A-Z Guide’ by Andrew Scull, 2013
noFaceForNames is an installation created to highlight various issues I care about and as part of an overarching project of mine (Future Soon). It is an exploration of life extension by recording of data and the issues of the use of the data. The form of the installation eventually ended up being a hanging buckminsterfullerene-like sphere with a small thermal printer printing into it, with headphones also hanging inside. The audience were invited into the sphere (one at a time) which was around 1.5 m diameter, to read the print out and listen to sounds from the headphones.
AR with L-Systems
Throughout our project we researched into Augmented Reality (AR) and our resulting final product was an A5 zine which uses a marker to augment the page through the generation of objects. A ‘natural’ scene is generated covering the marker consisting of an oscillating plane and procedurally generated ‘trees’. Through using AR and L-System generation addons for openframeworks, ofxAruco(1) and ofxRules(2) which have dependencies on ofxOpenCv,ofxCv and ofxXmlSettings and facilitate the recognition of the marker and thus place the generated object within the right dimensions of the users’ scene, we were able to realise an individual and changing AR scene for each marker. We intended for the project to consist of procedural themed scenes that would vary for each page in order to show the book pages coming to life and consisting of different scenes. The project wanted to include interactions for each scene by bringing in outside influence from the user, but we instead decided on a ‘regeneration’ button which updates the plane with randomly generated values and begins the steps of the L-System growth again. The project seeks to have users experience a procedurally generated scene through the pages of a booklet, by giving different augmented experiences.
Film documenting user interaction with the final Zine produced
Intended Audience/ Background Research
The projects intended audience wasn’t targeted towards any particular group but rather to convey an experience into a new form of media. The intended audience is for anyone wanting to experience a type of media that is not constrained to the pages of a book which led to some initial research into markerless AR from Disney Research(3) as this would have allowed direct physical interaction and potentially creation of the marker itself. The audience isn’t targeted based on the environments presented, but rather the concept of transforming two-dimensional paper media into three-dimensional environments. This was influenced by the work of painter Austin Lee who brought his prints to life through the use of AR(4). The intended audience for our project will vary from users of all ages, young children can experience the project as an interactive book to enjoy or for adults seeking to understand and experience augmented media. We hope that any age a project of such concept can seek to be informative but also enjoyable to use, we believe that any user can easily understand and use the project despite having any knowledge of the technologies being used, therefore allowing them to experience a whole new form of media with almost no knowledge or input needed.
Initially the project was intended to augment and procedurally generate objects within the user’s intended scene by using a camera to change their current location into a procedurally generated one. The project initially was to invoke an experience of augmenting a user’s surrounding, thereby allowing the user to fully relate to the scene being presented. As the intended design changed to exist upon a page of book, the original invoking experience stayed true but rather to a lesser level. Originally the concept was to use markerless AR in order to camouflage the actual markers that initiate the scenes, due to limitations of the addons we used, we had to change the design to include more visible markers.
Example of AR prototype and markers provided by ofxAruco
As the design process moved forward we initially intended to include multiple markers within each scene to invoke a more immersive experience, but after experimenting with multiple markers we felt that having one distinctive marker allowed us to focus upon one themed object within each scene. ofxRules comes with inbuilt xml files which step through the generation of L-Systems and as such we were able to easily include dynamic and animated L-Systems. We intended to use physics and created programs to do so but as we began to bring in the L-Systems we realized that the physics did not easily comply with the ofxRules addon. Therefore we showcased the L-systems as living by allowing them to recursively generate within each scene, and accomplished a similar ‘natural’ state to the physics implemented but through a change in the design concept.
Click to play gif of L-System growing
Originally, the concept was to allow lines that represent the two-dimensional version of the scene upon the paper media to interact with the augmented version. This would allow the use of textured shaders differently within in each augmented scene allowing for multiple shaders to be present upon each page. Instead we found that to create an oscillating plane that coordinated with the generated texture colours the shader could be used to alternate the y values across the plane and the input of the colours could be decided through noise and sinusoidal functions as well as being influenced by the marker recognised.
Click to play gif of oscillating plane created through textures and shaders
We decided to create a Zine to allow us to have more of a physical and single viable product to work towards. The three markers, and designs featured, produce a surreal digitally rendered scene that existed alongside the printed version of itself. Mockups of the zine were created which included several markers and different designs however, we decided that a single printed marker along with the texture coupled with the AR worked best.
Following the unsuccessful efforts of implementing ofxARToolKitPlus (another AR addon for openframeworks), we tried the application using older verisons of openframeworks but this led to other complications and thus we decided to use ofxAruco. ofxAruco is easily implemented and automatically calibrates the webcam information and configures the marker information using the provided ‘intrinsics.int’ and ‘boardConfiguration.yml’ files provided. From this point the use of the AR and implementing objects within the scene was accomplished relatively easily with slight adjustments to the scale of the objects created.
We decided to augment the pages of the book with digitally rendered scenery which took influences from nature both in its production, L-Systems use recursive algorithms found in nature, and in its aesthetic. The xml files provided by ofxRules allowed for the recursive drawing of trees to be easily implemented and so we wanted to create a ‘land’ for them to sit on that was had a similarly ‘natural’ animation. The shader examples provided by openframeworks include an example of displacing a plane primitive using the shader(5) but use a grayscale image as the texture. We adjusted this example to work with colour textures by using the green channel, which translates to the y position of the vertex, to update the y position of the plane creating a smooth, undulating surface that moves physically relates to the darkness of the colour created. The oscillating surface is created through generated 3D noise across the pixels of the textures which translates to the 3D noise of the plane.
We wanted the patterns to be generated somewhat by the user input as well and randomly generated. The inputs for the RGB colour generated use variables that return 3D noise and positive and scaled sine values which result in an interesting texture which updates in real time and can be altered by the user as the inputs – frequency, phase, noise scale and brightness – can be adjusted by a ‘random’ function.
Along with regenerating the plane we wanted to allow the user to themselves regenerate the L-System trees. ofxRules comes with many easy to use functions which allow for the clearing and updated of the L-Systems. Therefore, when the user uses the ‘a’ key to update the parameters of the plane, the existing tree is cleared and another begins to recursively grow differently from the previous.
Our project began to deviate from our original project proposal following the feedback on the feasibility of our plan. The minimum viable product was to be a piece of computational art that augments the user’s scene which we adjusted to suit a more realistic outcome for the time we had left. In this sense we succeeded as we created a scene of AR but we failed to included any more information from the users scene other than the correct dimensions of the space, used to generate a realistic augmented scene. As a result of several issues, which we will address, our project only began to follow through on a single and outcome at a later date rather than from the beginning of the project as desired.
There have been many changes since our proposal plan but ultimately our outcome is still an exploration into AR and procedural generation. We lost time trying to get addons to work with the current version of openframeworks when we should have been working towards creating an outcome that built on work created from the first weeks and was adjusted continually. As a result our decision to later create a physical outcome, a Zine, allowed us to focus towards a more interactive and interesting application. Although we had planned to implement some of our own generative work, based around the superformula(7) we had explored for our proposal, we decided to use the ofxRules as the animation and use of natural algorithms was seamless and relatively easily implemented. We understand that our project is not as successful as it could have been judging from our proposal and how far we have deviated from this and our plan which was regrettably not as realistic as it could have been considering what we wanted to achieve and the time had to achieve it in.
Any AR project is best when most accessible which means our app would be best suited for mobile as many AR projects are. Although the immersiveness of the intended outcome has not been at the level we had intended, the overall experience of augmented media was accomplished to a degree. The original concept of showcasing augmented media, procedural generation, and user interaction is all still present within our project. We’re confident that to some degree all three of these initial concepts exist within our project, although we feel frustrated and disappointed that our initial project idea hasn’t reached the level of detail we would have like. We feel that through all the hurdles we were faced with we were still able to deliver upon the concepts we initially intended to convey.
Link to GitLab Repository
Appropriate References and Bibliography
1. ofxAruco: https://github.com/arturoc/ofxAruco
Further information on ofxAruco library: http://www.uco.es/investiga/grupos/ava/node/26
2. L-Systems Library: http://www.neilmendoza.com/ofxrules/
Oscillating plane with AR tutorial: http://www.creativeapplications.net/processing/augmented-reality-with-processing-tutorial-processing/
Shader tutorials: http://openframeworks.cc/ofBook/chapters/shaders.html
Shader and Mesh tutorials:
Making Some Noise
Initially my project hoped to define and implement a physical, meaningful link between visual and auditory processes within the computer. Specifically working from the perspective that the user would make use of visual input/manipulation to produce audio output.
Two potential avenues were considered
1. the VGA Port: potential processing of VGA (a precursor to modern graphical output)with the signal being fed back into the computer and used to implement audio processing.
2. the main graphics pipeline itself (through monitoring processes happening on the chip or following the journey of pixels being processed).
With the idea for graphical input being to make use of keyboard and mouse to control a set of base shader functions.
Ultimately, approximately half way into the project these avenues were abandoned, not through lack of potential but more through lack of technical skill on my part and the time constraints for the project. Moving away from the physical connection between processes to the functional representation of signals (as a multitude of sine waves) was the new focus. With this switch it was necessary to shift from having a display based on direct input to one which would allow the user to explore the audio landscape of photographs or pre-rendered images.
Research process and outcomes:
Initial focus was on trying to understand the processing of a digital signal to produce a visual stimulus. Working from the top down I discovered that the visual output was stored as a bitmap in the frame buffer and the bit pattern per pixel was read to a graphics display at a refresh rate. So, how did these pixel values come to be generated and could I monitor the process to produce audio?
This question led me to the Graphics Pipeline where I was confronted with images such as this:
Clearly a pixel went through numerous stages and memory locations before reaching the Frame Buffer. And importantly these locations were on the graphics card.
A Brief History of the Graphics Card:
In the past, graphics operations were performed by the Video Graphics Array (VGA), a memory controller and display generator connected to some DRAM. Over time, these controllers moved to incorporate specific graphics necessary functions such as hardware for triangle setup and rasterisation, texture mapping and shading; and they became more programmable, with programmable processors replacing fixed function logic. Now, GPUs have become vastly parallel processors containing many processor cores, typically organised into multi-threaded multiprocessors.
Looking into these, it became apparent that most companies kept the inner workings of their processors to themselves. But there was one exception. The documentation for the GPU located on the Raspberry Pi (the Broadcom VideoCore IV 3D) has been released by Broadcom, leading to another diagram.
And even another one representing a QPU(Quad Processing Unit) pipeline:
These diagrams illustrate well, the minute scale from which pixel fragments are combined to form a pixel. And after reading the relevant portions of the documentation and the experiences of a couple of people who had worked with GPGPU on the raspberry pi I accepted that this was too complicated for my current project. If I were to go back to it I would perhaps start by looking at the texture and memory lookup unit but not only is this too complicated technically, resolving these processes into a form which would allow for meaningful audio output would be challenging to say the least.
Moving from this, I looked to the VGA port of my laptop. This is a 15Pin output with pins for red, green, blue, V-sync and Hsync to describe the image. Attempting to run these through an arduino proved fruitless. With me being unable to utilise the xrandr function to force a signal through the VGA port.
It was here that I switched to looking into implementation of fft to analyse an image and use this analysis to produce sound. Initially interested in 2DFFt implementations for calculating spacial frequencies I eventually settled for 1demensional transforms in ofxMaxim after being unable to implement the specific libraries offering a 2D feature (these being ‘FFTW’ and ‘OfxFft’).
The program is split into two parts. The ‘sample’ class runs in the main thread, setting up the audio context, fft and loading the images along with selecting and manipulating the image. The ‘noisemaker’ class runs in a separate thread and deals with calculating the audio samples and organising the bins by magnitude when necessary.
The image above shows the program running. The portion of the image being sampled is the highlighted central square whilst the full image is covered by a translucent mask. The fft analysis of the central square is represented by the white peaks towards the bottom of the image.
The central square (altered by moving the mouse) contains 32768 pixels. This was chosen as the largest power of 2 that could be processed by the fft. A power of 2 was desirable as it speeds up the fourier transform considerably. One downside to having this size of sample is the time it takes for the audio to be calculated. Using the standard wave equation of A*sin(TWO_PI * f * t+phi) for 16000 bins per audio sample means that unless you keep the mouse fixed in one spot for a long time, the current image portion being sampled will only account for a fraction of the sound being played. One bonus of this is that some interesting effects can be created through movement of the mouse around the image.
Basic image resizing functions are provided by using the ‘UP’ and ‘Down’ arrow keys while pressing ‘r’ will reset the image to its original state.
There are functions to filter the bins being used for a particular image sample sound by magnitude. Pressing ‘m’ enters this mode with the ‘UP’ and ‘Down’ arrows increasing or decreasing the number of bins to include. This was initially intended to provide a way of breaking up the noise with something different and offer an audio to image mode by generating the image based on this bin selection. However, this came with unforeseen problems. Firstly, it is necessary to wait for the noisemaker to finish its current cycle, and this takes a long time, breaking any feeling of engagement or exploration for the user. Secondly, because fewer frequencies are included there is less dampening from phase-shifting of sin waves and considering the ones which are present are of the highest magnitude – there is a large increase in volume as soon as the magnitude filter begins, this volume decreases the more bins are included.
Unfortunately I was unable to resolve this issue nor provide further image or sound manipulation functionality.
Evaluation and Conclusion:
Overall this was a poor project. The final outcome is limited and it is devoid of any deep understanding of the chosen area of study. The sound/image link is not particularly successful considering the lag and it is not fun enough to justify the arbitrary audio/visual connection.
Areas for improvement practically include increased functionality. For example the ability to save samples and alter multiple images, offering a range of image filters and sample playback effects, and smoother handling of levels and memory management.
However, the real downfall of this project was the planning. It would have been far better to start from a position of understanding and build from there (perhaps where the technical research left off) rather than attempting to learn three completely new topics. Ultimately the difficulty of the initial research bogged me down and I was unable to make any practical gains or headway.
On a positive note I will be more prepared in future and this project has given me new ideas regarding the handling of signals and data and the potential for making meaningful connections between the human experience of the digital and the way computers are used to abstract and analyse this experience.
Bibliography and References:
‘Computer Organization And Design: the hardware/software interface’ by David A. Patterson and John L. Hennessy
VideoCore IV 3D documentation: https://www.broadcom.com/docs/support/videocore/VideoCoreIV-AG100-R.pdf
Walkthrough for GPGPU on Raspberry Pi: https://rpiplayground.wordpress.com/2014/05/03/hacking-the-gpu-for-fun-and-profit-pt-1/
A project making use of Pi GPU before documentation was released: https://github.com/hermanhermitage/videocoreiv
A linux graphics overview: http://moi.vonos.net/linux/graphics-stack/
A VGA project converting audio to Video: http://crackedraytube.com/pdfs/hacking_a_vga_monitor.pdf
Figure 1: https://www.broadcom.com/docs/support/videocore/VideoCoreIV-AG100-R.pdf page 13
Figure 2: https://www.broadcom.com/docs/support/videocore/VideoCoreIV-AG100-R.pdf page 17