By Mateusz Janusz and Becky Johnson
Fractals and recursion are one of the most exciting and fascinating part of mathematics and computer science. The fact that even those who find no pleasure whatsoever in mathematics can still find interest in their visuality and complexity shows that. Whether it is the naturally emerging fractals that occur in everyday life or the computer generated fractals such as the Mandelbrot Set or the Menger Sponge, fractals and recursion is the centre of focus of our project.
Our project is an art tool that takes the voice of the user and turns it into a piece of artwork using fractals and recursion. Using the analysis package of the Minim library, sound analysis is conducted on the audio input to alter the generated fractals. There is a choice of 5 fractal designs to be selected at any point of running the program, where the frequency and amplitude of the user’s voice alters the position, size and colour of the designs.
DOWNLOAD LINK FOR PROJECT
DOCUMENTATION ON INTENDED AUDIENCE
Our idea caters to an audience who do not necessarily have an artistic ability on canvas or through digital technology. This lack of artistic ability on canvas may narrow down to those of a younger age, and the lack of digital artistic ability may focus on a senior generation who may not have a knowledge in digital mediums to create artwork.
DOCUMENTATION ON INTENDED OUTCOME
The outcome we wanted to achieve was for our user to be able to create a unique piece of artwork featuring fractals and recursion by the use of their voice. By being able to control the volume and pitch of their own voice, the user can have an element of decision over the development of the artwork. We wanted the user with this control to be able to see the effect their voice was having on the design by the change in colour, size and detail of the fractal, and the position of the design on the sketch based on these components of their voice.
The enthusiasm to achieve this outcome was fuelled by both of us not having an artistic ability on canvas, and the opportunity to create something that looked intricate and personally unique just from the use of a voice was an exciting outcome to try to achieve.
Firstly we researched into sound, as we felt like it was important when starting the project to make sure that we understood the fundamentals of this.
In the beginning, we were advised to use the Minim library as it would be able to analyse the components of the audio input we wanted to be able. The main feature of our program involved taking the audio input from the mic of a laptop and analysing the amplitude and frequency to create a useful value, so checking that Minim would do what we needed was extremely important to us at this stage. As well as Minim, we also looked at Processing’s Sound library just in case we felt like this would cater more to what we wanted. Overall, we felt that Minim seemed to be more advanced from the documentation, and decided to take the advice we were given at the start.
The main visual aspect of our project involved fractals and recursion so we extensively researched into these two topics. Firstly looking at the aspects of recursion and fractals in computer science and mathematics, and then looking at famous fractals to try and understand their algorithms to better our knowledge of recursion.
The famous fractals that we researched included the Barnsley Fern, the Mandelbrot set, the Menger Sponge, and the Sierpinski Triangle. This research into the algorithms was so that later on, once we were more comfortable using FFT and recursion, we could try to implement these fractals and manipulate them with the values obtained from analysing the audio input with FFT.
Name: Ken Headford
Ken is now a retired senior citizen and has taken up various hobbies in his retirement, such as oil painting, golfing, bird watching, and walking with a local group.
Ken has not got much experience with digital technology. He and his wife own a shared desktop computer, and he uses it to check his emails, and to research interests and hobbies of his. He tries to avoid using the computer as much as possible, as he finds that he does not understand various components of searching online, how the computer works, or how to fix the Internet connection when it has disconnected.
Ken enjoys artwork, but has never branched into the digital side of creating art. He loves oil painting, however he finds it frustrating for the amount of time it takes to paint certain pieces.
Name: Emily Rowe
Occupation: Primary School Student
Emily is a primary school student who enjoys going to the park with her friends and family, playing video games with her siblings, and her favourite subjects in school are Art and Maths.
Emily has showed an early interest in technology, from playing video games with her siblings on consoles such as a Game Cube, Playstation 3, and on the computer. Emily also has shown an interest in Art, but does not think her drawings are very good and is impatient because she wants to make good pieces of artwork.
She dislikes using pencils and pens, as she and her siblings are either very forgetful or messy and they lose or break all of them. Their Mother does not buy them paint anymore because it is expensive and from past experiences furniture has been ruined because of it.
After becoming enthusiastic about the idea for our project and researching the topics associated with it, we started to focus on the user centred design and functionality of the project.
We brainstormed how to change and alter between drawing different fractals. A couple of ideas that we had initially was creating a counter that would change the current fractal mode depending on the time passed, or so that the values of the amplitude and frequency would change the current fractal mode. However we felt that both options didn’t give the user the freedom to control the visuality of the sketch as much as our intended outcome was.
We also thought about the personas that we had created, and felt that the most simplistic user interface would be the most desired for them due to age or lack of technical knowledge. We felt that the focus for the user should be on what he or she would create, not on the appearance of the project’s user interface.
The first part of our build was getting the audio input from the speaker of a laptop and to be able to work with the values it produced. One of our problems was that the documentation for the Minim library did not seem very simple, and the examples for how to do this were very complex and did not conclude exactly what we needed. From this, we decided to look else where to research how to do this.
We found a video https://vimeo.com/7586074 that described exactly what we needed to do in order to take the input from a mic and perform FFT on the input. We transcribed this video and this served as an extremely simple template for which we could test design ideas using FFT to influence the designs.
The template worked by creating and initialising the objects we needed for analysis, a Minim, AudioInput, and FFT object. The AudioInput object uses the “getLineIn method of the Minim object to get an AudioInput where the audio buffer is 256 samples long. Later on in the build of the project, we increased the sample size to 1024. This size needs to be a power of two because the FFT object needs a buffer size of this size.
After calling the forward function on the FFT object to mix the left and right channels of the Audio Input, a for loop is created to loop through the number of bands in the FFT object. The “getBand(i)” function returns the value of the current frequency bands amplitude. With this, we could start to use the values of this input in coercion with fractal images.
In creating the project, a lot of issues we had stemmed from trying to use recursion as it was very easy to create a base case that did not escape infinite recursion and would therefore crash Processing. Once we had grasped the logic of creating and manipulating a variable to definitely change to or past a certain value, this limited the amount of crashes we had.
At first, it was very difficult to incorporate recursion using the values from FFT. Our very first sketch successfully using the values of the audio input with FFT created recursive ellipses that rotated based on the value of the frequency. When trying to use the values of our FFT as our exit conditions in recursion, this became sufficiently more difficult. Our FFT values that were returned ranged tremendously, especially when multiplying the values by a large coefficient to be able to use the values in the fractal designs.
This was solved by trial and error of measuring each coefficient as a multiplier against these values. We tested our voices by ranging the amplitude and frequency and recorded the results to investigate the range of values in order to plot our exit conditions.
Creating this simple design was the starting point however to be able to create more complex fractal designs with a knowledge on creating exit conditions using the FFT for the recursion.
Our next design was adapted from a recursive tree example by Daniel Shiffman , where the coordinates of the mouse alter the number of branches as well as their length. Instead we wanted the amplitude and frequency of the audio input to alter the size, length and number of branches. So instead, we created a condition for the branches to “start drawing” if the value of the amplitude of the left channel of the stereo input was over a certain amount, and if so, rotate these branches by the amplitude of the current frequency band. Both of these values were multiplied by larger coefficients to transform the values to a more useful value.
At this point, we had a conflict with the current design we had and the initial outcome we had wanted for the project. What we had wanted was for the values of the audio input to create a growing, slowly emerging fractal image. Instead, the fractal appeared with a “stamping” effect on the sketch, where it appeared as a singular fractal shape and differed by changing the length of branches and colour from the FFT values. We tried to fix this problem by using an example of a generative design that imitated the development that we wanted.
After trying to fix this problem for a long time and because of the time constraints we had on the project, we decided to try for a simpler technique of making the fractal stamp in different positions in the sketch instead of starting in the same place each time.
A solution we found to this was by translating the entire design by the FFT values, it’s x coordinate by the sine of the current amplitude of the current frequency band, and it’s y coordinate by the cosine of the current FFT frequency band. This technique we implemented for our other fractal designs using varying coefficients. We decided at this point that although it was not the outcome we had wanted for the project, it was best to move on and to try create more of these recursive fractal sketches due to the time constraints we had.
Once we had created this branching fractal design with the movement we were happy with, we decided to make a prototype of the sketch at this point with the user design we had designed based upon our wireframes.
Once happy with this prototype, we decided it was best to continue working on our own fractal designs and implementations of famous fractals that we had created in our background research. Once we were happy with each, we would continue to add the designs to the prototype with the same functionality as the first.
Some of those which we had researched in the beginning were too difficult for us to incorporate using FFT analysis with. For example, we tried to use the values of the amplitude and frequency to alter the appearance of the Mandelbrot set. However, for us it was too difficult to manipulate the formula for plotting which numbers diverged to infinity or bounded, and we encountered a lot of problems with infinite recursion as a result. The values in the Mandelbrot set coupled with the values from the FFT resulted in being far too complicated for us to create something mildly visually satisfying so we moved onto other fractals that we had researched.
These included the Sierpinski Triangle and the Barnsley Fern. Through our research, the algorithms for these two fractals had been the easiest for us to understand. The Sierpinski Triangle is an example of a finite subdivision rule, where the shape is broken down into smaller pieces of itself. The Barnsley Fern is an example of an iterated function system, meaning it is composed of several contractive copies of itself that draws the points of itself closer and closer together for however many iterations.
One problem that we faced at this point when trying to implement all of our designs together was that our “button menu” that we had designed was being drawn over by the fractals. At this point, the code for the project was incredible messy, as we underestimated how difficult it would be to implement all of these drawing modes together instantly.
It was an incredibly tedious problem where simple things such as different colour modes being active, such as using RGB for the button menu and using HSB for colouring the fractals. This as well as different stroke, fill, pushMatrix, and popMatrix function calls, interfered with each other. This made the solution a case of reading carefully through the project and neatening up the code until there was no more interference between functions.
After adding all of our drawing modes together and completing the functionality of the user centred design, it was a case of trial and error to initiate the drawing to start above a specific frequency value for each fractal. This did not take a long time in the build, and was the finishing touches to our project.
EVALUATION OF PROJECT
In terms of what our intended outcome was for this project, there are a few things that we think were successful and a few that did match up to our original hopes.
For instance, we had wanted our fractals to “grow” so that the users could see exactly how their voice was altering the position and movement of the design. This issue we had had in the beginning of the build of our project, and due to time constraints and after many days of trying to achieve this effect, we realised it was not practical to continue trying in this instance. Instead, we tried to make up for this problem by keeping this “stamping” effect, but using the values of the FFT on the user’s voice to change the position of where the fractal stamped. We felt that although this was not want we had initially wanted, it still gave the user freedom to partially control the design and that was the main desired outcome of our project.
We felt that we had successfully achieved correlating the values of the amplitude and frequency to noticeable colour changes that the user would be see and control the colour from this. For example, the higher pitch or louder the voice, the stronger the hue and saturation of the colours, resulting in the reds, pinks, and bright blues.
We feel that the program does not give the user the extreme control that we had hoped for though. For example, the user does not know exactly what his or her voice will create in each drawing mode, and in the beginning that was one of the aspects that we were really enthusiastic about. We understand that this lack of control stems from our previous problem of not being able to design the fractals to grow with the person’s voice, and that this desired outcome outweighed our knowledge in programming so it was unrealistic for us to expect to be able to create it so easily. However, we are happy with the results that can be obtained from using the program because we feel that it can create unique and complex pieces of artwork, and that was one of our desired outcomes.
For both of us, our knowledge of sound analysis and Minim started as null, and after completing this project we feel our comfortability in this area has increased immensely. Although it is not perfect and in-sync with our original desired outcomes, we feel we have achieved a large percentage of the overall outcome we wanted.
CONVARTSATION ART GALLERY
 Documentation for Minim library – http://code.compartmental.net/minim/
 Processing’s Sound library – https://processing.org/reference/libraries/sound/
 How To Use FFT and Minim Tutorial Video – https://vimeo.com/7586074
 Daniel Shiffman, Recursive Tree – https://processing.org/examples/tree.html
 Generative Design – https://github.com/generative-design/Code-Package-Processing-3.x/blob/master/01_P/P_2_2_4_01/P_2_2_4_01.pde
Fractals: Nature Of Code, fractal and recursion research -http://natureofcode.com/book/chapter-8-fractals/  FractalFoundation – http://fractalfoundation.org/resources/what-are-fractals/
Recursion: Introduction to Programming in Java – http://introcs.cs.princeton.edu/java/23recursion/  Khan Academy, Recursive Algorithms – https://www.khanacademy.org/computing/computer-science/algorithms/recursive-algorithms/a/recursion  Wikapedia, Recursion – https://en.wikipedia.org/wiki/Recursion_(computer_science)
Minim: Audio Visual Programming -http://sweb.cityu.edu.hk/sm1204/2012A/page20/index.html  Minim documentation – http://code.compartmental.net/minim/javadoc/ddf/minim/analysis/FFT.html  Processing Tutorials by Daniel Shiffman – https://www.youtube.com/user/shiffman  Generative Drawing – http://www.generative-gestaltung.de/P_2_2_4_01