Throughout our project we researched into Augmented Reality (AR) and our resulting final product was an A5 zine which uses a marker to augment the page through the generation of objects. A ‘natural’ scene is generated covering the marker consisting of an oscillating plane and procedurally generated ‘trees’. Through using AR and L-System generation addons for openframeworks, ofxAruco(1) and ofxRules(2) which have dependencies on ofxOpenCv,ofxCv and ofxXmlSettings and facilitate the recognition of the marker and thus place the generated object within the right dimensions of the users’ scene, we were able to realise an individual and changing AR scene for each marker. We intended for the project to consist of procedural themed scenes that would vary for each page in order to show the book pages coming to life and consisting of different scenes. The project wanted to include interactions for each scene by bringing in outside influence from the user, but we instead decided on a ‘regeneration’ button which updates the plane with randomly generated values and begins the steps of the L-System growth again. The project seeks to have users experience a procedurally generated scene through the pages of a booklet, by giving different augmented experiences.
Film documenting user interaction with the final Zine produced
Intended Audience/ Background Research
The projects intended audience wasn’t targeted towards any particular group but rather to convey an experience into a new form of media. The intended audience is for anyone wanting to experience a type of media that is not constrained to the pages of a book which led to some initial research into markerless AR from Disney Research(3) as this would have allowed direct physical interaction and potentially creation of the marker itself. The audience isn’t targeted based on the environments presented, but rather the concept of transforming two-dimensional paper media into three-dimensional environments. This was influenced by the work of painter Austin Lee who brought his prints to life through the use of AR(4). The intended audience for our project will vary from users of all ages, young children can experience the project as an interactive book to enjoy or for adults seeking to understand and experience augmented media. We hope that any age a project of such concept can seek to be informative but also enjoyable to use, we believe that any user can easily understand and use the project despite having any knowledge of the technologies being used, therefore allowing them to experience a whole new form of media with almost no knowledge or input needed.
Initially the project was intended to augment and procedurally generate objects within the user’s intended scene by using a camera to change their current location into a procedurally generated one. The project initially was to invoke an experience of augmenting a user’s surrounding, thereby allowing the user to fully relate to the scene being presented. As the intended design changed to exist upon a page of book, the original invoking experience stayed true but rather to a lesser level. Originally the concept was to use markerless AR in order to camouflage the actual markers that initiate the scenes, due to limitations of the addons we used, we had to change the design to include more visible markers.
Example of AR prototype and markers provided by ofxAruco
As the design process moved forward we initially intended to include multiple markers within each scene to invoke a more immersive experience, but after experimenting with multiple markers we felt that having one distinctive marker allowed us to focus upon one themed object within each scene. ofxRules comes with inbuilt xml files which step through the generation of L-Systems and as such we were able to easily include dynamic and animated L-Systems. We intended to use physics and created programs to do so but as we began to bring in the L-Systems we realized that the physics did not easily comply with the ofxRules addon. Therefore we showcased the L-systems as living by allowing them to recursively generate within each scene, and accomplished a similar ‘natural’ state to the physics implemented but through a change in the design concept.
Click to play gif of L-System growing
Originally, the concept was to allow lines that represent the two-dimensional version of the scene upon the paper media to interact with the augmented version. This would allow the use of textured shaders differently within in each augmented scene allowing for multiple shaders to be present upon each page. Instead we found that to create an oscillating plane that coordinated with the generated texture colours the shader could be used to alternate the y values across the plane and the input of the colours could be decided through noise and sinusoidal functions as well as being influenced by the marker recognised.
Click to play gif of oscillating plane created through textures and shaders
We decided to create a Zine to allow us to have more of a physical and single viable product to work towards. The three markers, and designs featured, produce a surreal digitally rendered scene that existed alongside the printed version of itself. Mockups of the zine were created which included several markers and different designs however, we decided that a single printed marker along with the texture coupled with the AR worked best.
Following the unsuccessful efforts of implementing ofxARToolKitPlus (another AR addon for openframeworks), we tried the application using older verisons of openframeworks but this led to other complications and thus we decided to use ofxAruco. ofxAruco is easily implemented and automatically calibrates the webcam information and configures the marker information using the provided ‘intrinsics.int’ and ‘boardConfiguration.yml’ files provided. From this point the use of the AR and implementing objects within the scene was accomplished relatively easily with slight adjustments to the scale of the objects created.
We decided to augment the pages of the book with digitally rendered scenery which took influences from nature both in its production, L-Systems use recursive algorithms found in nature, and in its aesthetic. The xml files provided by ofxRules allowed for the recursive drawing of trees to be easily implemented and so we wanted to create a ‘land’ for them to sit on that was had a similarly ‘natural’ animation. The shader examples provided by openframeworks include an example of displacing a plane primitive using the shader(5) but use a grayscale image as the texture. We adjusted this example to work with colour textures by using the green channel, which translates to the y position of the vertex, to update the y position of the plane creating a smooth, undulating surface that moves physically relates to the darkness of the colour created. The oscillating surface is created through generated 3D noise across the pixels of the textures which translates to the 3D noise of the plane.
We wanted the patterns to be generated somewhat by the user input as well and randomly generated. The inputs for the RGB colour generated use variables that return 3D noise and positive and scaled sine values which result in an interesting texture which updates in real time and can be altered by the user as the inputs – frequency, phase, noise scale and brightness – can be adjusted by a ‘random’ function.
Along with regenerating the plane we wanted to allow the user to themselves regenerate the L-System trees. ofxRules comes with many easy to use functions which allow for the clearing and updated of the L-Systems. Therefore, when the user uses the ‘a’ key to update the parameters of the plane, the existing tree is cleared and another begins to recursively grow differently from the previous.
Our project began to deviate from our original project proposal following the feedback on the feasibility of our plan. The minimum viable product was to be a piece of computational art that augments the user’s scene which we adjusted to suit a more realistic outcome for the time we had left. In this sense we succeeded as we created a scene of AR but we failed to included any more information from the users scene other than the correct dimensions of the space, used to generate a realistic augmented scene. As a result of several issues, which we will address, our project only began to follow through on a single and outcome at a later date rather than from the beginning of the project as desired.
There have been many changes since our proposal plan but ultimately our outcome is still an exploration into AR and procedural generation. We lost time trying to get addons to work with the current version of openframeworks when we should have been working towards creating an outcome that built on work created from the first weeks and was adjusted continually. As a result our decision to later create a physical outcome, a Zine, allowed us to focus towards a more interactive and interesting application. Although we had planned to implement some of our own generative work, based around the superformula(7) we had explored for our proposal, we decided to use the ofxRules as the animation and use of natural algorithms was seamless and relatively easily implemented. We understand that our project is not as successful as it could have been judging from our proposal and how far we have deviated from this and our plan which was regrettably not as realistic as it could have been considering what we wanted to achieve and the time had to achieve it in.
Any AR project is best when most accessible which means our app would be best suited for mobile as many AR projects are. Although the immersiveness of the intended outcome has not been at the level we had intended, the overall experience of augmented media was accomplished to a degree. The original concept of showcasing augmented media, procedural generation, and user interaction is all still present within our project. We’re confident that to some degree all three of these initial concepts exist within our project, although we feel frustrated and disappointed that our initial project idea hasn’t reached the level of detail we would have like. We feel that through all the hurdles we were faced with we were still able to deliver upon the concepts we initially intended to convey.
Link to GitLab Repository
Appropriate References and Bibliography
1. ofxAruco: https://github.com/arturoc/ofxAruco
Further information on ofxAruco library: http://www.uco.es/investiga/grupos/ava/node/26
2. L-Systems Library: http://www.neilmendoza.com/ofxrules/
Oscillating plane with AR tutorial: http://www.creativeapplications.net/processing/augmented-reality-with-processing-tutorial-processing/
Shader tutorials: http://openframeworks.cc/ofBook/chapters/shaders.html
Shader and Mesh tutorials: