Some examples of Ditto – Delaunay:Ditto-Delaunay is a piece of software that breaks down images into triangular segments and, from these segments, attempts to create a new image. The program works particularly well on images that contain buildings or architecture as they contain many corner points for the Harris Corner detection to pick up.
Audience, Research and Intended Outcomes
The project was intended to be an app that created works of ‘art’ based on an image that the user would input. Due to using computer vision to create these pieces of work, this would lead to unpredictable behaviour as the algorithms would have to attempt to divide the image and decide what it deems to be similar.
The audience for the piece wasn’t particularly directed at people with an interest in the field of computer vision but conversely would attempt to make an impression to those who do not understand or are aware of what computer vision has the ability to do.
After searching around we discovered a contemporary artist group under the name ‘!Mediengruppe Bitnik‘ [A1] who had several pieces of art and exhibitions. Namely, these particular pieces we were interested in were Darknet Shopper [A1-3], H3333333k [A1-1] and Download Finished [A1-2]. Darknet Shopper raised many questions as to the rights to software and to what extent software is autonomous. H3333333k and Download Finished however ventured down a different route, compared to Darknet Shopper; this being that both of these pieces explore the concept of file manipulation and how new art can manifest itself from changing and mutating the old art.Another influence I ended up finding was another piece called Pirate Cinema [A2]. This piece of work was able to sniff the torrent peer-to-peer network and display frames of what people were sharing onto a livestream. This idea of using other media to create new media was another core inspiration behind the project.
Design process (Creative)
Initially James and I created the project’s creative idea; the idea being to pass segments of an image into a database of other image segments. The research behind Bitnik’s work, including Darknet Shopper, H3333333k and Download Finished, heavily inspired the idea. With this inspiration, it brought up many questions behind the ethical legality of creating new content by manipulating the old content.
After searching for different methods, we soon found that the google images API was limiting on the images we got back as a result and so decided as to looking into algorithms that could recreate a style of architecture from different input images.
I decided to start to explore different methods to segment and compare the segments in order to see how a different approach could give different meaning to the overall concept. With a combination of Harris Corner detection and ofxDelaunay I saw potential with how this concept could reflect the prior research of H3333333k, Download Finished and Pirate Cinema by attempting to recreate an image out of itself.
Although, this implementation differs to the original singular app concept, it manages to capture the same core creative idea as was established with James.
Design process (Technical)
Comparison by shape
In order to compare these different segments, I started by looking into comparison between geometric properties as this was a relatively easy metric to compare against. To get these metrics, all I required from the segmentation algorithm was a vector of vertex position in either a consistent clockwise or anti-clockwise order. From these vertex positions, other metrics can be calculated; shape angles using dot product, edge lengths and shape rotations between each segment.
See initial post on comparison by shape:
See post on how to get angles from a segment:
However, issues arose when looking at some of the segments from the segmentation algorithm as there were too many different shaped segments (e.g. ranging from 3 sided to 9 sided irregular shapes). This made it pretty difficult to compare using these metrics.
ofxDelaunay [A3] is a library that allows us to use Delaunay triangulation from a set of points. From a set of points, the triangulation would create triangle segments which can be extracted.
This library was a solution that would reduce the segments that were unreproducible from the hough lines method. It would allow us to cut down the number of segments in total but, at the same time, allow us to have more comparable segments. This solution also allowed the use of comparison by geometric properties as all the segments were triangles.
Some issues with this library are quite apparent; as more points are added, the ITRIANGLES, from the object, do not seem to match the representation from what is given in the mesh drawing. Instead of being triangles containing uniques parts of the image, it shows that each triangle can overlap with another triangle, as shown here:
Repo and Executables
http://gitlab.doc.gold.ac.uk/joldf001/creative-project (James’`command-line` branch)
In order to get the triangular segments, I first used Harris corner point detection [A4] in order to have some points for the ofxDelaunay triangulation object. These points were then added to the ofxDelaunay object and the object could then triangulated these points to create triangles. The program then creates a ’Tri_Segment’ object from each ITRIANGLE the triangulation object had found.
A particular issue was trying to get the part of the image that the triangle had enclosed separate so it could be rotated, scaled and translated later if the comparison matched. In order to accomplish this, each segment contains a separate ofxDelaunay object with its vertex positions added into the object and this ofxDelaunay object calls the triangulation function.
As mentioned prior, the ofxDelaunay object creates triangles that overlap each other. This means that for any x and y coordinate, any number of triangles can contain that coordinate. There is also another slight issue with the library; the “getTriangleForPos” function only returns one ITRIANGLE which causes a slight issue. With the current solution it is solved because each segment contains it’s own ofxDelaunay object.
If it wasn’t for the ofxDelaunay function “getTriangleForPos”, it would have been a lot more difficult to find whether an x and y coordinate of the main image was in the triangle. Fortunately, all that was needed to do was to loop through the x and y coordinates of the main image and pass them through the “getTriangleForPos” function. This would then return the ITRIANGLE if the function returned true; if this happened then the colour pixel at the current x and y coordinate would be copied into the segments ofImage.
Below is a video demonstrating how the triangle segments are captured and the background is removed [B4]:
Comparison and Replacement
As comparison was simply based on shape geometry alone, all that was needed to be checked was if the angles of the shapes were the same. This means that the program loops through each segment and compares their angle array against all other segments. If the segment angles match then the segments take a reference of each other to be used for replacement.
Below demonstrates how segments are matched [B4]:
Due to the nature of shape angle comparisons, rotation and scaling are not a factor meaning that these are needed to be calculated. For rotation of the segment, an angle is calculated from a vector that is 300 pixels above the segment’s midpoint to the same vertex. The angles are then compared and an appropriate number of degrees are added to the one of the segments to ensure the rotation is the same. For scaling, the ratio is calculated by taking the length of one of the matched segment’s edges and dividing it by the same edge but from the transformed segment. The transformed segment is then translated to the midpoint to the matched segment.Below demonstrates how matched segments are rotated and scaled to the size of the other segment [B5]:
This project was challenging and we both understood what was needed to be done in order to accomplish it. I believe that the project, overall, was largely a success. By looking at the final pieces that were created, I am reminded that recreating images, whilst easy for a human to perceive patterns and match segments, is vastly difficult for a computer to achieve such a high level of accuracy.
I feel that the end results reflect the original intensions that we had strived for when starting this project. They take many aspects from the pieces that we had looked into; in particular, relating to Ditto – Delaunay, the pieces like Download Finished and Pirate Cinema show an idea of being able to create new media from already existing media. Elements of H333333k are also present within this piece by causing the original image to appear partially glitched demonstrating the slight randomness of the segments and the chaos it can show.
I feel that both projects compliment each other well, by exploring the shared core idea but using different methods to achieve this. After looking into different concepts, it became apparent that it would be an unnecessary idea to combine the two together as they use different technical skills.
References and Footnotes
Please refer to these links for more information on the project:
- Ditto – CV (James’ Side) [http://igor.gold.ac.uk/~skata001/creativeprojectspace/2016/04/20/ditto-cv/]
- Creative Research [http://igor.gold.ac.uk/~skata001/hiveMind/2015/10/28/bitnik/]
- Technical Research [http://gitlab.doc.gold.ac.uk/joldf001/technicalResearch]
- Blog James and I worked on [http://james-oldfield.github.io/creative-projects-blog/]
- Project Proposal [http://doc.gold.ac.uk/~mclar053/cp2-proposal-mclar-joldf.pdf]
- Project Brief [https://learn.gold.ac.uk/mod/resource/view.php?id=297382]
- Delaunay branch [http://gitlab.doc.gold.ac.uk/joldf001/creative-project/tree/delaunay-segs]
- James’`command-line` branch [http://gitlab.doc.gold.ac.uk/joldf001/creative-project]
References for materials used:
- Bitnik [https://wwwwwwwwwwwwwwwwwwwwww.bitnik.org/]
- H3333333k [https://wwwwwwwwwwwwwwwwwwwwww.bitnik.org/h3333333k/]
- Download Finished [https://wwwwwwwwwwwwwwwwwwwwww.bitnik.org/d/]
- Darknet Shopper [https://wwwwwwwwwwwwwwwwwwwwww.bitnik.org/r/]
- Pirate Cinema [http://thepiratecinema.com/]
- ofxDelaunay [https://github.com/obviousjim/ofxDelaunay]
- Harris Corner Point Detection (Open CV) [http://docs.opencv.org/2.4/doc/tutorials/features2d/trackingmotion/harris_detector/harris_detector.html]
I do not own rights to any images of the architecture used.
Here are links to images of architecture that has been used: