We usually consider art to be synonymous with creativity, originality, and abstraction— not precision. We see art as unpredictable. Nevertheless, art can be technical and even scientific. While we may never be able to systematically explain the processes behind abstract works, we can now understand and predict where brush will meet paper when it comes to artists’ renderings of objects: “algorithmic art” is no longer an oxymoron.

Research by Professor Adam Finkelstein of the Computer Science Department has found that algorithms are indeed capable of modeling where artists will draw lines, specifically those making up a two-dimensional representation of a three-dimensional figure. “Mathematicians had wondered [about this] before,” says Professor Finkelstein, but the way to accurately study this prediction was not always clear. In order to find the answer, his team developed a careful protocol, asking artists of varying levels to make drawings and then tracings of 3D shapes to see how those compared to the computer graphics (CG) renderings and with other artists’ drawings. Comparisons were made by the proximity of pixels and lines in human drawings to pixels and lines in CG renderings. In drawings done by hand, they found that “across all prompts, approximately 75% of human drawing pixels were within 1 mm of a drawn pixel in all other drawings for that prompt.”

It is “good that people are consistent,” says Professor Finkelstein, and it “means that there is hope that you can develop an algorithmic model for how people make drawings.” Yet, despite the possibilities implied by the ability to develop these models, Professor Finkelstein is certainly not looking to replace human artists with computer applications. In fact, much of his research is devoted to making the transition to digital media better for traditional artists.

The goal is to capture the best aspects of working in natural media and to combine them with the benefits of digital painting systems, rather than to exclude one space from the other. Professor Finkelstein believes that the tactile, vibrant quality of traditional media, including feeling “the connection of brush to paper,” is “part of the artist’s engagement with medium.” He suggests that this is one reason “why artists will continue to work in that traditional medium.” Still, he points out, digital media has its own advantages, such as the undo key, an elusive option when one is working with real paints. Professor Finkelstein notes that the fluidity of cut and paste has really benefited comic book artists, for example, because they can now extrapolate, alter, and replace aspects of their artistic creations efficiently. Other advantages include the level of detail and precision that some systems offer artists.

When an artist makes a stroke, RealBrush searches its library for the particular style selected by the artist and finds strokes with curves that match the shape of the stroke that’s been drawn. Then, the program synthesizes the stored strokes into a new stroke that matches the artist’s line.

When an artist makes a stroke, RealBrush searches its library for the particular style selected by the artist and finds strokes with curves that match the shape of the stroke that’s been drawn. Then, the program synthesizes the stored strokes into a new stroke that matches the artist’s line.

Nonetheless, despite the advancements, most tools, such as Photoshop, while applicable for certain artistic forms, fall short when it comes to digital painting. Indeed, making “plausible strokes” and stroke interactions (blending, layering, etc.) is somewhat limited with CG: “All of these systems suffer from a common limitation… they can only support a small range of deformations of example strokes before the altered appearance becomes unrealistic.” There are many challenges facing computer graphics—for instance, one would have to simulate the interaction of light at the molecular level with photons in order to perfectly represent a figure from every viewpoint and with every lighting effect—yet strides have been made in terms of creating the same effect as natural media using CG.

These strides include RealBrush, a project on which Professor Finkelstien has worked in conjunction with graduate student Jingwan Lu and researchers from Google Inc. and Adobe Systems. RealBrush, a new digital painting system that interacts with natural media, does not “restrict an artist with preprogrammed media.” Rather it allows artists to capture (with a photograph uploaded to the computer) the texture, for example, of a squeeze of toothpaste and use it in their digital creations. This system uses a factored algorithm, breaking the task of “data acquisition and search” into four separate “bases” – shape, smear, smudge, and composite – which thus decreases the amount of memory required of the data-driven system. It also better aligns the “trial and error” decision-making process of the artist with the exemplification of real media in CG and improves the realistic appearance of texture and stroke interaction on the screen. According to the paper, specifically “for common natural media (oil and watercolor), synthesized individual strokes are indistinguishable from real examples to casual viewers… [while] synthesized smears and smudges often look plausible, but are not indistinguishable from real examples.”

RealBrush’s library contains styles each artist can choose from when creating designs.

RealBrush’s library contains styles each artist can choose from when creating designs.

The unique foundation for RealBrush is a library that stores style. The authors state that “[w]ithin RealBrush, the user can select this library and make new strokes that are synthesized using the library to create novel marks in the same style.” These tools, which capture, register, record, and recapitulate parts of works from artists in their original media, are meant to “enhance what [professional] artists can do” as well as make digital stylized content creation more accessible for amateur artists. Professor Finkelstein mentions that developing more interactive video editing tools, which—in their current versions—are often cumbersome and “unusable by non-professionals,” could be a further step in extending access to art for these so-called non-artists. But as Professor Finkelstein points out: “Who are artists?” Are we not all part of the “spectrum of artists” in some way?

The focus is not just to connect the computer to the artist, but on “connecting inside the computer to the outside” world. In other words, we are “designing in the computer” and “realizing” our creations physically. Perhaps nothing emphasizes this point more than the result of a previous project of Professor Finkelstein’s: a physical, 3D dragon model carefully made so that perception of it at each angle matches that of the computer rendering originally used to create it. In a way, this dragon is as much a piece of art as one of science, a literal combination of the fantastical imagination within the constraints of reality. Upon seeing such work, it is no surprise to know that the possibilities of the real world are expanding through the collaboration of painters and computer scientists, of art and algorithms.

About The Author

Isabelle is a Projects Chair with Princeton Women in Computer Science and interned last summer in the CS department. She is passionate about technology, creative writing, and education.