Digital Crayon Wall

This project was created in collaboration with Emanuel Klein. It uses a Kinect and a projector to allow the user to “color” on the walls with large plastic crayons. The software for this project uses the Kinect depth and pixel data to track the crayons and a paint roller which erases the pixels.

The idea for this interaction came from memories of coloring on the walls as a child and getting into trouble. We began with the assumption that there is something inherently satisfying about drawing on the walls.  Paper offers a small and confined space but when you can draw on the walls your environment becomes your coloring book.  

Process

We wanted to give children an opportunity to draw on the walls without being scolded and hoped that adults would experience a childish playfulness and enjoy the interaction as well. This assumption was easy to test since the interaction was easy to fake. We built a simple drawing program that we projected on the wall and had users "draw" on the walls while  we followed their drawing path with our mouse from a laptop behind them. This worked surprisingly well and most users didn't notice that we were faking the interaction. 

Conclusion:  The interaction is somewhat addictive. Most users continued to draw as we asked them questions after they were asked to stop. Part of the excitement seemed to come from the novelty of drawing with plastic and not understanding how it worked, but many adults also had similar memories of drawing on the walls and expressed that  they enjoyed the interaction itself. 

Assumption #2: Increasing the size of the crayon will add to the playfulness of the interaction. 

In an attempt to make adults feel like children we tried using large crayons so that the user would feel smaller in comparison.  We found oversize plastic crayons that were meant to be piggybanks and began testing our assumption with them.  We had users draw with a standard pen, A PVC pipe with a foam tip and these large plastic crayons. 

Conclusion:  The overwhelming consensus was that the plastic crayons were more fun to draw with although they had some sensory issues. They were difficult to grip and they made a sound on the wall similar to nails on a chalkboard. They also felt too light to most users.

Solution:  We covered the center of the crayon with paper to improve the grip and covered the tip with felt. We also filled the crayon with insulation material to give it more weight. 

Erasing

Our initial assumptions did not take clearing the area into account. While user testing we hit the space bar to clear the drawings so that users can continue to draw. This created an abrupt end to the interaction that felt very digital. We could have added a delete option for users in the form of a physical button or an area on the wall that they would touch to clear the canvas but we wanted the experience to be more organic. 

Solution: We added a paint roller to the interaction. The paint roller would erase the drawings so that the installation can be set up and left alone for users to play with, adding to the drawings of other users or clearing specific areas to draw on. It also allowed for multiple users to engage with the interaction simultaneously without deleting the work of other users.  

Graphics

In order to create an interaction that feels as real as possible, it was important to us that the lines drawn look like crayon lines. We experimented with a few different options.

• The first was taking a circular jpg image of a crayon texture and having it repeat following the mouse. This looks repetitive and unrealistic.

• Our second option was to create a texture layer beneath a solid black layer that would be erased by the crayons. Beneath the texture layer there would be a solid color determined by the color of the crayon. While this worked it only allowed for the use of one color at a time since a new crayon would change the color of the entire base layer.

• The third option was to draw multiple lines with different angles at the location of the crayon creating a star shape that would repeat itself as the user draws. These shapes create a solid center but the edges are jagged and random much like a real crayon. This was both the simplest and most realistic looking option. 

Program

While designing the interaction we were simultaneously working on building the installation. We explored two different options. The first was using force sensors on the tips of the crayons to select the colors and an existing touch wall technology such as Ubi or Touchless Touch to draw on the walls. The second was using a Kinect for color and position tracking.

 Understanding that the crayons would most likely take a lot of abuse from the users made us reluctant to add hardware to the crayons themselves but I worked on the hardware option while Emmanuel worked on the software option in Processing  until we were sure that the software route would work. 

Our setup used a projector and a Kinect mounted on the ceiling facing the wall. The area viewed by the Kinect would be the drawing area. When the crayon passed a threshold close to the wall it would trigger a touch event and the crayon would begin to draw. The color would be determined with HSB values by the Kinect as well. 

Crayon Tracking  

The first step was separating out the different colors so that we could track different colors. Each pixel is analyzed based on Hue, Saturation, and Brightness to determine if it is a color that looks like a crayon. To cut back on noise an averaging method was used. We took out the yellow crayon early on since the reflections of other colors on the crayon itself made it impossible to determine that the crayon was yellow. This was less of a problem with darker colors.

Once the colors have been separated out into different layers the screen is scanned for the closest point to the wall for each x-value. All the points that fall within some threshold are stored inside a tip object. This stores the basic tip information which is based on an averaging of these points.

One problem that arises with this method is that the tip when it is against the wall is lost. This is due to the angle of the Kinect relative to the wall and crayon and due to the noise of the sensor itself. The program uses a threshold for distance to check if a point seems closer then the calibration point. This made it difficult to track when a crayon is in contact with the wall. Essentially the signal for contact (distance between the wall and the tip) is dampened as the tip approaches the wall. This forced the threshold for what is considered to be contact higher and the results less consistent.

To make the tip tracking more accurate, the pixels near the tip are tested for hue, saturation, and brightness to check if the tip is closer than what could be sense with the depth information. This allows for a much more accurate testing particularly in the ability of the program to detect proximity to the wall.

Users

Our winter show offered us a steady stream of users of different ages for two days. The show is documented in the video above. While adults did enjoy the interaction, children enjoyed it so much that parents often left their kids to play with the installation while they went to see the rest of the show. We were relieved that there was no hardware in the crayons because many kids enjoyed hitting other kids with the crayons. The eraser proved to be particularly satisfying. Users invented games amongst themselves and some users only wanted to erase other people's drawings. 

Conclusions

This installation was definitely more successful amongst children. While some adults were able to play, most had a more analytical approach to the installation.  They wanted to know how it worked, looked for buttons, and often hovered the crayon near the wall instead of making contact testing the proximity sensing. Children had an easier time throwing themselves into the fun often had to be torn away from the wall.