The Design Process

We followed an iterative design process, moving from understanding the problem space to testing and refining our prototype with users. The diagram below provides a high-level overview of our approach.

Overview of our iterative design process for the Digital Color Book project.

User Research

Our process began with a comprehensive literature review focused on color learning and embodied interaction for young children. Key findings highlighted that active learning and the direct, immediate experiencing of objects, people, ideas, and events are crucial for cognitive restructuring. Children explore the world through direct manipulation of physical objects and enjoy showing and sharing what they know. Since the passion to discover isn't always inherent, stimulating children's interest through challenges and feedback is vital. Furthermore, research indicated that active learning improves children’s aspiration, motivation, socialization, and self-esteem.

To gain practical insights, we conducted one semi-structured interview with a preschool teacher, focusing on pedagogical approaches to color learning and the use of technology in the classroom. We also conducted fieldwork at the Tekniska Museum in Malmö, observing existing interactive art installations related to color (like an RGB mixer and a color quiz, figures 1 and 2) to understand how tangible interaction could be applied effectively. We compiled insights from all sources, identifying key themes around active learning and interaction.

Figure 1. Inspiration at the Tekniska museum: an interactive RGB value mixer.

Figure 2. Inspiration at the Tekniska museum: an interactive color quiz.

Related Work & Inspiration

We also examined existing projects that shared similarities with our goals:

• I/O Brush: Extracts colors, patterns and movements of physical objects by brushing over them. The user can then freely use that color, pattern or movement to paint and draw on a computer screen. The I/O Brush engage users to incorporate their physical environment to their digital and enables users to easier mimic colors of real life.
• Tilt Brush: Is a VR application where the user can paint in three dimensions, leading to creative 3D drawings. The application inspired our concept in allowing the user to make creations in and through the environment.
• Beelight: Is a concept which includes an interactive platform shaped as a honeycomb and a color collector in form of a bee. Beelight focuses on educating children about colors through embodied and tangible interactions with beelight and the childrens environment.

Ideation: An Interactive Color Book

Synthesizing our research and inspiration, we brainstormed several concepts in an ideation session, sketching them on paper to facilitate communication and visualize ideas. Sketching proved effective for showcasing and explaining concepts visually. My core idea, which the group decided to develop further, was a digital color book utilizing computer vision to bridge physical objects and digital coloring. I was particularly interested in how children could actively engage with learning by interacting with their immediate surroundings.

In my initial sketch (figure 3), the computer would guide the child linearly, focusing on specific areas to color with hues captured directly from real-life objects held to the camera. After group discussion, we agreed the core concept of extracting colors was intriguing but decided to remove the restrictions of realistic colors and linear progression to allow for more creativity (figure 4).

The concept evolved into "Objex," featuring a physical box with an internal camera and lights designed to help children capture colors from objects placed inside its opening. The system extracts the dominant color, which the child can then apply to parts of a digital drawing. We also conceptualized two distinct modes to add fun and challenge.

Figure 3. The first sketching iteration of my chosen concept.

Figure 4. The next sketch iteration of the color book. It showcases the utility of the box and the idea of the memory mode.

Ideation: The Modes

To structure the interaction and add engaging elements, we designed two primary modes for the digital color book (figure 5):

1. Free Coloring Mode (figure 6): This mode allows children complete creative freedom. They can capture colors from any object within the recognized range and apply them to any part of the drawing as they please, letting their imagination guide them.
2. Memory Mode (figure 7): This mode introduces a challenge. A drawing with pre-assigned, randomized colors is shown for just 10 seconds. The colors then disappear, and the child's task is to recall the original colors, find objects matching those colors in their environment, capture the colors using the box, and apply them to the correct parts of the drawing. This adds a layer of memory recall and color matching engagement.

Figure 5. The GUI of the home screen showcases the coloring mode and memory mode.

Figure 6. A GIF showing the user interface of the coloring mode which showcases the free coloring mode and one of the drawings. The dynamic colored circle displays for the user which color is currently the main color.

Figure 7. A GIF showing the memory mode and its stages.

Prototype Development

As one of the programmers, I utilized HTML, CSS, and JavaScript to build the high-fidelity prototype. We integrated the `tracking.js` library for real-time color tracking via the camera feed. The library's algorithm measures color areas in pixels; we adjusted it to prioritize the largest detected color area as the main selected color to improve stability.

We implemented a palette of 6 target colors: green, blue, magenta (purple), yellow, red, and orange. However, we immediately encountered challenges in reliably distinguishing between similar hues, particularly yellow and orange, under varying lighting conditions.

Despite this technical hurdle, we created two distinct drawings using Adobe Illustrator—one for each mode—and exported them as SVG files. Using SVGs allowed individual paths and shapes within the drawings to be targeted and filled with the captured colors, enabling the core interactive coloring experience. The physical box was developed later, during the iteration phase, specifically to address the color recognition challenges.

Ethical Considerations

Before conducting user tests with children, we ensured ethical guidelines were followed. We obtained verbal informed consent from the children's guardians prior to each session. We explained the activity's purpose to the children in simple, age-appropriate language and clearly stated they could stop participating at any time they wished. Observations and any photographs taken during the tests were handled confidentially and anonymized in our reports.

First User Testing (7-year-old girl, without box)

Our initial user test involved a 7-year-old girl (figure 8). At this stage, the physical box hadn't been built, so we used the laptop's camera aimed at objects placed on a white surface. We explained the concept of coloring with real objects.

Initial Interaction & Usability: Her first instinct was asking, "Can you paint with the finger?" and attempting to interact directly with the screen. After learning to use the touchpad, she adapted quickly but struggled with precision when coloring smaller elements, requiring several attempts yet showing notable patience and concentration.

Feedback & Color Perception: The unreliable color recognition meant the wrong color was sometimes applied. The on-screen color indicator circle was often only noticed when an error occurred. In Memory Mode, she found remembering the colors difficult and tried restarting to see the original again (which wasn't possible due to randomization).

Realism & Expectations: When asked to find objects, she specifically brought colored pens, explaining her toys weren't the "right" colors for the prototype's limited palette. She had a strong desire for realism and specific colors, stating "the bubbles have to be brown." She expected exact shade matching from the objects. Even when holding glittery red nail polish, she noted the pattern but expected only the solid red color to be captured. This test highlighted a conflict between the user's desire for realism/precision and the prototype's technical limitations.

Figure 8. The first user testing with a 7-year old girl without the box.

Iteration Of The Prototype

Based on the valuable feedback from the first user test, we made key iterations (figure 9):

1. Memory Mode Aid ("Sneak Peek"): Since remembering all colors proved too difficult, we added a "Sneak Peek" button. Hovering over this button temporarily revealed the correct colors, allowing children to check their memory and complete the task without removing the challenge entirely.
2. Physical Box for Color Capture: To address the unreliable color recognition, we constructed a physical box with integrated LED lighting and placed the camera inside. This created a controlled lighting environment, allowing us to fine-tune the color detection values for significantly more reliable and consistent color capture. We also designed the box to look playful, hoping to add a sense of "magic."

Figure 9. A pilot test of the sneak peek button and memory mode that includes the box with its external camera and lights.

Second User Testing (5 & 8-year-old sisters, with box)

The second test involved two sisters, aged 5 and 8, using the iterated prototype with the physical box (figure 10). They were immediately fascinated by using real objects for coloring.

Collaboration & Engagement: They chose Memory Mode first and spontaneously began cooperating, discussing color placements. They found the memorization time sufficient but struggled with positional recall. They appreciated the sneak peek function, using it strategically only near the end when stuck (approx. 2-3 times), enabling them to finish successfully.

Realism & Environment Interaction: Echoing the first test, they preferred realistic coloring. When encouraged to find objects, they selected a wooden doll, specifically wanting both the beige color and the wood grain pattern for a tree element – highlighting the desire for texture/pattern capture.

Persistence & Usability: The girls showed remarkable patience with remaining bugs (like color misidentification) and difficulty clicking small elements. They were highly determined to complete the drawings, naturally dividing roles and even physically guiding each other's hands. They successfully completed the Free Coloring task.

Figure 10. In the second user test, two girls participated in the ages of 5 and 8.

Child-Specific UI Considerations

The user tests highlighted several UI/UX considerations crucial for designing for children:

• Target Size & Precision: Difficulty clicking small SVG elements underscores the need for larger targets or zoom, respecting children's developing motor skills.
• Direct Manipulation Preference: The instinct to use touch suggests exploring touch interfaces for better usability.
• Clear Feedback: Feedback mechanisms (like the color indicator) need to be prominent and clearly integrated into the interaction flow.

Analysis of Color Choices (Realistic vs. Fantasy)

The strong and consistent preference for realistic colors across tests suggests that children may initially focus on accurately representing the world they know. This aligns with developmental stages where understanding and categorizing the environment is key.

To encourage more creative exploration beyond realism, future iterations could incorporate specific features or prompts. Examples include adding a 'magic brush' with patterns/textures, introducing fantastical color palettes alongside realistic ones, or framing tasks explicitly around imagination ("Color the tree purple!").

Potential Application & Scalability

Objex leverages everyday objects, making it potentially usable in various contexts like preschools (connecting physical exploration with digital tools), homes (as an educational toy), or even museum workshops.

The concept is scalable:

• Age Adaptability: Drawing complexity and Memory Mode difficulty can be adjusted.
• Content Expansion: New themes can easily be added.
• Platform Flexibility: Could be adapted for tablets or interactive displays.

Future Directions

Based on user feedback and our reflections, several avenues exist for future development to enhance Objex:

• Improved Cursor Affordance: Use a paintbrush icon showing the current color.
• Color Mixing Feature: Allow interactive mixing of captured colors.
• Expanded Recognition: Improve vision techniques for more colors, shades, and patterns.
• Enhance the "Magic Box": Redesign to better conceal tech, increasing the sense of magic.
• User-Generated Content: Allow drawing simple shapes or scanning objects for coloring pages.
• Touch Interface Adaptation: Port to tablets for direct manipulation.
• Sound Feedback Integration: Add subtle sounds for successful actions.

Learning Takeaways / Conclusion

This project provided deep insights into the iterative design process, the challenges of tangible and embodied interaction, and the nuances of designing for children.

Value of User Testing with Target Audience: The most critical learning was the irreplaceable value of testing directly with children. Their unexpected interactions and clear preferences directly shaped our iterations (e.g., adding the sneak peek, building the box).

Bridging Physical & Digital: We experienced the technical challenges (color recognition) and design opportunities (physical box) in linking physical objects to digital experiences. Reliability requires careful control or advanced sensing.

Designing for Children's Abilities: The project highlighted the need to consider children's developing motor skills (precision issues) and cognitive understanding (memory limits, realism focus).

Iterative Refinement: The process clearly demonstrated how feedback loops – identifying problems (imprecise capture, memory difficulty) and implementing solutions (box, sneak peek button) – lead to an improved user experience.

This project significantly shaped my understanding of UX and Interaction Design by offering hands-on experience with the full cycle, from research and embodied ideation to technical prototyping (HTML/CSS/JS, `tracking.js`) and iterative user testing, particularly highlighting the unique considerations when designing tangible interactions for children. Seeing the children's engagement was particularly rewarding.