Lightfarm is an educational experience which teaches smallholder farmers sustainable agriculture practices. Users explore a physical projection-mapped world, using touch inputs to maintain their own farm. The project uses visualizations to teach impact of actions, while striking a balance between educational and game components. This project was made as a part of a 2 semester long capstone project.
Role: UI/UX, Visual Design, Interaction Design, Game Design, Unity Developer, Producer
Team size: 5
Date: Sept 2019
Scope: 6 months
Tools: Unity C#, Physical Prototyping, Figma, Adobe Creative Suite, Projection Mapping, Maya
The challenge
500 million smallholder farming families produce 70% of the world’s food. Many of these farmers do everything they can to create income for their family, and maximizing their farm yields is a top concern.
A way to increase yields is through adopting sustainable practices, which will create healthier farms over time. However, it’s difficult to adapt without knowing the mechanisms behind why. Learning the “why” is even harder in developing countries like India, where there’s very limited access to free information. For Lightfarm, we partnered with Marcatus QED, an agriculture solutions company, to develop a solution that will be implemented in their field workshops.
“How might we create an interactive experience to teach farmers sustainable farming practices?”
Breaking down the problems further:
The existing solution is a workshop where field officers play videos and offer live demonstrations. In which ways might we enhance the educational component of the workshop?
Many farmers in developing countries aren’t familiar with modern technology. How might we build an experience that feels natural to use?
Farming can be a very emotional topic when it affects the family’s income. How might we create an educational experience that feels empathetic?
Solution
The final solution uses projection mapping and computer vision technology, to allow for natural and easily-learned interactions. Users explore and manage their farm, discovering how their actions impact the future of their farm. Users learn through interactive visualizations of soil quality, crop health and water use.
A world to explore.
A farm to manage.
A physical projection-mapped diorama to play with.
Visualize and learn how your actions affect the future of your farm.
Process overview:
Why projection mapping?
Without access to modern technology and electricity, most of our users are tech-illiterate. They haven’t had the opportunity to learn how to use nuanced interactions attached to mice & keyboards, or even mobile touch gestures. That’s why we explored ideas of creating a physical experience, with natural interactions.
Early AR prototype
Testing concepts in VR
Ideating different interactions with different tech
The project is designed to be deployed in workshops in India, where field officers give presentations on sustainability to groups of farmers at once. We are restricted by the amount of technology we physically bring there, so we designed an experience to leverage the projector and laptop computers they already use!
Workshops in India with educational slideshows on sustainability.
Projecting a texture onto a physical diorama created the strongest and most immersive experience, and even offered a social component where multiple users can interact at once. We found that users didn’t need a tutorial or guide on how to use it -- it felt natural to touch the diorama due to its physicality.
Usability testing of early prototype. Testing for natural-feeling touch input. Learned that unregistered touch inputs broke user immersion really quickly. Re-prioritized our tech issues to address this.
Testing with users unfamiliar with tech. Discovered that visuals and motions are the most effective for these users.
Designing for physical interaction
During tests, we came across some unexpected problems with the setup. Because we were using a projector, users would occasionally block the light coming from the projector and obscure their vision. In qualitative user tests, we found that being suddenly aware of the existence of the projector breaks the immersion of the experience.
To solve this, we dictated an optimal standing position and approach angle. With the users placed 45 degrees adjacent to the projector, we struck a balance between maximum visible projected surface, balanced with amount of shadow created.
Another consideration was the metrics of users. We prototyped varying levels of stand heights, viewing distances and diorama scales. We chose the combination that was big enough to project onto, yet didn’t strain users’ arms. We built a detachable stand separate from the diorama to adjust heights if necessary.
Product thinking challenges
PRODUCT & PLATFORM
When developing the project, we were running into situations where we had a choice. We could design something very specific to our problem, or design a general solution that can be applied to other use cases. A physical projection mapped diorama could be useful in a lot of other cases. That’s when we realized: we designed both an experience and a platform.
The challenge came from where to be specific to our problem, and where to be generic. In the end, we segmented the design, making the physical diorama completely generic but the farming experience extremely specific. This means there are no farming related visuals in the physical component.
Top portion is separate from the stand, letting us easily switch out the top portion during prototyping.
Final design is completely generic, with projected textures!
TECH OVERHAUL
For the first half of the project’s lifespan, we used Leap Motion to detect hands and touch interaction. However, it was extremely unreliable, since the device needed to detect a whole hand. We found in tests that users’ taps only registered about 30% of the time, which broke the entire experience. An interaction overhaul was necessary.
Prototyping with a bunch of tech, including Arduino, Makey-Makey, Kinect, Wacom tablets, mobile tablets and keyboards, led us to our final solution: a webcam in a box. It uses computer vision to detect the shadows made when touching the diorama surface. When testing, we found the taps now register about 95% of the time, which makes the experience actually playable. Although we lost some functionality, such as hand detection and hover capabilities, a base level of reliability was achieved.
Experimenting with Leap Motion + Wacom Tablets, Kinect, Arduino + Makey Makey, and IR cameras. We even explored crazier ideas like electric origami, IR blocking paint, using real dirt, and RC cars with AR attachments.
DIRECTION PIVOT
Halfway through the project, we were offered an opportunity to bring the project to compete in a games-focused competition. With this opportunity, we decided to pivot the project to be more game-like. We were met with a lot of challenges: how do we make it educational? Which user group do we prioritize?
In the end, we designed a simple collection quest in order to tutorialize touch interaction. The educational component remains the same, but we targeted the visuals towards western audiences.
Conclusions
The project was a great opportunity to learn about designing for physical interfaces and the challenges with creating an educational experience. I learned so much about working with unknowns, and effective ways of breakthrough during early research stages. I learned about how to validate my designs, and consider both the user and the product needs when building a prototype.
The project won Most Innovative at Intel’s GDC university games showcase 2020.
Event was held on Twitch.tv due to covid-19.