Creating an Interactive Sound Experience for SVA Open Studio

Weave is an immersive, interactive sound experience that lets you become a music conductor. Using hand gestures, you can create your own symphony in real-time. This sound installation was exhibited at the SVA Interaction Design Open Studio for the Hello World Class in New York City.

I believe our hands hold incredible energy, but in today’s AI-driven world, we barely use them meaningfully. I wanted to create something that would help people slow down, focus, and rediscover the power of their hands.

Using the ml5.js HandPose model, Weave detects hand movements to trigger different sounds. Moving your hand over floating orbs activates music loops, blending them into a personalized symphony. Sounds were preloaded from free online sources.

One of the hardest parts was making Weave intuitive for all ages. Getting the hand gestures to feel natural and ensuring the setup (like screen size) worked smoothly took a lot of tweaking. Blending the sounds in a way that felt harmonious was also tricky.

Watching people interact with Weave was eye-opening. Many tried gestures I hadn’t coded, like raising their hand to increase volume or making a fist to stop the sound. This feedback showed me how natural gesture-based control can be and gave me ideas for future updates.

Weave was displayed in a cozy, semi-private space with a glass wall and a projector that showcased participants’ movements. Visitors received playful cards with mystical symbols and thoughtful quotes to take home, adding to the immersive experience. People of all ages loved experimenting with sounds and seeing how their hands could "conduct" music.

Future versions of Weave will include more hand gestures, like fist-clenching to stop sounds or pinching to adjust volume. I’m also considering allowing users to upload their own sounds to make the experience even more personal.