We present a proof-of-concept system exploring multimodal interaction with AI-infused Shape-Changing Interfaces. Our prototype integrates inFORCE, a 10x5 pin-based shape display, with AI tools for 3D mesh generation and editing. Users can create and modify 3D shapes through speech, gesture, and tangible inputs. We demonstrate potential applications including AI-assisted 3D modeling, adaptive physical controllers, and dynamic furniture. Our implementation, which translates text to point clouds for physical rendering, reveals both the potential and challenges of combining AI with shape-changing interfaces. This work explores how AI can enhance tangible interaction with 3D information and opens up new possibilities for multimodal shape-changing UIs.
2023
DA Wand: Distortion-Aware Selection Using Neural Mesh Parameterization
Richard Liu, Noam Aigerman, Vladimir G. Kim, and 1 more author
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2023
Blender extension available at \urlhttps://extensions.blender.org/approval-queue/da-wand/
