Painting with Movement

New technologies, such as Augmented Reality (AR) and Machine Learning (ML) have opened doors for hybrid projects between Art & Technology, and consequently new experimental artistic approaches. A new application concept is proposed in this work, through dance movement and AR, the user (e.g. spectator, museum visitor) can experience the three-dimensional body-movement of a performer in the physical space. One of the main goals is to record the sequence of dance movements in the real space, to further analyze and understand the use of space.

 

Usability tests

 

Through this concept idea, an AR application has been developed to record the movement of a performer and represent it in a 3D visual composition, through simple elements. The user which is regarding the dance performance can experiencing the visual recording, in real-time and walk around the space always with different perspectives about the dance movement. Painting with Movement allows us to have a clear observation of the space traveled by the dancer, but also the body movement dimension in the 3D space. This project could be use in different real-life scenarios with multiple approaches.

 

Process

The technological process includes real-time human body detection, movement visualization in real-time and movement tracking history. For this purpose, the ML model detects different parts of the human body, keypoints, and records the movement made by the keypoints detection. This project has a core focus on dance and performance, though the framework is targeted to anyone interesting in bodymovement and creative experience, only by using a smartphone camera.

 

Drawing on a grid, through body-movement

 

 

Publications

 

Nogueira, Maria Rita, Paulo Menezes, and Bruno Patrão. "Understanding Art through Augmented Reality: Exploring Mobile Tools for Everyone's Use." 2021 9th International Conference on Information and Education Technology (ICIET). IEEE, 2021.

Nogueira, Maria Rita, Paulo Menezes, and Bruno Patrão. "Painting with movement". The 17th International Conference on Virtual-Reality Continuum and its Applications in Industry. 2019.