Egocentric vision is one of the methods for understanding human interaction patterns with the surrounding environment. In this paper, we introduce EgoMask-3DGS, a novel pipeline designed to represent dynamic 3D scenes and decompose hand movements within these scenes from an egocentric perspective. EgoMask-3DGS focuses on egocentric datasets from the manufacturing industry, which feature complex hand motions and a variety of materials encountered in real-world scenarios. To accurately capture and represent diverse materials—such as transparent, deformable, and solid objects—under different lighting conditions, we use the 3D Gaussian models, depth prior information and hand prior masks to guide the dynamic scene representation process. We validate our pipeline using a custom handcrafted dataset that includes industrial scenes with different materials. Our method effectively represents these varied materials and decomposes hand movements within dynamic scenes. Experimental results highlight the potential of our approach for enhanced 3D dynamic egocentric perception and enabling precise hand movement decomposition.
We present the rendering outcomes of our pipeline. In the first row, the rendering video, using the initial camera views, is shown. From left to right, we display the ground truth video, the rendering video, and the decomposition results for both the static and hands scenes. The second row shows the results from a fixed view, with the dynamic scene, static scene, and hands layer visible from left to right.
The marble craft case involves two distinct actions: first, the right hand traces the engraved lines while the left hand steadies the tool on the table; second, both hands grip the tool to carve the marble, following the engraved lines as the body rotates to the right in a repetitive motion.
Glass craft involves the process of heating glass. In the first video, the left hand holds a glass tube, rotating it with the fingers to heat it while the right hand adjusts the heater. The second video demonstrates the process of holding the glass tube with both hands, continuously adjusting its position to ensure even heating.
Glove craft involves a detailed handcraft process, including intricate hand and finger motions. In the first video, the left hand holds the leather while the right hand searches for a tool, with many instant motions. The second video demonstrates the leather strip process, where the left hand secures the leather and the right hand uses a tool to split it.