Similar to PoseNet, the input is image to video. Where is the hand, crop the image and detect the landmark for the model. It has 21 3D keypoints for the 2 hands.
Has 2 models.
They used 30k real world images manually annotated with 21 31 coordinates. They made this depth map. Because creating or detecting real life data is really time consuming, they also generated synthetic hand model that was rendered over various backgrounds. Combining both data worked really well.
There are a lot of models that input the rgb images and do an estimation with 0-1 value with each pixels to know if its closer or further from the camera.
They also used GHUM model to convert data to 3D models?
multimodal. You can make the model have conditional audio like conditioning through the audio. Works like the image generation algorithm that starts from noise. Input is audio and output is the video. There’s pre training part or post training part also. These are to help fine tune the model.