human/TODO.md

To-Do list for Human library

Work in Progress

Models

Hand

• Evaluate and switch default default model from handdetect to handtrack

Body

• Implement new variations of BlazePose models
• Add virtual box frame caching to MoveNet

Face

• Reimplement BlazeFace, FaceMesh, Iris with new pipeline and frame caching

Backends

WebGL

• Optimize shader packing for WebGL backend:
  https://github.com/tensorflow/tfjs/issues/5343

WASM

• Backend WASM incorrect handling of int32 tensors
  https://github.com/tensorflow/tfjs/issues/5641

WebGPU

Implementation of WebGPU backend
Experimental support only until support is officially added in Chromium

• Evaluate WGSL vs GLSL for WebGPU
• Backend WebGPU missing kernel ops https://github.com/tensorflow/tfjs/issues/5496
• Backend WebGPU incompatible with web workers https://github.com/tensorflow/tfjs/issues/5467
• Backend WebGPU incompatible with sync read calls https://github.com/tensorflow/tfjs/issues/5468

Exploring

• Optical Flow: https://docs.opencv.org/3.3.1/db/d7f/tutorial_js_lucas_kanade.html
• TFLite Models: https://js.tensorflow.org/api_tflite/0.0.1-alpha.4/

---

Known Issues

Face Detection

Enhanced rotation correction for face detection is not working in NodeJS due to missing kernel op in TFJS
Feature is automatically disabled in NodeJS without user impact

• Backend NodeJS missing kernel op RotateWithOffset
  https://github.com/tensorflow/tfjs/issues/5473

Hand Detection

Enhanced rotation correction for hand detection is not working in NodeJS due to missing kernel op in TFJS
Feature is automatically disabled in NodeJS without user impact

• Backend NodeJS missing kernel op RotateWithOffset
  https://github.com/tensorflow/tfjs/issues/5473

Body Detection

MoveNet MultiPose model does not work with WASM backend due to missing F32 broadcast implementation

• Backend WASM missing F32 broadcat implementation
  https://github.com/tensorflow/tfjs/issues/5516

Object Detection

Object detection using CenterNet or NanoDet models is not working when using WASM backend due to missing kernel ops in TFJS

• Backend WASM missing kernel op Mod
  https://github.com/tensorflow/tfjs/issues/5110
• Backend WASM missing kernel op SparseToDense
  https://github.com/tensorflow/tfjs/issues/4824