human/TODO.md

# To-Do list for Human library

## Work-in-Progress

N/A

<hr><br>

## Exploring

- **Optical flow** for intelligent temporal interpolation  
  <https://docs.opencv.org/3.3.1/db/d7f/tutorial_js_lucas_kanade.html>
- **CLAHE** advanced histogram equalization for optimization of badly lit scenes  
- **TFLite** models  
  <https://js.tensorflow.org/api_tflite/0.0.1-alpha.4/>
- **Principal Components Analysis** for reduction of descriptor complexity  
  <https://github.com/vladmandic/human-match/blob/main/src/pca.js>  
- **Temporal guidance** for face/body segmentation  
  <https://github.com/PeterL1n/RobustVideoMatting>

<hr><br>

## Known Issues

### Face with Attention

`FaceMesh-Attention` is not supported when using `WASM` backend due to missing kernel op in **TFJS**  

### Object Detection

`NanoDet` model is not supported when using `WASM` backend due to missing kernel op in **TFJS**  

### WebGPU

Experimental support only until support is officially added in Chromium  
Enable via <chrome://flags/#enable-unsafe-webgpu>

### Firefox

Running in **web workers** requires `OffscreenCanvas` which is still disabled by default in **Firefox**  
Enable via `about:config` -> `gfx.offscreencanvas.enabled`

<hr><br>

## Pending Release Changes

- New methods [`human.webcam.*`](https://vladmandic.github.io/human/typedoc/classes/WebCam.html)  
  Enables built-in configuration and control of **WebCam** streams  
- New method [`human.video()`](https://vladmandic.github.io/human/typedoc/classes/Human.html#video)  
  Runs continous detection of an input **video**  
  instead of processing each frame manually using `human.detect()`  
- New simple demo [*Live*](https://vladmandic.github.io/human/demo/video/index.html) | [*Code*](https://github.com/vladmandic/human/blob/main/demo/video/index.html)  
  *Full HTML and JavaScript code in less than a screen*  
- New advanced demo using **BabylonJS and VRM** [*Live*](https://vladmandic.github.io/human-bjs-vrm) | [*Code*](https://github.com/vladmandic/human-bjs-vrm)
- Update **TypeDoc** generation [*Link*](https://vladmandic.github.io/human/typedoc)  
- Update **TypeDefs** bundle generation [*Link*](https://github.com/vladmandic/human/blob/main/types/human.d.ts)  
  No external dependencies  
- Fix model caching when using web workers  
- Fix `face.rotation` when using interpolation  
- Improve NodeJS resolver when using ESM  
- Update demo `demo/typescript`  
- Update demo `demo/faceid`  
- Update demo `demo/nodejs/process-folder.js`  
  and re-process `/samples` [*Link*](https://vladmandic.github.io/human/samples)
add todo list 2021-03-03 18:04:59 +01:00			`# To-Do list for Human library`

webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`## Work-in-Progress`
major update for 1.8 release candidate 2021-04-25 19:16:04 +02:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`N/A`
update todo 2021-09-26 16:09:30 +02:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`<hr><br>`
update dependencies and refresh release 2021-10-19 13:58:51 +02:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`## Exploring`

update dependencies 2022-09-27 17:51:55 +02:00			`- Optical flow for intelligent temporal interpolation`
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`<https://docs.opencv.org/3.3.1/db/d7f/tutorial_js_lucas_kanade.html>`
update dependencies 2022-09-27 17:51:55 +02:00			`- CLAHE advanced histogram equalization for optimization of badly lit scenes`
			`- TFLite models`
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`<https://js.tensorflow.org/api_tflite/0.0.1-alpha.4/>`
update dependencies 2022-09-27 17:51:55 +02:00			`- Principal Components Analysis for reduction of descriptor complexity`
			`<https://github.com/vladmandic/human-match/blob/main/src/pca.js>`
			`- Temporal guidance for face/body segmentation`
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`<https://github.com/PeterL1n/RobustVideoMatting>`

			`<hr><br>`
update todo 2021-07-31 13:43:50 +02:00
switch to async data reads 2021-08-12 15:31:16 +02:00			`## Known Issues`
release rebuild 2022-07-25 14:33:07 +02:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`### Face with Attention`
update todo 2021-07-31 13:43:50 +02:00
update typedefs and typedocs 2022-09-30 16:20:08 +02:00			`FaceMesh-Attention` is not supported when using `WASM` backend due to missing kernel op in TFJS
add softwareKernels config option 2022-08-30 16:28:33 +02:00
			`### Object Detection`

update typedefs and typedocs 2022-09-30 16:20:08 +02:00			`NanoDet` model is not supported when using `WASM` backend due to missing kernel op in TFJS
add extra face rotation prior to mesh 2021-11-16 19:07:44 +01:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`### WebGPU`

			`Experimental support only until support is officially added in Chromium`
			`Enable via <chrome://flags/#enable-unsafe-webgpu>`
update todo 2021-07-31 13:43:50 +02:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`### Firefox`

			Running in web workers requires `OffscreenCanvas` which is still disabled by default in Firefox
			Enable via `about:config` -> `gfx.offscreencanvas.enabled`

			`<hr><br>`
add liveness module and facerecognition demo 2021-11-09 20:37:50 +01:00
webgpu and wasm optimizations 2022-06-02 16:39:53 +02:00			`## Pending Release Changes`
add load monitor test 2022-07-18 14:22:19 +02:00
add human.webcam methods 2022-09-30 03:28:13 +02:00			- New methods [`human.webcam.*`](https://vladmandic.github.io/human/typedoc/classes/WebCam.html)
update typedefs and typedocs 2022-09-30 16:20:08 +02:00			`Enables built-in configuration and control of WebCam streams`
add human.webcam methods 2022-09-30 03:28:13 +02:00			- New method [`human.video()`](https://vladmandic.github.io/human/typedoc/classes/Human.html#video)
update typedefs and typedocs 2022-09-30 16:20:08 +02:00			`Runs continous detection of an input video`
			instead of processing each frame manually using `human.detect()`
			`- New simple demo [Live](https://vladmandic.github.io/human/demo/video/index.html) \| [Code](https://github.com/vladmandic/human/blob/main/demo/video/index.html)`
			`Full HTML and JavaScript code in less than a screen`
			`- New advanced demo using BabylonJS and VRM [Live](https://vladmandic.github.io/human-bjs-vrm) \| [Code](https://github.com/vladmandic/human-bjs-vrm)`
			`- Update TypeDoc generation [Link](https://vladmandic.github.io/human/typedoc)`
			`- Update TypeDefs bundle generation [Link](https://github.com/vladmandic/human/blob/main/types/human.d.ts)`
			`No external dependencies`
			`- Fix model caching when using web workers`
			- Fix `face.rotation` when using interpolation
add human.video method 2022-09-17 23:19:51 +02:00			`- Improve NodeJS resolver when using ESM`
add human.webcam methods 2022-09-30 03:28:13 +02:00			- Update demo `demo/typescript`
update faceid 2022-09-25 16:15:47 +02:00			- Update demo `demo/faceid`
update samples 2022-09-19 16:46:11 +02:00			- Update demo `demo/nodejs/process-folder.js`
update typedefs and typedocs 2022-09-30 16:20:08 +02:00			and re-process `/samples` [Link](https://vladmandic.github.io/human/samples)