From 55e2b2ab00a2b500a0a1e9a75a5785853196c214 Mon Sep 17 00:00:00 2001 From: Vladimir Mandic Date: Thu, 2 Jun 2022 10:39:53 -0400 Subject: [PATCH] webgpu and wasm optimizations --- CHANGELOG.md | 6 +- TODO.md | 60 +- demo/benchmark/index.html | 67 +- demo/benchmark/node.js | 2 +- demo/typescript/index.js | 4 +- demo/typescript/index.js.map | 4 +- demo/typescript/index.ts | 3 +- dist/human.esm-nobundle.js | 12 +- dist/human.esm-nobundle.js.map | 4 +- dist/human.esm.js | 23040 +++++++++++++++--------------- dist/human.esm.js.map | 4 +- dist/human.js | 1530 +- dist/human.node-gpu.js | 8 +- dist/human.node-wasm.js | 8 +- dist/human.node.js | 8 +- dist/tfjs.esm.js | 1936 ++- package.json | 10 +- src/face/attention.ts | 10 +- src/face/faceboxes.ts | 2 +- src/face/facemesh.ts | 2 +- src/human.ts | 4 + src/tfjs/backend.ts | 13 + test/build.log | 49 +- test/test-node-wasm.js | 5 +- test/test.log | 1368 +- typedoc/assets/main.js | 4 +- typedoc/assets/search.js | 2 +- typedoc/classes/GraphModel.html | 2 +- typedoc/classes/Human.html | 54 +- typedoc/classes/Tensor.html | 12 +- typedoc/enums/Rank.html | 2 +- typedoc/index.html | 2 +- typedoc/modules/match.html | 2 +- 33 files changed, 14158 insertions(+), 14081 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2906ed31..4c03089c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,9 +9,13 @@ ## Changelog -### **HEAD -> main** 2022/05/24 mandic00@live.com +### **HEAD -> main** 2022/05/30 mandic00@live.com +### **origin/main** 2022/05/29 mandic00@live.com + +- full rebuild + ### **2.7.4** 2022/05/24 mandic00@live.com diff --git a/TODO.md b/TODO.md index 24665425..408513ef 100644 --- a/TODO.md +++ b/TODO.md @@ -1,38 +1,64 @@ # To-Do list for Human library -## Work in Progress +## Work-in-Progress -### Exploring +N/A -- Optical flow: -- Advanced histogram equalization: Adaptive, Contrast Limited, CLAHE -- TFLite models: -- Body segmentation: `robust-video-matting` +

-


+## Exploring + +- Optical flow for intelligent temporal interpolation + +- Advanced histogram equalization for optimization of badly lit scenes + **Adaptive**, **Contrast Limited**, **CLAHE** +- TFLite models + +- Body segmentation with temporal analysis + + +

## Known Issues +### Face with Attention +`FaceMesh-Landmarks` model is supported only with `CPU` and `WebGL` backends due to `TFJS` issues -#### WebGPU +### WASM -Experimental support only until support is officially added in Chromium +Support for **WASM SIMD** and **WASM MultiThreading** is still disabled by default in **Chromium** based browsers +Suggestion is to enable it manually for major performance boost +Enable via + +### WebGPU + +Experimental support only until support is officially added in Chromium +Enable via + +### Firefox + +Running in **web workers** requires `OffscreenCanvas` which is still disabled by default in **Firefox** +Enable via `about:config` -> `gfx.offscreencanvas.enabled` ### Face Detection -Enhanced rotation correction for face detection is not working in NodeJS due to missing kernel op in TFJS -Feature is automatically disabled in NodeJS without user impact +Enhanced rotation correction for face detection is not working in **NodeJS** due to missing kernel op in **TFJS** +Feature is automatically disabled in **NodeJS** without user impact -- Backend NodeJS missing kernel op `RotateWithOffset` - +

-


- -## Pending Release Notes +## Pending Release Changes - Updated **FaceMesh-Landmarks** models - Added **FaceMesh-with-Attention** model is disabled by defauls, enable using `config.face.mesh.attention = true` -- If **FaceMesh-with-Attention** model is anbled, Iris model gets disabled +- If **FaceMesh-with-Attention** model is enabled, Iris model gets disabled as its functionality is superseded by attention model - Results include more detailed face mesh annotations `result.face[].annotations` +- Update benchmarks: `demo/benchmark` +- Slight performance improvement with canvas reads using `WASM` backend +- Default to RTM version of `WASM` binaries +- Enhanced `WebGPU` compatibility +- Support for **NodeJS v18** +- Updated **Known Issues** + See diff --git a/demo/benchmark/index.html b/demo/benchmark/index.html index 918a8471..26da599f 100644 --- a/demo/benchmark/index.html +++ b/demo/benchmark/index.html @@ -17,25 +17,40 @@ html { font-family: 'Lato', 'Segoe UI'; font-size: 16px; font-variant: small-caps; } body { margin: 0; background: black; color: white; overflow-x: hidden; width: 100vw; height: 100vh; } body::-webkit-scrollbar { display: none; } - .status { position: absolute; width: 100vw; bottom: 10%; text-align: center; font-size: 3rem; font-weight: 100; text-shadow: 2px 2px #303030; } - .log { position: absolute; bottom: 0; margin: 0.4rem 0.4rem 0 0.4rem; font-size: 0.9rem; } -
-
+
Options
All
  • Public
  • Public/Protected
  • All
Menu

Index

Enumeration members

Enumeration members

R0 = "R0"
R1 = "R1"
R2 = "R2"
R3 = "R3"
R4 = "R4"
R5 = "R5"
R6 = "R6"

Legend

  • Constructor
  • Property
  • Method
  • Property

Settings

Theme

\ No newline at end of file +Rank | @vladmandic/human - v2.8.0
Options
All
  • Public
  • Public/Protected
  • All
Menu

Index

Enumeration Members

Enumeration Members

R0: "R0"
R1: "R1"
R2: "R2"
R3: "R3"
R4: "R4"
R5: "R5"
R6: "R6"

Legend

  • Constructor
  • Property
  • Method
  • Property

Settings

Theme

\ No newline at end of file diff --git a/typedoc/index.html b/typedoc/index.html index a9b8d7b9..3c4f3997 100644 --- a/typedoc/index.html +++ b/typedoc/index.html @@ -1,4 +1,4 @@ -@vladmandic/human - v2.8.0
Options
All
  • Public
  • Public/Protected
  • All
Menu

@vladmandic/human - v2.8.0

Index

References

Re-exports Descriptor
Re-exports Models
Renames and re-exports Human

Type aliases

AnyCanvas: HTMLCanvasElement | OffscreenCanvas
+@vladmandic/human - v2.8.0
Options
All
  • Public
  • Public/Protected
  • All
Menu

@vladmandic/human - v2.8.0

Index

References

Re-exports Descriptor
Re-exports Models
Renames and re-exports Human

Type Aliases

AnyCanvas: HTMLCanvasElement | OffscreenCanvas

Defines all possible canvas types

AnyImage: HTMLImageElement | typeof Image

Defines all possible image types

diff --git a/typedoc/modules/match.html b/typedoc/modules/match.html index e159b81c..4c6e8d1c 100644 --- a/typedoc/modules/match.html +++ b/typedoc/modules/match.html @@ -1,6 +1,6 @@ match | @vladmandic/human - v2.8.0
Options
All
  • Public
  • Public/Protected
  • All
Menu

Class Human as default export

-

Index

Type aliases

Descriptor: number[]
+

Index

Type Aliases

Descriptor: number[]

Face descriptor type as number array

MatchOptions: { max?: number; min?: number; multiplier?: number; order?: number; threshold?: number } | undefined

Functions