diff --git a/Backends.md b/Backends.md index fd9299e..b82b345 100644 --- a/Backends.md +++ b/Backends.md @@ -1,10 +1,10 @@ -## Backends +# Backends Human library includes pre-bundled `CPU`, `WASM` and `WebGL` backends, there is no need to load them externally.
-### Notes: +## Notes - CPU backend usage is not recommended for performance reasons as TensorFlow implementation in JavaScript is by far the slowest @@ -16,23 +16,26 @@ Human library includes pre-bundled `CPU`, `WASM` and `WebGL` backends, there is - Reduce size of input image (e.g. webcam) as WASM does not have optimized image extraction methods -#### How to enable WASM SIMD support +### How to enable WASM SIMD support Chrome: + - Navigate to - `WebAssembly SIMD support` set to Enabled Edge: + - Navigate to - `WebAssembly SIMD support` set to Enabled Firefox: + - Navigate to - `javascript.options.wasm_simd` set to True
-### Known Issues: +## Known Issues - **WASM** backend without SIMD has limited processing capabilities before it causes issues To avoid issues, start `human` with most models disabled and if needed, enable them one by one as each initializes @@ -42,4 +45,3 @@ Firefox: - **NodeJS** backend is missing key operations and thus neither `Face` nor `Hand` models are currently supported
- diff --git a/Build-Process.md b/Build-Process.md index a086ba3..3c5af89 100644 --- a/Build-Process.md +++ b/Build-Process.md @@ -1,4 +1,4 @@ -## Build Process +# Build Process If you want to modify the library and perform a full rebuild: diff --git a/Change-Log.md b/Change-Log.md index 8028b6b..2f8007a 100644 --- a/Change-Log.md +++ b/Change-Log.md @@ -1,7 +1,6 @@ - # @vladmandic/human -Version: **0.30.4** +Version: **0.30.5** Description: **Human: AI-powered 3D Face Detection, Face Embedding & Recognition, Body Pose Tracking, Hand & Finger Tracking, Iris Analysis, Age & Gender & Emotion Prediction & Gesture Recognition** Author: **Vladimir Mandic ** @@ -10,209 +9,271 @@ Repository: **** ## Changelog -### **HEAD -> main, tag: v0.30.4, origin/main, origin/HEAD** 2021/03/01 mandic00@live.com +### **HEAD -> main** 2021/03/02 mandic00@live.com + +- add debug logging flag + +### **0.30.4** 2021/03/01 mandic00@live.com + - added skipinitial flag -### **tag: v0.30.3** 2021/02/28 mandic00@live.com +### **0.30.3** 2021/02/28 mandic00@live.com + - typo -### **tag: v0.30.2** 2021/02/26 mandic00@live.com +### **0.30.2** 2021/02/26 mandic00@live.com + - rebuild - fix typo -### **tag: v0.30.1** 2021/02/25 mandic00@live.com +### **0.30.1** 2021/02/25 mandic00@live.com -### **tag: v0.20.11** 2021/02/24 mandic00@live.com -### **tag: v0.20.10** 2021/02/22 mandic00@live.com +### **0.20.11** 2021/02/24 mandic00@live.com + + +### **0.20.10** 2021/02/22 mandic00@live.com + + +### **0.20.9** 2021/02/21 mandic00@live.com -### **tag: v0.20.9** 2021/02/21 mandic00@live.com - remove extra items - simmilarity fix -### **tag: v0.20.8** 2021/02/21 mandic00@live.com +### **0.20.8** 2021/02/21 mandic00@live.com + - embedding fix -### **tag: v0.20.7** 2021/02/21 mandic00@live.com +### **0.20.7** 2021/02/21 mandic00@live.com + - build fix -### **tag: v0.20.6** 2021/02/21 mandic00@live.com +### **0.20.6** 2021/02/21 mandic00@live.com + - embedding fix -### **tag: v0.20.5** 2021/02/21 mandic00@live.com +### **0.20.5** 2021/02/21 mandic00@live.com + - fix imagefx and add dev builds -### **tag: v0.20.4, tag: 0.3, tag: 0.2, tag: 0.1** 2021/02/19 mandic00@live.com +### **0.20.4** 2021/02/19 mandic00@live.com + - 0.20.4 -### **tag: v0.20.3** 2021/02/17 mandic00@live.com +### **0.20.3** 2021/02/17 mandic00@live.com + - rebuild -### **tag: v0.20.2** 2021/02/13 mandic00@live.com +### **0.20.2** 2021/02/13 mandic00@live.com + - merge branch 'main' of https://github.com/vladmandic/human into main - create codeql-analysis.yml - create security.md - add templates -### **tag: v0.20.1** 2021/02/08 mandic00@live.com +### **0.20.1** 2021/02/08 mandic00@live.com + - menu fixes - convert to typescript -### **tag: v0.11.5** 2021/02/06 mandic00@live.com +### **0.11.5** 2021/02/06 mandic00@live.com + - added faceboxes alternative model -### **tag: v0.11.4** 2021/02/06 mandic00@live.com +### **0.11.4** 2021/02/06 mandic00@live.com -### **tag: v0.11.3** 2021/02/02 mandic00@live.com -### **tag: v0.11.2** 2021/01/30 mandic00@live.com +### **0.11.3** 2021/02/02 mandic00@live.com + + +### **0.11.2** 2021/01/30 mandic00@live.com + - added warmup for nodejs ### **update for tfjs 3.0.0** 2021/01/29 mandic00@live.com -### **tag: v0.11.1** 2021/01/29 mandic00@live.com -### **tag: v0.10.2** 2021/01/22 mandic00@live.com +### **0.11.1** 2021/01/29 mandic00@live.com + + +### **0.10.2** 2021/01/22 mandic00@live.com + + +### **0.10.1** 2021/01/20 mandic00@live.com -### **tag: v0.10.1** 2021/01/20 mandic00@live.com ### **0.9.26** 2021/01/18 mandic00@live.com + - fix face detection when mesh is disabled - added minification notes - version bump ### **0.9.25** 2021/01/13 mandic00@live.com + - added humangl custom backend - rebuild - code cleanup and enable minification - fix safari incopatibility ### **0.9.24** 2021/01/12 mandic00@live.com + - work on blazepose - full rebuild ### **0.9.23** 2021/01/11 mandic00@live.com + - added iris gesture - fix emotion labels - full rebuild ### **0.9.22** 2021/01/05 mandic00@live.com + - remove iris coords if iris is disabled - web worker fix ### **0.9.21** 2021/01/03 mandic00@live.com + ### **0.9.20** 2021/01/03 mandic00@live.com + - stricter linting, fix face annotations ### **0.9.19** 2020/12/23 mandic00@live.com + - added rawbox and rawmesh - variable name changes, setting .rawcoords only if necessary - option to return raw data (mesh, box) for facemesh / "preserve aspect ratio" fix from facemesh upstream ### **0.9.18** 2020/12/16 mandic00@live.com + - add z axis scaling - major work on body module - republish due to tfjs 2.8.0 issues ### **0.9.17** 2020/12/15 mandic00@live.com + - added custom webgl backend ### **0.9.16** 2020/12/12 mandic00@live.com + - change default ports ### **0.9.15** 2020/12/11 mandic00@live.com + - improved caching and warmup - rebuild ### **0.9.14** 2020/12/10 mandic00@live.com + - conditional hand rotation - staggered skipframes - fix permissions ### **0.9.13** 2020/12/08 mandic00@live.com + - implemented face and hand boundary checks - embedded sample for warmup - switch to central logger ### **0.9.12** 2020/11/26 mandic00@live.com + - minor compatibility fixes ### **0.9.11** 2020/11/23 mandic00@live.com + - implement multi-person gestures - modularize pipeline models ### **0.9.10** 2020/11/21 mandic00@live.com + - changed build for optimized node & browser ### **0.9.9** 2020/11/21 mandic00@live.com + - new screenshots - camera exception handling ### **0.9.8** 2020/11/19 mandic00@live.com + - force f16 textures - bugfix embedding check ### **0.9.7** 2020/11/19 mandic00@live.com + - ui redesign ### **0.9.6** 2020/11/18 mandic00@live.com + - optimize camera resize on mobile - completed tfjs wrapper ### **0.9.5** 2020/11/17 mandic00@live.com + - fix serious performance bug around skipframes - swtich to custom tfjs bundle ### **0.9.4** 2020/11/17 mandic00@live.com + - swtich to tfjs source import ### **0.9.3** 2020/11/16 mandic00@live.com + - switched to minified build -### **tag: 1.2** 2020/11/15 mandic00@live.com +### **release: 1.2** 2020/11/15 mandic00@live.com + - full rebuild ### **0.9.2** 2020/11/14 mandic00@live.com + - fix camera restart on resize ### **0.9.1** 2020/11/13 mandic00@live.com + - version bump - full rebuild - implemented face embedding - added internal benchmark tool ### **0.8.8** 2020/11/12 mandic00@live.com + - reduced bundle size - implemented buffered processing - fix for conditional model loading ### **0.8.7** 2020/11/11 mandic00@live.com + - added performance notes - added notes on models - fix bug in async ops and change imports - fix wiki links ### **0.8.6** 2020/11/09 mandic00@live.com + - add wasm bundle ### **0.8.5** 2020/11/09 mandic00@live.com + - reimplemented blazeface processing ### **0.8.4** 2020/11/09 mandic00@live.com + - added additional gestures - implemented blink detection - fix wasm module ### **0.8.3** 2020/11/08 mandic00@live.com + - refresh - optimizations ### **0.8.2** 2020/11/08 mandic00@live.com + - fix typos - commit ### **0.8.1** 2020/11/07 mandic00@live.com + - fix hand detection performance - optimized model loader - merge branch 'main' of https://github.com/vladmandic/human into main @@ -222,52 +283,65 @@ Repository: **** - fix nms sync call ### **0.7.6** 2020/11/06 mandic00@live.com + - fixed memory leaks and updated docs - model tuning - cache invalidation improvements - full async operations ### **0.7.5** 2020/11/05 mandic00@live.com + - implemented dev-server ### **0.7.4** 2020/11/05 mandic00@live.com + - fix canvas size on different orientation - switched from es2020 to es2018 build target ### **0.7.3** 2020/11/05 mandic00@live.com + - optimized camera and mobile layout - fixed worker and filter compatibility - removed test code ### **0.7.2** 2020/11/04 mandic00@live.com + - major work on handpose model ### **0.7.1** 2020/11/04 mandic00@live.com + - changed demo build process ### **0.6.7** 2020/11/04 mandic00@live.com + - implemented simple gesture recognition ### **0.6.6** 2020/11/04 mandic00@live.com + - remove debug code ### **0.6.5** 2020/11/04 mandic00@live.com + - redo hand detection ### **0.6.4** 2020/11/03 mandic00@live.com + - added manifest ### **0.6.3** 2020/11/03 mandic00@live.com + - enhanced processing resolution - fix pause restart - complete model refactoring - fixed typo ### **0.6.2** 2020/11/02 mandic00@live.com + - optimized demo - package update ### **0.6.1** 2020/11/02 mandic00@live.com + - major performance improvements for all models - revert "optimized canvas handling" - optimized canvas handling @@ -276,61 +350,80 @@ Repository: **** - added tfjs-vis to distribution ### **0.5.5** 2020/11/01 mandic00@live.com + - changed defaults ### **0.5.4** 2020/11/01 mandic00@live.com + - implemented memory profiler ### **0.5.3** 2020/10/30 mandic00@live.com + - improved debug logging ### **0.5.2** 2020/10/30 mandic00@live.com + - added wasm and webgpu backends ### **0.5.1** 2020/10/30 mandic00@live.com + - improve demo line continous draws ### **0.4.10** 2020/10/30 mandic00@live.com + - fix for seedrandom ### **0.4.9** 2020/10/29 mandic00@live.com + ### **0.4.8** 2020/10/28 mandic00@live.com + - revert "updated menu handler" - fix webpack compatibility issue ### **0.4.7** 2020/10/27 mandic00@live.com + ### **0.4.6** 2020/10/27 mandic00@live.com + - fix firefox compatibility bug ### **0.4.5** 2020/10/27 mandic00@live.com + ### **0.4.4** 2020/10/27 mandic00@live.com + - implelented input resizing ### **0.4.3** 2020/10/22 mandic00@live.com + ### **0.4.2** 2020/10/20 mandic00@live.com + - log initialization ### **0.4.1** 2020/10/19 mandic00@live.com + - breaking change: convert to object class - compatibility notes ### **0.3.9** 2020/10/18 mandic00@live.com + - implemented image filters - pure tensor pipeline without image converts - autodetect skipframes ### **0.3.8** 2020/10/17 mandic00@live.com + - new menu layout ### **0.3.7** 2020/10/17 mandic00@live.com + - added diagnostics output - parallelized agegender operations ### **0.3.6** 2020/10/17 mandic00@live.com + - fixed webcam initialization - fixed memory leaks and added scoped runs - modularized draw @@ -338,17 +431,21 @@ Repository: **** - refactored package file layout ### **0.3.5** 2020/10/16 mandic00@live.com + - added auto-generated changelog ### **0.3.4** 2020/10/16 mandic00@live.com + - added camera selection - optimized blazeface anchors - added error handling ### **0.3.3** 2020/10/15 mandic00@live.com + - added blazeface back and front models ### **0.3.2** 2020/10/15 mandic00@live.com + - reduced web worker latency - added debugging and versioning - optimized demos and added scoped runs @@ -356,37 +453,49 @@ Repository: **** ### **0.3.1** 2020/10/14 mandic00@live.com + ### **0.2.10** 2020/10/14 mandic00@live.com + - added emotion backend - module parametrization and performance monitoring - implemented multi-hand support - fixed documentation typos ### **0.2.9** 2020/10/13 mandic00@live.com + - added node build and demo ### **0.2.8** 2020/10/13 mandic00@live.com + - added example image ### **0.2.7** 2020/10/13 mandic00@live.com + - new examples ### **0.2.6** 2020/10/13 mandic00@live.com + - enable all models by default ### **0.2.5** 2020/10/12 mandic00@live.com + - fixed memory leak ### **0.2.4** 2020/10/12 mandic00@live.com + - removed extra files ### **0.2.3** 2020/10/12 mandic00@live.com + ### **0.2.2** 2020/10/12 mandic00@live.com -### **tag: 1.0** 2020/10/12 mandic00@live.com + +### **release: 1.0** 2020/10/12 mandic00@live.com + ### **0.2.1** 2020/10/12 mandic00@live.com + - added sample image - initial public commit - initial commit diff --git a/Configuration.md b/Configuration.md index 4f54a15..27eeb1c 100644 --- a/Configuration.md +++ b/Configuration.md @@ -1,4 +1,4 @@ -## Configuration +# Configuration Detailed configuration options are explained below, but they are best seen in the menus present in the `demo` application: *note: some advanced configuration options are not exposed in the UI* @@ -10,12 +10,14 @@ Detailed configuration options are explained below, but they are best seen in th
Main configuration objects are: + - **config.filter**: controls image pre-processing - **config.face**: controls face detection - **config.body**: controls body pose detection - **config.hand**: contros hand and finger detection With **config.face** having several subsections: + - **config.face.mesh**: controls facial mesh and landscape detection - **config.face.iris**: controls iris detection - **config.face.age**: controls age prediction @@ -31,7 +33,6 @@ Note that user object and default configuration are merged using deep-merge, so All configuration details can be changed in real-time! - ```js config = { backend: 'webgl', // select tfjs backend to use diff --git a/Credits.md b/Credits.md index 0483d12..d29e503 100644 --- a/Credits.md +++ b/Credits.md @@ -1,4 +1,4 @@ -## Credits +# Credits - Face Detection: [**MediaPipe BlazeFace**](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view) - Face Boxes: [**FaceBoxes**](https://github.com/TropComplique/FaceBoxes-tensorflow/) diff --git a/Demos.md b/Demos.md index 7e42acc..0727d15 100644 --- a/Demos.md +++ b/Demos.md @@ -1,10 +1,11 @@ -## Demos +# Demos Demos are included in `/demo`:
**Browser**: + - `index.html`: Full demo using Browser with ESM module, includes selectable backends and webworkers *You can run browser demo either live from git pages, by serving demo folder from your web server or use @@ -12,7 +13,7 @@ included micro http2 server with source file monitoring and dynamic rebuild*
-### Changing Demo Target +## Changing Demo Target Demo in `demo/index.html` loads `dist/demo-browser-index.js` which is built from sources in `demo`, starting with `demo/browser` This bundled version is needed since mobile browsers (e.g. Chrome on Android) do not support native modules loading yet @@ -21,10 +22,10 @@ If your target is desktop, alternatively you can load `demo/browser.js` directly
-### Demo Options +## Demo Options Demo implements several ways to use `Human` library, -all configurable in `browse.js:ui` configuration object and in the UI itself: +all configurable in `browse.js:ui` configuration object and in the UI itself: - `ui.buffered`: run detection and screen refresh in a sequence or as separate buffered functions - `ui.bufferedFPSTarget`: when using buffered execution this target fps for screen refresh @@ -41,18 +42,20 @@ all configurable in `browse.js:ui` configuration object and in the UI itself:
-### Dev Server +## Dev Server To use dev server, you must have an SSL certificate since browsers enforce *https* for webcam access You can use a provided ones self-signed certificate or configure your own Self-signed test certificate was generated using: + ```shell openssl req -x509 -newkey rsa:4096 -nodes -keyout https.key -out https.crt -days 365 -subj "/C=US/ST=Florida/L=Miami/O=@vladmandic" ``` If you want to use your own certificate, edit `server/dev.js`: + ```js const options = { key: fs.readFileSync('server/https.key'), @@ -61,9 +64,11 @@ const options = { ``` Once SSL certificates have been provided (defalt test certificate or your own), simply run + ```shell npm run dev ``` + On first start, it will install all development dependencies required to rebuild `Human` library Dev version of the library runs without code minification for easier debugging while production version is fully minified @@ -113,9 +118,11 @@ found 0 vulnerabilities 2020-12-12 10:12:55 DATA: GET/2.0 200 application/octet-stream 5502280 /models/handskeleton.bin ::ffff:192.168.0.200 ... ``` +
**NodeJS**: + - `node.js`: Demo using NodeJS with CommonJS module This is a very simple demo as althought `Human` library is compatible with NodeJS execution and is able to load images and models from local filesystem, diff --git a/Embedding.md b/Embedding.md index a66423d..8ec9c01 100644 --- a/Embedding.md +++ b/Embedding.md @@ -1,4 +1,4 @@ -## Face Feature Embedding and Simmilarity Compare +# Face Feature Embedding and Simmilarity Compare
diff --git a/Gesture.md b/Gesture.md index d755c20..c51f9f6 100644 --- a/Gesture.md +++ b/Gesture.md @@ -1,4 +1,4 @@ -## Gesture Recognition +# Gesture Recognition Gesture recognition is done by looking up coordinates of different detected landmarks @@ -21,6 +21,7 @@ There are three pre-defined methods: - "*`` forward `` up*" Example output of `result.gesture`: + ```js [ {face: "0", gesture: "facing camera"} @@ -31,4 +32,5 @@ Example output of `result.gesture`: {hand: "0", gesture: "thumb forward middlefinger up"} ] ``` + Where number after gesture refers to number of person that detection belongs to in scenes with multiple people. diff --git a/Home.md b/Home.md index c779a70..299d294 100644 --- a/Home.md +++ b/Home.md @@ -4,7 +4,7 @@
-### Project pages: +### Project pages - [**Live Demo**](https://vladmandic.github.io/human/demo/index.html) - [**Code Repository**](https://github.com/vladmandic/human) @@ -12,7 +12,7 @@ - [**Issues Tracker**](https://github.com/vladmandic/human/issues) - [**Change Log**](https://github.com/vladmandic/human/wiki/Change-Log) -### Wiki pages: +### Wiki pages - [**Home**](https://github.com/vladmandic/human/wiki) - [**Demos**](https://github.com/vladmandic/human/wiki/Demos) @@ -23,7 +23,7 @@ - [**Face Embedding and Recognition**](https://github.com/vladmandic/human/wiki/Embedding) - [**Gesture Recognition**](https://github.com/vladmandic/human/wiki/Gesture) -### Additional notes: +### Additional notes - [**Notes on Backends**](https://github.com/vladmandic/human/wiki/Backends) - [**Development Server**](https://github.com/vladmandic/human/wiki/Development-Server) @@ -35,7 +35,8 @@
-Compatible with *Browser*, *WebWorker* and *NodeJS* execution on both Windows and Linux +Compatible with *Browser*, *WebWorker* and *NodeJS* execution on both Windows and Linux + - Browser/WebWorker: Compatible with *CPU*, *WebGL*, *WASM* and *WebGPU* backends - NodeJS: Compatible with software *tfjs-node* and CUDA accelerated backends *tfjs-node-gpu* - (and maybe with React-Native as it doesn't use any DOM objects) diff --git a/Install.md b/Install.md index e676592..ba77fba 100644 --- a/Install.md +++ b/Install.md @@ -1,19 +1,19 @@ -## Installation +# Installation
-**Important** +**Important** *The packaged version of `Human` includes `TensorFlow/JS (TFJS) 3.2.0` library which can be accessed via `human.tf`* -*You should NOT manually load another instance of `tfjs` unless you're using specific `.nobudle` edition of `Human, +*You should NOT manually load another instance of `tfjs` unless you're using specific `.nobudle` edition of `Human, but if you do, be aware of possible version conflicts*
There are multiple ways to use `Human` library, pick one that suits you: -### Included +## Included - `dist/human.js`: IIFE format bundle with TFJS for Browsers - `dist/human.esm.js`: ESM format bundle with TFJS for Browsers @@ -25,6 +25,7 @@ All versions include `sourcemap` *(.map)* and build `manifest` *(.json)* While `Human` is in pre-release mode, all bundles are non-minified Defaults: + ```json { "main": "dist/human.node.js", @@ -35,7 +36,7 @@ Defaults:
-### 1. [IIFE](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) script +## 1. [IIFE](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) script *Simplest way for usage within Browser* @@ -43,10 +44,10 @@ Simply download `dist/human.js`, include it in your `HTML` file & it's ready to ```html