ambiser
/
human
mirror of https://github.com/vladmandic/human
9
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

updated docs

master
Vladimir Mandic 2020-11-08 12:26:28 -05:00
parent a9c1394caf
commit e73a55ab96
7 changed files with 111 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
Build-Process.md

@ -14,9 +14,12 @@ npm run build
This will rebuild library itself (all variations) as well as demo
<br>
Project is written in pure `JavaScript` [ECMAScript version 2020](https://www.ecma-international.org/ecma-262/11.0/index.html)
Build target is `JavaScript` **EMCAScript version 2018**
Only project depdendency is [@tensorflow/tfjs](https://github.com/tensorflow/tfjs)
Development dependencies are [eslint](https://github.com/eslint) used for code linting and [esbuild](https://github.com/evanw/esbuild) used for IIFE and ESM script bundling
<br>
Only project depdendency is [@tensorflow/tfjs](https://github.com/tensorflow/tfjs)
Development dependencies are [eslint](https://github.com/eslint) used for code linting and [esbuild](https://github.com/evanw/esbuild) used for IIFE and ESM script bundling

133
Configuration.md

@ -16,27 +16,35 @@ config = {
backend: 'webgl', // select tfjs backend to use
console: true, // enable debugging output to console
async: true, // execute enabled models in parallel
// this disables per-model performance data but slightly increases performance
// this disables per-model performance data but
// slightly increases performance
// cannot be used if profiling is enabled
profile: false, // enable tfjs profiling
// this has significant performance impact, only enable for debugging purposes
// this has significant performance impact
// only enable for debugging purposes
// currently only implemented for age,gender,emotion models
deallocate: false, // aggresively deallocate gpu memory after each usage
// only valid for webgl backend and only during first call, cannot be changed unless library is reloaded
// this has significant performance impact, only enable on low-memory devices
// only valid for webgl backend and only during first call
// cannot be changed unless library is reloaded
// this has significant performance impact
// only enable on low-memory devices
scoped: false, // enable scoped runs
// some models *may* have memory leaks, this wrapps everything in a local scope at a cost of performance
// some models *may* have memory leaks,
// this wrapps everything in a local scope at a cost of performance
// typically not needed
videoOptimized: true, // perform additional optimizations when input is video, must be disabled for images
filter: { // note: image filters are only available in Browser environments and not in NodeJS as they require WebGL for processing
videoOptimized: true, // perform additional optimizations when input is video,
// must be disabled for images
// basically this skips object box boundary detection for every n frames
// while maintaining in-box detection since objects cannot move that fast
filter: {
enabled: true, // enable image pre-processing filters
return: true, // return processed canvas imagedata in result
width: 0, // resize input width
height: 0, // resize input height
// usefull on low-performance devices to reduce the size of processed input
// if both width and height are set to 0, there is no resizing
// if just one is set, second one is scaled automatically
// if both are set, values are used as-is
return: true, // return processed canvas imagedata in result
brightness: 0, // range: -1 (darken) to 1 (lighten)
contrast: 0, // range: -1 (reduce contrast) to 1 (increase contrast)
sharpness: 0, // range: 0 (no sharpening) to 1 (maximum sharpening)
@ -51,90 +59,115 @@ config = {
polaroid: false, // image polaroid camera effect
pixelate: 0, // range: 0 (no pixelate) to N (number of pixels to pixelate)
},
gesture: {
enabled: true, // enable simple gesture recognition
},
face: {
enabled: true, // controls if specified modul is enabled
// face.enabled is required for all face models: detector, mesh, iris, age, gender, emotion
// note: module is not loaded until it is required
// face.enabled is required for all face models:
// detector, mesh, iris, age, gender, emotion
// (note: module is not loaded until it is required)
detector: {
modelPath: '../models/blazeface/back/model.json', // can be 'front' or 'back'.
// 'front' is optimized for large faces such as front-facing camera and 'back' is optimized for distanct faces.
modelPath: '../models/blazeface-back.json', // can be 'front' or 'back'.
// 'front' is optimized for large faces
// such as front-facing camera and
// 'back' is optimized for distanct faces.
inputSize: 256, // fixed value: 128 for front and 256 for 'back'
maxFaces: 10, // maximum number of faces detected in the input, should be set to the minimum number for performance
skipFrames: 10, // how many frames to go without re-running the face bounding box detector
// only used for video inputs, ignored for static inputs
// if model is running st 25 FPS, we can re-use existing bounding box for updated face mesh analysis
// as the face probably hasn't moved much in short time (10 * 1/25 = 0.25 sec)
minConfidence: 0.5, // threshold for discarding a prediction
iouThreshold: 0.3, // threshold for deciding whether boxes overlap too much in non-maximum suppression
scoreThreshold: 0.7, // threshold for deciding when to remove boxes based on score in non-maximum suppression
maxFaces: 10, // maximum number of faces detected in the input
// should be set to the minimum number for performance
skipFrames: 15, // how many frames to go without re-running the face bounding box detector
// only used for video inputs
// e.g., if model is running st 25 FPS, we can re-use existing bounding
// box for updated face analysis as the head probably hasn't moved much
// in short time (10 * 1/25 = 0.25 sec)
minConfidence: 0.1, // threshold for discarding a prediction
iouThreshold: 0.1, // threshold for deciding whether boxes overlap too much in
// non-maximum suppression (0.1 means drop if overlap 10%)
scoreThreshold: 0.2, // threshold for deciding when to remove boxes based on score
// in non-maximum suppression,
// this is applied on detection objects only and before minConfidence
},
mesh: {
enabled: true,
modelPath: '../models/facemesh/model.json',
modelPath: '../models/facemesh.json',
inputSize: 192, // fixed value
},
iris: {
enabled: true,
modelPath: '../models/iris/model.json',
enlargeFactor: 2.3, // empiric tuning
modelPath: '../models/iris.json',
inputSize: 64, // fixed value
},
age: {
enabled: true,
modelPath: '../models/ssrnet-age/imdb/model.json', // can be 'imdb' or 'wiki'
// which determines training set for model
modelPath: '../models/age-ssrnet-imdb.json', // can be 'age-ssrnet-imdb' or 'age-ssrnet-wiki'
// which determines training set for model
inputSize: 64, // fixed value
skipFrames: 10, // how many frames to go without re-running the detector, only used for video inputs
skipFrames: 15, // how many frames to go without re-running the detector
// only used for video inputs
},
gender: {
enabled: true,
minConfidence: 0.8, // threshold for discarding a prediction
modelPath: '../models/ssrnet-gender/imdb/model.json',
minConfidence: 0.1, // threshold for discarding a prediction
modelPath: '../models/gender-ssrnet-imdb.json', // can be 'gender', 'gender-ssrnet-imdb' or 'gender-ssrnet-wiki'
inputSize: 64, // fixed value
skipFrames: 15, // how many frames to go without re-running the detector
// only used for video inputs
},
emotion: {
enabled: true,
inputSize: 64, // fixed value
minConfidence: 0.5, // threshold for discarding a prediction
skipFrames: 10, // how many frames to go without re-running the detector, only used for video inputs
modelPath: '../models/emotion/model.json',
minConfidence: 0.2, // threshold for discarding a prediction
skipFrames: 15, // how many frames to go without re-running the detector
modelPath: '../models/emotion-large.json', // can be 'mini', 'large'
},
},
body: {
enabled: true,
modelPath: '../models/posenet/model.json',
modelPath: '../models/posenet.json',
inputResolution: 257, // fixed value
outputStride: 16, // fixed value
maxDetections: 10, // maximum number of people detected in the input, should be set to the minimum number for performance
scoreThreshold: 0.7, // threshold for deciding when to remove boxes based on score in non-maximum suppression
maxDetections: 10, // maximum number of people detected in the input
// should be set to the minimum number for performance
scoreThreshold: 0.8, // threshold for deciding when to remove boxes based on score
// in non-maximum suppression
nmsRadius: 20, // radius for deciding points are too close in non-maximum suppression
},
hand: {
enabled: true,
inputSize: 256, // fixed value
skipFrames: 10, // how many frames to go without re-running the hand bounding box detector
skipFrames: 15, // how many frames to go without re-running the hand bounding box detector
// only used for video inputs
// if model is running st 25 FPS, we can re-use existing bounding box for updated hand skeleton analysis
// as the hand probably hasn't moved much in short time (10 * 1/25 = 0.25 sec)
// e.g., if model is running st 25 FPS, we can re-use existing bounding
// box for updated hand skeleton analysis as the hand probably
// hasn't moved much in short time (10 * 1/25 = 0.25 sec)
minConfidence: 0.5, // threshold for discarding a prediction
iouThreshold: 0.3, // threshold for deciding whether boxes overlap too much in non-maximum suppression
scoreThreshold: 0.7, // threshold for deciding when to remove boxes based on score in non-maximum suppression
enlargeFactor: 1.65, // empiric tuning as skeleton prediction prefers hand box with some whitespace
maxHands: 10, // maximum number of hands detected in the input, should be set to the minimum number for performance
iouThreshold: 0.1, // threshold for deciding whether boxes overlap too much
// in non-maximum suppression
scoreThreshold: 0.8, // threshold for deciding when to remove boxes based on
// score in non-maximum suppression
maxHands: 1, // maximum number of hands detected in the input
// should be set to the minimum number for performance
landmarks: true, // detect hand landmarks or just hand boundary box
detector: {
modelPath: '../models/handdetect/model.json',
modelPath: '../models/handdetect.json',
},
skeleton: {
modelPath: '../models/handskeleton/model.json',
modelPath: '../models/handskeleton.json',
},
},
gesture: {
enabled: true, // enable simple gesture recognition
// takes processed data and based on geometry detects simple gestures
// easily expandable via code, see `src/gesture.js`
},
};
```
<br>
Any user configuration and default configuration are merged using deep-merge, so you do not need to redefine entire configuration
Configurtion object is large, but typically you only need to modify few values:

3
Demos.md

@ -39,6 +39,8 @@ npm run dev
```
On first start, it will install all development dependencies required to rebuild `Human` library
By default, web server will run on port `8000` which is configurable in `dev-server.js:options.port`
```log
> @vladmandic/human@0.7.5 dev /home/vlado/dev/human
> npm install && node --trace-warnings --unhandled-rejections=strict --trace-uncaught --no-deprecation dev-server.js
@ -62,4 +64,3 @@ found 0 vulnerabilities
- `node.js`: Demo using NodeJS with CommonJS module
This is a very simple demo as althought `Human` library is compatible with NodeJS execution
and is able to load images and models from local filesystem,

8
Home.md

@ -27,16 +27,16 @@
- [**Performance Notes**](https://github.com/vladmandic/human/wiki/Performance)
- [**Credits**](https://github.com/vladmandic/human/wiki/Credits)
<br>
Compatible with *Browser*, *WebWorker* and *NodeJS* execution on both Windows and Linux
- Browser/WebWorker: Compatible with *CPU*, *WebGL*, *WASM* and *WebGPU* backends
- NodeJS: Compatible with software *tfjs-node* and CUDA accelerated backends *tfjs-node-gpu*
(and maybe with React-Native as it doesn't use any DOM objects)
<br>
*This is a pre-release project, see [issues](https://github.com/vladmandic/human/issues) for list of known limitations and planned enhancements*
*Suggestions are welcome!*
<br>
<hr>
<br>

7
Install.md

@ -25,6 +25,7 @@ Defaults:
"browser": "dist/human.esm.js",
}
```
<br>
### 1. [IIFE](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) script
@ -45,6 +46,8 @@ Which you can use to create instance of `human` library:
This way you can also use `Human` library within embbedded `<script>` tag within your `html` page for all-in-one approach
<br>
### 2. [ESM](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import) module
*Recommended for usage within `Browser`*
@ -90,6 +93,8 @@ Install with:
const human = new Human();
```
<br>
### 3. [NPM](https://www.npmjs.com/) module
*Recommended for `NodeJS` projects that will execute in the backend*
@ -117,6 +122,8 @@ const config = {
}
```
<br>
### Weights
Pretrained model weights are includes in `./models`

4
Performance.md

@ -4,6 +4,8 @@ Performance will vary depending on your hardware, but also on number of resoluti
For example, it can perform multiple face detections at 60+ FPS, but drops to ~15 FPS on a medium complex images if all modules are enabled
<br>
### Performance per module on a **notebook** with nVidia GTX1050 GPU on a FullHD input:
- Enabled all: 15 FPS
@ -32,5 +34,7 @@ For example, it can perform multiple face detections at 60+ FPS, but drops to ~1
- Hand: 40 FPS (standalone)
- Body: 10 FPS (standalone)
<br>
For performance details, see output of `result.performance` object during after running inference

8
Usage.md

@ -1,9 +1,11 @@
## Usage
`Human` library does not require special initialization.
All configuration is done in a single JSON object and all model weights will be dynamically loaded upon their first usage
`Human` library does not require special initialization
All configuration is done in a single JSON object and all model weights are dynamically loaded upon their first usage
(and only then, `Human` will not load weights that it doesn't need according to configuration).
<br>
There is only *ONE* method you need:
```js
@ -34,6 +36,8 @@ Additionally, `Human` library exposes several objects and methods:
// if you want to pre-load them instead of on-demand loading during 'human.detect()'
```
<br>
Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards
Input format is `Tensor4D[1, width, height, 3]` of type `float32`