ambiser
/
human
mirror of https://github.com/vladmandic/human
9
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

update wiki pages

master
Vladimir Mandic 2021-11-10 11:53:01 -05:00
parent 60b5007b96
commit 2a937c42e7
4 changed files with 188 additions and 447 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

353
Demos.md

@ -1,347 +1,26 @@
# Demos
Demos are included in `/demo`:
All demos are included in `/demo` and come with individual documentation per-demo
## Browser Demos
- `index.html` & `index.js`: [browser]
Main browser demo app that showcases all Human capabilities:
- Optional web workers
- Optional synchronous or asynchronous workflow
- Interpolation & smoothing
- Image processing
- `typescript`: [browser]
Simple demo in WebCam processing demo in TypeScript
- `facematch`: [browser]
Extracts faces from images, calculates face descriptors and simmilarities and matches them to known database
- `facerecognition`: [browser]
Runs multiple checks to validate webcam input before performing face match, similar to *FaceID*
- `multithread`: [browser]
Runs each `human` module in a separate web worker for highest possible performance
See <https://github.com/vladmandic/human/demo/facematch/node-match.md> for details
- `face3d`: [browser]
Uses WebCam as input and draws 3D render of face mesh using `Three.js`
- **Full** [[*Live*]](https://vladmandic.github.io/human/demo/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo): Main browser demo app that showcases all Human capabilities
- **Simple** [[*Live*]](https://vladmandic.github.io/human/demo/typescript/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/typescript): Simple demo in WebCam processing demo in TypeScript
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and simmilarities and matches them to known database
- **Face Recognition** [[*Live*]](https://vladmandic.github.io/human/demo/facerecognition/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facerecognition): Runs multiple checks to validate webcam input before performing face match, similar to *FaceID*
- **Multi-thread** [[*Live*]](https://vladmandic.github.io/human/demo/multithread/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/multithread): Runs each `human` module in a separate web worker for highest possible performance
- **Face 3D** [[*Live*]](https://vladmandic.github.io/human/demo/face3d/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/face3d): Uses WebCam as input and draws 3D render of face mesh using `Three.js`
- **Virtual Avatar** [[*Live*]](https://vladmandic.github.io/human-vrm/src/human-vrm.html) [[*Details*]](https://github.com/vladmandic/human-vrm): VR model with head, face, eye, body and hand tracking
## NodeJS Demos
- `nodejs/node`: [nodejs]
Process images from files, folders or URLs
- `nodejs/node-canvas`: [nodejs]
Process image from file or URL and draw results to a new image file using `node-canvas`
- `nodejs/node-multiprocess` & `nodejs/node-multiprocess-worker`: [nodejs]
Parallel processing in `human` **detect** in multiple child worker processes
- `facematch/node-match` & `facematch/node-match-worker`: [nodejs]
Parallel processing of face **match** in multiple child worker threads
- `nodejs/node-video`: [nodejs]
Processing of video input using `ffmpeg`
- `nodejs/node-webcam`: [nodejs]
Processing of webcam screenshots using `fswebcam`
## External Demos
- `Human-VRM`
VR model with head, face, eye, body and hand tracking
Using [`three`](https://github.com/mrdoob/three.js) for 3D display and scene management and [`@pixiv/three-vrm`](https://github.com/pixiv/three-vrm) for VR model mapping
[Code](https://github.com/vladmandic/human-vrm) | [Demo](https://vladmandic.github.io/human-vrm/src/human-vrm.html)
- **Main** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process images from files, folders or URLs using native methods
- **Canvas** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process image from file or URL and draw results to a new image file using `node-canvas`
- **Video** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of video input using `ffmpeg`
- **WebCam** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of webcam screenshots using `fswebcam`
- **Events** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Showcases usage of `Human` eventing to get notifications on processing
- **Similarity** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Compares two input images for similarity of detected faces
- **Face Match** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Parallel processing of face **match** in multiple child worker threads
- **Multiple Workers** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Runs multiple parallel `human` by dispaching them to pool of pre-created worker processes
<br><hr><br>
## Main Demo
- `index.html`: Full demo using `Human` ESM module running in Browsers,
includes selectable backends and WebWorkers
*You can run browser demo either live from git pages, by serving demo folder from your web server or use
included micro http2 server with source file monitoring and dynamic rebuild*
On notes on how to use built-in micro server, see notes on [**Development Server**](https://github.com/vladmandic/human/wiki/Development-Server)
<br>
### Demo Inputs
Demo is in `demo/index.html` loads `demo/index.js`
Demo can process:
- Sample images
- WebCam input
- WebRTC input
Note that WebRTC connection requires a WebRTC server that provides a compatible media track such as H.264 video track
For such a WebRTC server implementation see <https://github.com/vladmandic/stream-rtsp> project
that implements a connection to IP Security camera using RTSP protocol and transcodes it to WebRTC
ready to be consumed by a client such as `Human`
<br>
### Demo Options
Demo implements several ways to use `Human` library,
all configurable in `browse.js:ui` configuration object and in the UI itself:
```js
const ui = {
crop: true, // video mode crop to size or leave full frame
columns: 2, // when processing sample images create this many columns
facing: true, // camera facing front or back
useWorker: false, // use web workers for processing
worker: 'index-worker.js',
samples: ['../assets/sample6.jpg', '../assets/sample1.jpg', '../assets/sample4.jpg', '../assets/sample5.jpg', '../assets/sample3.jpg', '../assets/sample2.jpg'],
compare: '../assets/sample-me.jpg',
useWebRTC: false, // use webrtc as camera source instead of local webcam
webRTCServer: 'http://localhost:8002',
webRTCStream: 'reowhite',
console: true, // log messages to browser console
maxFPSframes: 10, // keep fps history for how many frames
modelsPreload: true, // preload human models on startup
modelsWarmup: true, // warmup human models on startup
busy: false, // internal camera busy flag
buffered: true, // should output be buffered between frames
bench: true, // show gl fps benchmark window
};
```
Additionally, some parameters are held inside `Human` instance:
```ts
human.draw.options = {
color: <string>'rgba(173, 216, 230, 0.3)', // 'lightblue' with light alpha channel
labelColor: <string>'rgba(173, 216, 230, 1)', // 'lightblue' with dark alpha channel
shadowColor: <string>'black',
font: <string>'small-caps 16px "Segoe UI"',
lineHeight: <number>20,
lineWidth: <number>6,
pointSize: <number>2,
roundRect: <number>28,
drawPoints: <Boolean>false,
drawLabels: <Boolean>true,
drawBoxes: <Boolean>true,
drawPolygons: <Boolean>true,
fillPolygons: <Boolean>false,
useDepth: <Boolean>true,
useCurves: <Boolean>false,
bufferedOutput: <Boolean>true,
useRawBoxes: <Boolean>false,
};
```
Demo app can use URL parameters to override configuration values
For example:
- Force using `WASM` as backend: <https://vladmandic.github.io/human/demo/index.html?backend=wasm>
- Enable `WebWorkers`: <https://vladmandic.github.io/human/demo/index.html?worker=true>
- Skip pre-loading and warming up: <https://vladmandic.github.io/human/demo/index.html?preload=false&warmup=false>
<br><hr><br>
## Face 3D Rendering using OpenGL
- `face3d`: Demo for Browsers that uses `Three.js` for 3D OpenGL rendering of a detected face
<br><hr><br>
## Face Recognition Demo
- `demo/facematch`: Demo for Browsers that uses all face description and embedding features to
detect, extract and identify all faces plus calculate simmilarity between them
It highlights functionality such as:
- Loading images
- Extracting faces from images
- Calculating face embedding descriptors
- Finding face similarity and sorting them by similarity
- Finding best face match based on a known list of faces and printing matches
<br><hr><br>
## Multithreaded Demo
- `demo/multithread`: Demo for Browsers that uses multiple worker processes
- Showcases maximum performance sinch each `human` model runs in a separate worker thread
while main thread just combines and draws results
- Achieves 20+ detection frame rate and 60+ refresh frame rate on a medium hardware
<br><hr><br>
## NodeJS Demo
- `nodejs/node.js`: Demo using NodeJS with CommonJS module
Simple demo that can process any input image
Note that you can run demo as-is and it will perform detection on provided sample images,
or you can pass a path to image to analyze, either on local filesystem or using URL
```shell
node demo/nodejs/node.js
```
```json
2021-06-01 08:52:15 INFO: @vladmandic/human version 2.0.0
2021-06-01 08:52:15 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0
2021-06-01 08:52:15 INFO: Current folder: /home/vlado/dev/human
2021-06-01 08:52:15 INFO: Human: 2.0.0
2021-06-01 08:52:15 INFO: Active Configuration {
backend: 'tensorflow',
modelBasePath: 'file://models/',
wasmPath: '../node_modules/@tensorflow/tfjs-backend-wasm/dist/',
debug: true,
async: false,
warmup: 'full',
cacheSensitivity: 0.75,
filter: {
enabled: true,
width: 0,
height: 0,
flip: true,
return: true,
brightness: 0,
contrast: 0,
sharpness: 0,
blur: 0,
saturation: 0,
hue: 0,
negative: false,
sepia: false,
vintage: false,
kodachrome: false,
technicolor: false,
polaroid: false,
pixelate: 0
},
gesture: { enabled: true },
face: {
enabled: true,
detector: { modelPath: 'blazeface.json', rotation: false, maxDetected: 10, skipFrames: 15, minConfidence: 0.2, iouThreshold: 0.1, return: false, enabled: true },
mesh: { enabled: true, modelPath: 'facemesh.json' },
iris: { enabled: true, modelPath: 'iris.json' },
description: { enabled: true, modelPath: 'faceres.json', skipFrames: 16, minConfidence: 0.1 },
emotion: { enabled: true, minConfidence: 0.1, skipFrames: 17, modelPath: 'emotion.json' }
},
body: { enabled: true, modelPath: 'movenet-lightning.json', maxDetected: 1, minConfidence: 0.2 },
hand: {
enabled: true,
rotation: true,
skipFrames: 18,
minConfidence: 0.1,
iouThreshold: 0.1,
maxDetected: 2,
landmarks: true,
detector: { modelPath: 'handdetect.json' },
skeleton: { modelPath: 'handskeleton.json' }
},
object: { enabled: true, modelPath: 'mb3-centernet.json', minConfidence: 0.2, iouThreshold: 0.4, maxDetected: 10, skipFrames: 19 }
}
08:52:15.673 Human: version: 2.0.0
08:52:15.674 Human: tfjs version: 3.6.0
08:52:15.674 Human: platform: linux x64
08:52:15.674 Human: agent: NodeJS v16.0.0
08:52:15.674 Human: setting backend: tensorflow
08:52:15.710 Human: load model: file://models/blazeface.json
08:52:15.743 Human: load model: file://models/facemesh.json
08:52:15.744 Human: load model: file://models/iris.json
08:52:15.760 Human: load model: file://models/emotion.json
08:52:15.847 Human: load model: file://models/handdetect.json
08:52:15.847 Human: load model: file://models/handskeleton.json
08:52:15.914 Human: load model: file://models/movenet-lightning.json
08:52:15.957 Human: load model: file://models/mb3-centernet.json
08:52:16.015 Human: load model: file://models/faceres.json
08:52:16.015 Human: tf engine state: 50796152 bytes 1318 tensors
2021-06-01 08:52:16 INFO: Loaded: [ 'face', 'movenet', 'handpose', 'emotion', 'centernet', 'faceres', [length]: 6 ]
2021-06-01 08:52:16 INFO: Memory state: { unreliable: true, numTensors: 1318, numDataBuffers: 1318, numBytes: 50796152 }
2021-06-01 08:52:16 INFO: Loading image: private/daz3d/daz3d-kiaria-02.jpg
2021-06-01 08:52:16 STATE: Processing: [ 1, 1300, 1000, 3, [length]: 4 ]
2021-06-01 08:52:17 DATA: Results:
2021-06-01 08:52:17 DATA: Face: #0 boxScore:0.88 faceScore:1 age:16.3 genderScore:0.97 gender:female emotionScore:0.85 emotion:happy iris:61.05
2021-06-01 08:52:17 DATA: Body: #0 score:0.82 keypoints:17
2021-06-01 08:52:17 DATA: Hand: #0 score:0.89
2021-06-01 08:52:17 DATA: Hand: #1 score:0.97
2021-06-01 08:52:17 DATA: Gesture: face#0 gesture:facing left
2021-06-01 08:52:17 DATA: Gesture: body#0 gesture:leaning right
2021-06-01 08:52:17 DATA: Gesture: hand#0 gesture:pinky forward middlefinger up
2021-06-01 08:52:17 DATA: Gesture: hand#1 gesture:pinky forward middlefinger up
2021-06-01 08:52:17 DATA: Gesture: iris#0 gesture:looking left
2021-06-01 08:52:17 DATA: Object: #0 score:0.55 label:person
2021-06-01 08:52:17 DATA: Object: #1 score:0.23 label:bottle
2021-06-01 08:52:17 DATA: Persons:
2021-06-01 08:52:17 DATA: #0: Face:score:1 age:16.3 gender:female iris:61.05 Body:score:0.82 keypoints:17 LeftHand:no RightHand:yes Gestures:4
```
<br><hr><br>
## NodeJS Multi-process Demo
- `nodejs/node-multiprocess.js` and `nodejs/node-multiprocess-worker.js`: Demo using NodeJS with CommonJS module
Demo that starts n child worker processes for parallel execution
```shell
node demo/nodejs/node-multiprocess.js
```
```json
2021-06-01 08:54:19 INFO: @vladmandic/human version 2.0.0
2021-06-01 08:54:19 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0
2021-06-01 08:54:19 INFO: FaceAPI multi-process test
2021-06-01 08:54:19 STATE: Enumerated images: ./assets 15
2021-06-01 08:54:19 STATE: Main: started worker: 130362
2021-06-01 08:54:19 STATE: Main: started worker: 130363
2021-06-01 08:54:19 STATE: Main: started worker: 130369
2021-06-01 08:54:19 STATE: Main: started worker: 130370
2021-06-01 08:54:20 STATE: Worker: PID: 130370 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
2021-06-01 08:54:20 STATE: Worker: PID: 130362 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
2021-06-01 08:54:20 STATE: Worker: PID: 130369 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
2021-06-01 08:54:20 STATE: Worker: PID: 130363 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130370
2021-06-01 08:54:21 INFO: Latency: worker initializtion: 1348 message round trip: 0
2021-06-01 08:54:21 DATA: Worker received message: 130370 { test: true }
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130362
2021-06-01 08:54:21 DATA: Worker received message: 130362 { image: 'samples/ai-face.jpg' }
2021-06-01 08:54:21 DATA: Worker received message: 130370 { image: 'samples/ai-body.jpg' }
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130369
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130363
2021-06-01 08:54:21 DATA: Worker received message: 130369 { image: 'assets/human-sample-upper.jpg' }
2021-06-01 08:54:21 DATA: Worker received message: 130363 { image: 'assets/sample-me.jpg' }
2021-06-01 08:54:24 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1
2021-06-01 08:54:24 STATE: Main: dispatching to worker: 130362
2021-06-01 08:54:24 DATA: Worker received message: 130362 { image: 'assets/sample1.jpg' }
2021-06-01 08:54:25 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 0 objects: 1
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130369
2021-06-01 08:54:25 DATA: Main: worker finished: 130370 detected faces: 1 bodies: 1 hands: 0 objects: 1
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130370
2021-06-01 08:54:25 DATA: Worker received message: 130369 { image: 'assets/sample2.jpg' }
2021-06-01 08:54:25 DATA: Main: worker finished: 130363 detected faces: 1 bodies: 1 hands: 0 objects: 2
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130363
2021-06-01 08:54:25 DATA: Worker received message: 130370 { image: 'assets/sample3.jpg' }
2021-06-01 08:54:25 DATA: Worker received message: 130363 { image: 'assets/sample4.jpg' }
2021-06-01 08:54:30 DATA: Main: worker finished: 130362 detected faces: 3 bodies: 1 hands: 0 objects: 7
2021-06-01 08:54:30 STATE: Main: dispatching to worker: 130362
2021-06-01 08:54:30 DATA: Worker received message: 130362 { image: 'assets/sample5.jpg' }
2021-06-01 08:54:31 DATA: Main: worker finished: 130369 detected faces: 3 bodies: 1 hands: 0 objects: 5
2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130369
2021-06-01 08:54:31 DATA: Worker received message: 130369 { image: 'assets/sample6.jpg' }
2021-06-01 08:54:31 DATA: Main: worker finished: 130363 detected faces: 4 bodies: 1 hands: 2 objects: 2
2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130363
2021-06-01 08:54:39 STATE: Main: worker exit: 130370 0
2021-06-01 08:54:39 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1
2021-06-01 08:54:39 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 1 objects: 3
2021-06-01 08:54:39 STATE: Main: worker exit: 130362 0
2021-06-01 08:54:39 STATE: Main: worker exit: 130369 0
2021-06-01 08:54:41 DATA: Main: worker finished: 130363 detected faces: 9 bodies: 1 hands: 0 objects: 10
2021-06-01 08:54:41 STATE: Main: worker exit: 130363 0
2021-06-01 08:54:41 INFO: Processed: 15 images in total: 22006 ms working: 20658 ms average: 1377 ms
```
<br><hr><br>
## NodeJS Demo for Video Input Processing
- `nodejs/node-video.js`: Demo that uses `ffmpeg` to decode video input (can be a file, stream or device such as webcam) and output results in a pipe that are captured by demo app as frames and processed by Human library
<br><hr><br>
## NodeJS Demo for WebCam Screenshot Processing
- `nodejs/node-webcam.js`: Demo that uses `fswebcam` to connect to web cam and take screenshots at regular interval which are then processed by Human library

79
Inputs.md Normal file

@ -0,0 +1,79 @@
# Inputs
- In **Browser** environments, actual browser provides functionality of decoding inputs (e.g. JPG image or MP4 video)
- In **NodeJS** environments, decoding must be performed manually
<br>
## Valid Inputs in Browser Environments
`Human` allows input to be in many different formats and will perform automatic processing of inputs to interally required format
```ts
type Input = Tensor | AnyCanvas | AnyImage | AnyVideo | ImageObjects | ExternalCanvas;
type AnyCanvas = HTMLCanvasElement | OffscreenCanvas;
type AnyImage = HTMLImageElement | typeof Image
type AnyVideo = HTMLMediaElement | HTMLVideoElement
type ImageObjects = ImageData | ImageBitmap
type ExternalCanvas = typeof env.Canvas | typeof globalThis.Canvas;
```
<br>
## Examples of Input processing in NodeJS
### 1. Using decode functionality from `tfjs-node`:
All primary functionality of `Human` is available, but `human.draw` methods cannot be used as `canvas` implementation is not present
```js
const tf = require('@tensorflow/tfjs-node');
const buffer = fs.readFileSync(inputFile); // read file content into a binary buffer
const tensor = human.tf.node.decodeImage(buffer); // decode jpg/png data to raw pixels
const result = await human.detect(tensor); // perform processing
human.tf.dispose(tensor); // dispose input data, required when working with tensors
```
*Note:* For all processing, correct input tensor **shape** `[1, height, width, 3]` and **dtype** `float32`
- 1 means batch number and is a fixed value
- 3 means number of channels so 3 is used for RGB format
However `Human` will automatically convert input tensor to a correct shape
- if batch number is omitted
- if input image is 4-channels such as in **RGBA** images with alpha channel
- if input tensor is in different data type such as `int32`
### 2. Using Canvas functionality from `node-canvas`
By instructing `Human` to use 3rd party module for `canvas` operations
This method allows `Human` to use `human.draw.*` methods in **NodeJS**
```js
const canvas = require('canvas');
globalThis.Canvas = canvas.Canvas; // patch global namespace with canvas library
globalThis.ImageData = canvas.ImageData; // patch global namespace with canvas library
// human.env.Canvas = canvas.Canvas; // alternatively monkey-patch human to use external canvas library
// human.env.ImageData = canvas.ImageData; // alternatively monkey-patch human to use external canvas library
const inputImage = await canvas.loadImage(inputFile); // load image using canvas library
const inputCanvas = new canvas.Canvas(inputImage.width, inputImage.height); // create canvas
const ctx = inputCanvas.getContext('2d');
ctx.drawImage(inputImage, 0, 0); // draw input image onto canvas
const result = await human.detect(inputCanvas);
```
### 3. Using Canvas functionality from `node-canvas`
Using `node-canvas` to load and decode input files only
```js
const canvas = require('canvas');
const img = await canvas.loadImage(inputFile); // read and decode image file
const canvas = canvas.createCanvas(img.width, img.height); // create canvas element
const ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0, img.width, img.height); // draw loaded image onto canvas
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height); // read pixel data from canvas
const tensor = human.tf.tensor(imageData.data); // create tensor from pixel data
const result = await human.detect(tensor); // perform processing
human.tf.dispose(tensor); // dispose input data, required when working with tensors
```

14
Models.md

@ -4,17 +4,17 @@
Default models in Human library are:
- **Face Detection**: MediaPipe BlazeFace - Back variation
- **Face Detection**: MediaPipe BlazeFace Back variation
- **Face Mesh**: MediaPipe FaceMesh
- **Face Iris Analysis**: MediaPipe Iris
- **Face Description**: HSE FaceRes
- **Emotion Detection**: Oarriaga Emotion
- **Body Analysis**: MoveNet - Lightning variation
- **Hand Analysis**: MediaPipe Hands
- **Body Segmentation**: Google Selfie
- **Object Detection**: MB3 CenterNet
- **Body Segmentation**: Google Selfie
- **Face Anti-Spoofing**: Real-or-Fake
- **Body Analysis**: MoveNet Lightning variation
- **Hand Analysis**: HandTrack combined with MediaPipe Hands
- **Object Detection**: MB3 CenterNet (not enabled by default)
- **Body Segmentation**: Google Selfie (not enabled by default)
- **Face Anti-Spoofing**: Real-or-Fake (not enabled by default)
- **Face Live Detection**: Liveness (not enabled by default)
<br>

189
Usage.md

@ -13,75 +13,117 @@ All configuration is done in a single JSON object and all model weights are dyna
There is only *ONE* method you need:
```js
// 'image': can be of any type of an image object: HTMLImage, HTMLVideo, HTMLMedia, Canvas, Tensor4D
// 'config': optional parameter used to override any options present in default configuration
// configuration is fully dynamic and can change between different calls to 'detect()'
const result = await human.detect(image, config?)
const human = new Human(config?) // create instance of human
const result = await human.detect(input) // run detection
```
or if you want to use promises
```js
human.detect(image, config?).then((result) => {
human.detect(input, config?).then((result) => {
// your code
})
```
## Results Smoothing
- [**Valid Inputs**](https://github.com/vladmandic/human/wiki/Inputs)
- [**Configuration Details**](https://github.com/vladmandic/human/wiki/Config)
If you're processing video input, you may want to interpolate results for smoother output
After calling `detect` method, simply call `next` method when you need a new interpolated frame
If no other methods are called, `Human` will
1. select best detected engine
2. use default configuration
3. load required models
4. perform warmup operations
5. preprocess input
6. perform detection
`Result` Parameter to `next` method is optional and if not provided, it will use last known result
<br>
Example that performs single detection and then draws new interpolated result at 50 frames per second:
## Results Caching and Smoothing
- By default, `Human` uses frame change detection for results caching
- For on-screen display best results, it is recommended to use results smoothing
For details, see <https://github.com/vladmandic/human/wiki/Caching
## Demos
`Human` library comes with number of **browser-based** and **nodejs-based** demo apps in `/demo` folder
For details, see <https://github.com/vladmandic/human/wiki/Demos>
<br>
## Human Properties
`Human` library exposes several dynamically generated properties:
```js
const result = await human.detect(image, config?)
setInterval(() => {
const interpolated = human.next(result);
human.draw.all(canvas, result);
}, 1000 / 50);
human.version // string containing version of human library
human.config // access to current configuration object
// normally set during call to constructor or as parameter to detect()
human.result // access to last known result object, normally returned via call to detect()
human.performance // access to current performance counters
human.state // <string> describing current operation in progress
// progresses through: 'config', 'check', 'backend', 'load', 'run:<model>', 'idle'
human.models // dynamically maintained list of loaded models
human.env // detected platform capabilities
human.events // container for events dispateched by human
Human.defaults // static property of Human class that contains default configuration
```
## Extra Properties and Methods
<br>
`Human` library exposes several additional objects and methods:
## Human Methods
### General
General purpose methods exposed by `Human`
```js
human.version // string containing version of human library
human.config // access to configuration object, normally set as parameter to detect()
human.result // access to last known result object, normally returned via call to detect()
human.performance // access to last known performance counters
human.state // <string> describing current operation in progress
// progresses through: 'config', 'check', 'backend', 'load', 'run:<model>', 'idle'
human.sysinfo // object containing current client platform and agent
human.load(config) // explicitly call load method that loads configured models
// if you want to pre-load them instead of on-demand loading during 'human.detect()'
human.image(image, config?) // runs image processing without detection and returns canvas
human.warmup(config, image? // warms up human library for faster initial execution after loading
// if image is not provided, it will generate internal sample
human.models // dynamically maintained list of object of any loaded models
human.tf // instance of tfjs used by human
human.reset() // reset current configuration to default values
human.validate(config?) // validate configuration values
// if config is not provided it will validate current values
human.load(config?) // explicitly call load method that loads configured models
human.image(input, config?) // runs image processing without detection and returns canvas and tensor
human.warmup(config?) // warms up human library for faster initial execution after loading
human.next(result?) // returns time variant smoothened/interpolated result based on last known result
```
## Face Recognition
### Utility
Utility methods exposed by `Human` that can be used in advanced cases but are typically not needed
```js
human.init() // explict backend initialization
human.validate(config?) // validate configuration values
human.reset() // reset current configuration to default values
human.now() // returns platform-independent timestamp, used for performance measurements
human.profile(input, config?) // runs detection with profiling enabled and returns information on top-20 kernels
human.compare(input1, input2) // runs pixel-compare on two different inputs and returns score
// internally used to detect frame-changes and cache validations
```
### TensorFlow
`Human` internally uses `TensorFlow/JS` for all ML processing
Access to interal instance of `tfjs` used by `human` is possible via:
```js
human.tf // instance of tfjs used by human, can be embedded or externally loaded
```
### Face Recognition
Additional functions used for face recognition:
For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding)
```js
human.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors
// vectors for source and target must be previously detected using
// face.description module
human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces
human.enhance(face) // returns enhanced tensor of a previously detected face that can be used for visualizations
human.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors
// vectors for source and target must be previously detected using
// face.description module
human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces
human.distance(descriptor1, descriptor2) // checks algorithmic distance between two descriptors
// opposite of `similarity`
human.enhance(face) // returns enhanced tensor of a previously detected face
// that can be used for visualizations
```
## Input Segmentation and Backgroun Removal or Replacement
### Input Segmentation and Backgroun Removal or Replacement
`Human` library can attempt to detect outlines of people in provided input and either remove background from input
or replace it with a user-provided background image
@ -94,7 +136,7 @@ For details on parameters and return values see [API Documentation](https://vlad
human.segmentation(input, background);
```
## Draw Functions
### Draw Functions
Additional helper functions inside `human.draw`:
@ -125,71 +167,12 @@ Style of drawing is configurable via `human.draw.options` object:
drawLabels: true, // draw labels with detection results
drawBoxes: true, // draw boxes around detected faces
roundRect: 8, // should boxes have round corners and rounding value
drawGestures: true, // should draw gestures in top-left part of the canvas
drawGaze: true, // should draw gaze arrows
drawPolygons: true, // draw polygons such as body and face mesh
fillPolygons: false, // fill polygons in face mesh
useDepth: true, // use z-axis value when available to determine color shade
useCurves: false, // draw polygons and boxes using smooth curves instead of lines
bufferedOutput: true, // experimental: buffer and interpolate results between frames
```
<br>
## Example Video Processing
Example simple app that uses Human to process video input and
draw output on screen using internal draw helper functions
```js
import Human from '@vladmandic/human';
// create instance of human with simple configuration using default values
const config = { backend: 'webgl' };
const human = new Human(config);
function detectVideo() {
// select input HTMLVideoElement and output HTMLCanvasElement from page
const inputVideo = document.getElementById('video-id');
const outputCanvas = document.getElementById('canvas-id');
// perform processing using default configuration
human.detect(inputVideo).then((result) => {
// result object will contain detected details
// as well as the processed canvas itself
// so lets first draw processed frame on canvas
human.draw.canvas(result.canvas, outputCanvas);
// then draw results on the same canvas
human.draw.face(outputCanvas, result.face);
human.draw.body(outputCanvas, result.body);
human.draw.hand(outputCanvas, result.hand);
human.draw.gesture(outputCanvas, result.gesture);
// loop immediate to next frame
requestAnimationFrame(detectVideo);
});
}
detectVideo();
```
<br>
## Example for NodeJS
Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards
Input format is `Tensor4D[1, width, height, 3]` of type `float32`
For example:
```js
const imageFile = '../assets/sample1.jpg';
const buffer = fs.readFileSync(imageFile);
const decoded = tf.node.decodeImage(buffer);
const casted = decoded.toFloat();
const image = casted.expandDims(0);
decoded.dispose();
casted.dispose();
logger.log('Processing:', image.shape);
const human = new Human.Human();
const result = await human.detect(image, config);
image.dispose();
```
<br><hr>