mirror of https://github.com/vladmandic/human
update wiki pages
Vladimir Mandic
parent
60b5007b96
commit
2a937c42e7
353
Demos.md
353
Demos.md
|
|
@ -1,347 +1,26 @@
|
||||||
# Demos
|
# Demos
|
||||||
|
|
||||||
Demos are included in `/demo`:
|
All demos are included in `/demo` and come with individual documentation per-demo
|
||||||
|
|
||||||
## Browser Demos
|
## Browser Demos
|
||||||
|
|
||||||
- `index.html` & `index.js`: [browser]
|
- **Full** [[*Live*]](https://vladmandic.github.io/human/demo/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo): Main browser demo app that showcases all Human capabilities
|
||||||
Main browser demo app that showcases all Human capabilities:
|
- **Simple** [[*Live*]](https://vladmandic.github.io/human/demo/typescript/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/typescript): Simple demo in WebCam processing demo in TypeScript
|
||||||
- Optional web workers
|
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and simmilarities and matches them to known database
|
||||||
- Optional synchronous or asynchronous workflow
|
- **Face Recognition** [[*Live*]](https://vladmandic.github.io/human/demo/facerecognition/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facerecognition): Runs multiple checks to validate webcam input before performing face match, similar to *FaceID*
|
||||||
- Interpolation & smoothing
|
- **Multi-thread** [[*Live*]](https://vladmandic.github.io/human/demo/multithread/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/multithread): Runs each `human` module in a separate web worker for highest possible performance
|
||||||
- Image processing
|
- **Face 3D** [[*Live*]](https://vladmandic.github.io/human/demo/face3d/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/face3d): Uses WebCam as input and draws 3D render of face mesh using `Three.js`
|
||||||
- `typescript`: [browser]
|
- **Virtual Avatar** [[*Live*]](https://vladmandic.github.io/human-vrm/src/human-vrm.html) [[*Details*]](https://github.com/vladmandic/human-vrm): VR model with head, face, eye, body and hand tracking
|
||||||
Simple demo in WebCam processing demo in TypeScript
|
|
||||||
- `facematch`: [browser]
|
|
||||||
Extracts faces from images, calculates face descriptors and simmilarities and matches them to known database
|
|
||||||
- `facerecognition`: [browser]
|
|
||||||
Runs multiple checks to validate webcam input before performing face match, similar to *FaceID*
|
|
||||||
- `multithread`: [browser]
|
|
||||||
Runs each `human` module in a separate web worker for highest possible performance
|
|
||||||
See <https://github.com/vladmandic/human/demo/facematch/node-match.md> for details
|
|
||||||
- `face3d`: [browser]
|
|
||||||
Uses WebCam as input and draws 3D render of face mesh using `Three.js`
|
|
||||||
|
|
||||||
## NodeJS Demos
|
## NodeJS Demos
|
||||||
|
|
||||||
- `nodejs/node`: [nodejs]
|
- **Main** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process images from files, folders or URLs using native methods
|
||||||
Process images from files, folders or URLs
|
- **Canvas** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process image from file or URL and draw results to a new image file using `node-canvas`
|
||||||
- `nodejs/node-canvas`: [nodejs]
|
- **Video** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of video input using `ffmpeg`
|
||||||
Process image from file or URL and draw results to a new image file using `node-canvas`
|
- **WebCam** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of webcam screenshots using `fswebcam`
|
||||||
- `nodejs/node-multiprocess` & `nodejs/node-multiprocess-worker`: [nodejs]
|
- **Events** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Showcases usage of `Human` eventing to get notifications on processing
|
||||||
Parallel processing in `human` **detect** in multiple child worker processes
|
- **Similarity** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Compares two input images for similarity of detected faces
|
||||||
- `facematch/node-match` & `facematch/node-match-worker`: [nodejs]
|
- **Face Match** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Parallel processing of face **match** in multiple child worker threads
|
||||||
Parallel processing of face **match** in multiple child worker threads
|
- **Multiple Workers** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Runs multiple parallel `human` by dispaching them to pool of pre-created worker processes
|
||||||
- `nodejs/node-video`: [nodejs]
|
|
||||||
Processing of video input using `ffmpeg`
|
|
||||||
- `nodejs/node-webcam`: [nodejs]
|
|
||||||
Processing of webcam screenshots using `fswebcam`
|
|
||||||
|
|
||||||
## External Demos
|
|
||||||
|
|
||||||
- `Human-VRM`
|
|
||||||
VR model with head, face, eye, body and hand tracking
|
|
||||||
Using [`three`](https://github.com/mrdoob/three.js) for 3D display and scene management and [`@pixiv/three-vrm`](https://github.com/pixiv/three-vrm) for VR model mapping
|
|
||||||
[Code](https://github.com/vladmandic/human-vrm) | [Demo](https://vladmandic.github.io/human-vrm/src/human-vrm.html)
|
|
||||||
|
|
||||||
|
|
||||||
<br><hr><br>
|
<br><hr><br>
|
||||||
|
|
||||||
## Main Demo
|
|
||||||
|
|
||||||
- `index.html`: Full demo using `Human` ESM module running in Browsers,
|
|
||||||
includes selectable backends and WebWorkers
|
|
||||||
|
|
||||||
*You can run browser demo either live from git pages, by serving demo folder from your web server or use
|
|
||||||
included micro http2 server with source file monitoring and dynamic rebuild*
|
|
||||||
|
|
||||||
On notes on how to use built-in micro server, see notes on [**Development Server**](https://github.com/vladmandic/human/wiki/Development-Server)
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
### Demo Inputs
|
|
||||||
|
|
||||||
Demo is in `demo/index.html` loads `demo/index.js`
|
|
||||||
|
|
||||||
Demo can process:
|
|
||||||
|
|
||||||
- Sample images
|
|
||||||
- WebCam input
|
|
||||||
- WebRTC input
|
|
||||||
|
|
||||||
Note that WebRTC connection requires a WebRTC server that provides a compatible media track such as H.264 video track
|
|
||||||
For such a WebRTC server implementation see <https://github.com/vladmandic/stream-rtsp> project
|
|
||||||
that implements a connection to IP Security camera using RTSP protocol and transcodes it to WebRTC
|
|
||||||
ready to be consumed by a client such as `Human`
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
### Demo Options
|
|
||||||
|
|
||||||
Demo implements several ways to use `Human` library,
|
|
||||||
all configurable in `browse.js:ui` configuration object and in the UI itself:
|
|
||||||
|
|
||||||
```js
|
|
||||||
const ui = {
|
|
||||||
crop: true, // video mode crop to size or leave full frame
|
|
||||||
columns: 2, // when processing sample images create this many columns
|
|
||||||
facing: true, // camera facing front or back
|
|
||||||
useWorker: false, // use web workers for processing
|
|
||||||
worker: 'index-worker.js',
|
|
||||||
samples: ['../assets/sample6.jpg', '../assets/sample1.jpg', '../assets/sample4.jpg', '../assets/sample5.jpg', '../assets/sample3.jpg', '../assets/sample2.jpg'],
|
|
||||||
compare: '../assets/sample-me.jpg',
|
|
||||||
useWebRTC: false, // use webrtc as camera source instead of local webcam
|
|
||||||
webRTCServer: 'http://localhost:8002',
|
|
||||||
webRTCStream: 'reowhite',
|
|
||||||
console: true, // log messages to browser console
|
|
||||||
maxFPSframes: 10, // keep fps history for how many frames
|
|
||||||
modelsPreload: true, // preload human models on startup
|
|
||||||
modelsWarmup: true, // warmup human models on startup
|
|
||||||
busy: false, // internal camera busy flag
|
|
||||||
buffered: true, // should output be buffered between frames
|
|
||||||
bench: true, // show gl fps benchmark window
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
Additionally, some parameters are held inside `Human` instance:
|
|
||||||
|
|
||||||
```ts
|
|
||||||
human.draw.options = {
|
|
||||||
color: <string>'rgba(173, 216, 230, 0.3)', // 'lightblue' with light alpha channel
|
|
||||||
labelColor: <string>'rgba(173, 216, 230, 1)', // 'lightblue' with dark alpha channel
|
|
||||||
shadowColor: <string>'black',
|
|
||||||
font: <string>'small-caps 16px "Segoe UI"',
|
|
||||||
lineHeight: <number>20,
|
|
||||||
lineWidth: <number>6,
|
|
||||||
pointSize: <number>2,
|
|
||||||
roundRect: <number>28,
|
|
||||||
drawPoints: <Boolean>false,
|
|
||||||
drawLabels: <Boolean>true,
|
|
||||||
drawBoxes: <Boolean>true,
|
|
||||||
drawPolygons: <Boolean>true,
|
|
||||||
fillPolygons: <Boolean>false,
|
|
||||||
useDepth: <Boolean>true,
|
|
||||||
useCurves: <Boolean>false,
|
|
||||||
bufferedOutput: <Boolean>true,
|
|
||||||
useRawBoxes: <Boolean>false,
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
Demo app can use URL parameters to override configuration values
|
|
||||||
For example:
|
|
||||||
|
|
||||||
- Force using `WASM` as backend: <https://vladmandic.github.io/human/demo/index.html?backend=wasm>
|
|
||||||
- Enable `WebWorkers`: <https://vladmandic.github.io/human/demo/index.html?worker=true>
|
|
||||||
- Skip pre-loading and warming up: <https://vladmandic.github.io/human/demo/index.html?preload=false&warmup=false>
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## Face 3D Rendering using OpenGL
|
|
||||||
|
|
||||||
- `face3d`: Demo for Browsers that uses `Three.js` for 3D OpenGL rendering of a detected face
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## Face Recognition Demo
|
|
||||||
|
|
||||||
- `demo/facematch`: Demo for Browsers that uses all face description and embedding features to
|
|
||||||
detect, extract and identify all faces plus calculate simmilarity between them
|
|
||||||
|
|
||||||
It highlights functionality such as:
|
|
||||||
|
|
||||||
- Loading images
|
|
||||||
- Extracting faces from images
|
|
||||||
- Calculating face embedding descriptors
|
|
||||||
- Finding face similarity and sorting them by similarity
|
|
||||||
- Finding best face match based on a known list of faces and printing matches
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## Multithreaded Demo
|
|
||||||
|
|
||||||
- `demo/multithread`: Demo for Browsers that uses multiple worker processes
|
|
||||||
|
|
||||||
- Showcases maximum performance sinch each `human` model runs in a separate worker thread
|
|
||||||
while main thread just combines and draws results
|
|
||||||
- Achieves 20+ detection frame rate and 60+ refresh frame rate on a medium hardware
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## NodeJS Demo
|
|
||||||
|
|
||||||
- `nodejs/node.js`: Demo using NodeJS with CommonJS module
|
|
||||||
Simple demo that can process any input image
|
|
||||||
|
|
||||||
Note that you can run demo as-is and it will perform detection on provided sample images,
|
|
||||||
or you can pass a path to image to analyze, either on local filesystem or using URL
|
|
||||||
|
|
||||||
```shell
|
|
||||||
node demo/nodejs/node.js
|
|
||||||
```
|
|
||||||
|
|
||||||
```json
|
|
||||||
2021-06-01 08:52:15 INFO: @vladmandic/human version 2.0.0
|
|
||||||
2021-06-01 08:52:15 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0
|
|
||||||
2021-06-01 08:52:15 INFO: Current folder: /home/vlado/dev/human
|
|
||||||
2021-06-01 08:52:15 INFO: Human: 2.0.0
|
|
||||||
2021-06-01 08:52:15 INFO: Active Configuration {
|
|
||||||
backend: 'tensorflow',
|
|
||||||
modelBasePath: 'file://models/',
|
|
||||||
wasmPath: '../node_modules/@tensorflow/tfjs-backend-wasm/dist/',
|
|
||||||
debug: true,
|
|
||||||
async: false,
|
|
||||||
warmup: 'full',
|
|
||||||
cacheSensitivity: 0.75,
|
|
||||||
filter: {
|
|
||||||
enabled: true,
|
|
||||||
width: 0,
|
|
||||||
height: 0,
|
|
||||||
flip: true,
|
|
||||||
return: true,
|
|
||||||
brightness: 0,
|
|
||||||
contrast: 0,
|
|
||||||
sharpness: 0,
|
|
||||||
blur: 0,
|
|
||||||
saturation: 0,
|
|
||||||
hue: 0,
|
|
||||||
negative: false,
|
|
||||||
sepia: false,
|
|
||||||
vintage: false,
|
|
||||||
kodachrome: false,
|
|
||||||
technicolor: false,
|
|
||||||
polaroid: false,
|
|
||||||
pixelate: 0
|
|
||||||
},
|
|
||||||
gesture: { enabled: true },
|
|
||||||
face: {
|
|
||||||
enabled: true,
|
|
||||||
detector: { modelPath: 'blazeface.json', rotation: false, maxDetected: 10, skipFrames: 15, minConfidence: 0.2, iouThreshold: 0.1, return: false, enabled: true },
|
|
||||||
mesh: { enabled: true, modelPath: 'facemesh.json' },
|
|
||||||
iris: { enabled: true, modelPath: 'iris.json' },
|
|
||||||
description: { enabled: true, modelPath: 'faceres.json', skipFrames: 16, minConfidence: 0.1 },
|
|
||||||
emotion: { enabled: true, minConfidence: 0.1, skipFrames: 17, modelPath: 'emotion.json' }
|
|
||||||
},
|
|
||||||
body: { enabled: true, modelPath: 'movenet-lightning.json', maxDetected: 1, minConfidence: 0.2 },
|
|
||||||
hand: {
|
|
||||||
enabled: true,
|
|
||||||
rotation: true,
|
|
||||||
skipFrames: 18,
|
|
||||||
minConfidence: 0.1,
|
|
||||||
iouThreshold: 0.1,
|
|
||||||
maxDetected: 2,
|
|
||||||
landmarks: true,
|
|
||||||
detector: { modelPath: 'handdetect.json' },
|
|
||||||
skeleton: { modelPath: 'handskeleton.json' }
|
|
||||||
},
|
|
||||||
object: { enabled: true, modelPath: 'mb3-centernet.json', minConfidence: 0.2, iouThreshold: 0.4, maxDetected: 10, skipFrames: 19 }
|
|
||||||
}
|
|
||||||
08:52:15.673 Human: version: 2.0.0
|
|
||||||
08:52:15.674 Human: tfjs version: 3.6.0
|
|
||||||
08:52:15.674 Human: platform: linux x64
|
|
||||||
08:52:15.674 Human: agent: NodeJS v16.0.0
|
|
||||||
08:52:15.674 Human: setting backend: tensorflow
|
|
||||||
08:52:15.710 Human: load model: file://models/blazeface.json
|
|
||||||
08:52:15.743 Human: load model: file://models/facemesh.json
|
|
||||||
08:52:15.744 Human: load model: file://models/iris.json
|
|
||||||
08:52:15.760 Human: load model: file://models/emotion.json
|
|
||||||
08:52:15.847 Human: load model: file://models/handdetect.json
|
|
||||||
08:52:15.847 Human: load model: file://models/handskeleton.json
|
|
||||||
08:52:15.914 Human: load model: file://models/movenet-lightning.json
|
|
||||||
08:52:15.957 Human: load model: file://models/mb3-centernet.json
|
|
||||||
08:52:16.015 Human: load model: file://models/faceres.json
|
|
||||||
08:52:16.015 Human: tf engine state: 50796152 bytes 1318 tensors
|
|
||||||
2021-06-01 08:52:16 INFO: Loaded: [ 'face', 'movenet', 'handpose', 'emotion', 'centernet', 'faceres', [length]: 6 ]
|
|
||||||
2021-06-01 08:52:16 INFO: Memory state: { unreliable: true, numTensors: 1318, numDataBuffers: 1318, numBytes: 50796152 }
|
|
||||||
2021-06-01 08:52:16 INFO: Loading image: private/daz3d/daz3d-kiaria-02.jpg
|
|
||||||
2021-06-01 08:52:16 STATE: Processing: [ 1, 1300, 1000, 3, [length]: 4 ]
|
|
||||||
2021-06-01 08:52:17 DATA: Results:
|
|
||||||
2021-06-01 08:52:17 DATA: Face: #0 boxScore:0.88 faceScore:1 age:16.3 genderScore:0.97 gender:female emotionScore:0.85 emotion:happy iris:61.05
|
|
||||||
2021-06-01 08:52:17 DATA: Body: #0 score:0.82 keypoints:17
|
|
||||||
2021-06-01 08:52:17 DATA: Hand: #0 score:0.89
|
|
||||||
2021-06-01 08:52:17 DATA: Hand: #1 score:0.97
|
|
||||||
2021-06-01 08:52:17 DATA: Gesture: face#0 gesture:facing left
|
|
||||||
2021-06-01 08:52:17 DATA: Gesture: body#0 gesture:leaning right
|
|
||||||
2021-06-01 08:52:17 DATA: Gesture: hand#0 gesture:pinky forward middlefinger up
|
|
||||||
2021-06-01 08:52:17 DATA: Gesture: hand#1 gesture:pinky forward middlefinger up
|
|
||||||
2021-06-01 08:52:17 DATA: Gesture: iris#0 gesture:looking left
|
|
||||||
2021-06-01 08:52:17 DATA: Object: #0 score:0.55 label:person
|
|
||||||
2021-06-01 08:52:17 DATA: Object: #1 score:0.23 label:bottle
|
|
||||||
2021-06-01 08:52:17 DATA: Persons:
|
|
||||||
2021-06-01 08:52:17 DATA: #0: Face:score:1 age:16.3 gender:female iris:61.05 Body:score:0.82 keypoints:17 LeftHand:no RightHand:yes Gestures:4
|
|
||||||
```
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## NodeJS Multi-process Demo
|
|
||||||
|
|
||||||
- `nodejs/node-multiprocess.js` and `nodejs/node-multiprocess-worker.js`: Demo using NodeJS with CommonJS module
|
|
||||||
Demo that starts n child worker processes for parallel execution
|
|
||||||
|
|
||||||
```shell
|
|
||||||
node demo/nodejs/node-multiprocess.js
|
|
||||||
```
|
|
||||||
|
|
||||||
```json
|
|
||||||
2021-06-01 08:54:19 INFO: @vladmandic/human version 2.0.0
|
|
||||||
2021-06-01 08:54:19 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0
|
|
||||||
2021-06-01 08:54:19 INFO: FaceAPI multi-process test
|
|
||||||
2021-06-01 08:54:19 STATE: Enumerated images: ./assets 15
|
|
||||||
2021-06-01 08:54:19 STATE: Main: started worker: 130362
|
|
||||||
2021-06-01 08:54:19 STATE: Main: started worker: 130363
|
|
||||||
2021-06-01 08:54:19 STATE: Main: started worker: 130369
|
|
||||||
2021-06-01 08:54:19 STATE: Main: started worker: 130370
|
|
||||||
2021-06-01 08:54:20 STATE: Worker: PID: 130370 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
|
|
||||||
2021-06-01 08:54:20 STATE: Worker: PID: 130362 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
|
|
||||||
2021-06-01 08:54:20 STATE: Worker: PID: 130369 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
|
|
||||||
2021-06-01 08:54:20 STATE: Worker: PID: 130363 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow
|
|
||||||
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130370
|
|
||||||
2021-06-01 08:54:21 INFO: Latency: worker initializtion: 1348 message round trip: 0
|
|
||||||
2021-06-01 08:54:21 DATA: Worker received message: 130370 { test: true }
|
|
||||||
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130362
|
|
||||||
2021-06-01 08:54:21 DATA: Worker received message: 130362 { image: 'samples/ai-face.jpg' }
|
|
||||||
2021-06-01 08:54:21 DATA: Worker received message: 130370 { image: 'samples/ai-body.jpg' }
|
|
||||||
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130369
|
|
||||||
2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130363
|
|
||||||
2021-06-01 08:54:21 DATA: Worker received message: 130369 { image: 'assets/human-sample-upper.jpg' }
|
|
||||||
2021-06-01 08:54:21 DATA: Worker received message: 130363 { image: 'assets/sample-me.jpg' }
|
|
||||||
2021-06-01 08:54:24 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1
|
|
||||||
2021-06-01 08:54:24 STATE: Main: dispatching to worker: 130362
|
|
||||||
2021-06-01 08:54:24 DATA: Worker received message: 130362 { image: 'assets/sample1.jpg' }
|
|
||||||
2021-06-01 08:54:25 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 0 objects: 1
|
|
||||||
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130369
|
|
||||||
2021-06-01 08:54:25 DATA: Main: worker finished: 130370 detected faces: 1 bodies: 1 hands: 0 objects: 1
|
|
||||||
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130370
|
|
||||||
2021-06-01 08:54:25 DATA: Worker received message: 130369 { image: 'assets/sample2.jpg' }
|
|
||||||
2021-06-01 08:54:25 DATA: Main: worker finished: 130363 detected faces: 1 bodies: 1 hands: 0 objects: 2
|
|
||||||
2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130363
|
|
||||||
2021-06-01 08:54:25 DATA: Worker received message: 130370 { image: 'assets/sample3.jpg' }
|
|
||||||
2021-06-01 08:54:25 DATA: Worker received message: 130363 { image: 'assets/sample4.jpg' }
|
|
||||||
2021-06-01 08:54:30 DATA: Main: worker finished: 130362 detected faces: 3 bodies: 1 hands: 0 objects: 7
|
|
||||||
2021-06-01 08:54:30 STATE: Main: dispatching to worker: 130362
|
|
||||||
2021-06-01 08:54:30 DATA: Worker received message: 130362 { image: 'assets/sample5.jpg' }
|
|
||||||
2021-06-01 08:54:31 DATA: Main: worker finished: 130369 detected faces: 3 bodies: 1 hands: 0 objects: 5
|
|
||||||
2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130369
|
|
||||||
2021-06-01 08:54:31 DATA: Worker received message: 130369 { image: 'assets/sample6.jpg' }
|
|
||||||
2021-06-01 08:54:31 DATA: Main: worker finished: 130363 detected faces: 4 bodies: 1 hands: 2 objects: 2
|
|
||||||
2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130363
|
|
||||||
2021-06-01 08:54:39 STATE: Main: worker exit: 130370 0
|
|
||||||
2021-06-01 08:54:39 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1
|
|
||||||
2021-06-01 08:54:39 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 1 objects: 3
|
|
||||||
2021-06-01 08:54:39 STATE: Main: worker exit: 130362 0
|
|
||||||
2021-06-01 08:54:39 STATE: Main: worker exit: 130369 0
|
|
||||||
2021-06-01 08:54:41 DATA: Main: worker finished: 130363 detected faces: 9 bodies: 1 hands: 0 objects: 10
|
|
||||||
2021-06-01 08:54:41 STATE: Main: worker exit: 130363 0
|
|
||||||
2021-06-01 08:54:41 INFO: Processed: 15 images in total: 22006 ms working: 20658 ms average: 1377 ms
|
|
||||||
```
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## NodeJS Demo for Video Input Processing
|
|
||||||
|
|
||||||
- `nodejs/node-video.js`: Demo that uses `ffmpeg` to decode video input (can be a file, stream or device such as webcam) and output results in a pipe that are captured by demo app as frames and processed by Human library
|
|
||||||
|
|
||||||
<br><hr><br>
|
|
||||||
|
|
||||||
## NodeJS Demo for WebCam Screenshot Processing
|
|
||||||
|
|
||||||
- `nodejs/node-webcam.js`: Demo that uses `fswebcam` to connect to web cam and take screenshots at regular interval which are then processed by Human library
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,79 @@
|
||||||
|
# Inputs
|
||||||
|
|
||||||
|
- In **Browser** environments, actual browser provides functionality of decoding inputs (e.g. JPG image or MP4 video)
|
||||||
|
- In **NodeJS** environments, decoding must be performed manually
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
## Valid Inputs in Browser Environments
|
||||||
|
|
||||||
|
`Human` allows input to be in many different formats and will perform automatic processing of inputs to interally required format
|
||||||
|
|
||||||
|
```ts
|
||||||
|
type Input = Tensor | AnyCanvas | AnyImage | AnyVideo | ImageObjects | ExternalCanvas;
|
||||||
|
type AnyCanvas = HTMLCanvasElement | OffscreenCanvas;
|
||||||
|
type AnyImage = HTMLImageElement | typeof Image
|
||||||
|
type AnyVideo = HTMLMediaElement | HTMLVideoElement
|
||||||
|
type ImageObjects = ImageData | ImageBitmap
|
||||||
|
type ExternalCanvas = typeof env.Canvas | typeof globalThis.Canvas;
|
||||||
|
```
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
## Examples of Input processing in NodeJS
|
||||||
|
|
||||||
|
### 1. Using decode functionality from `tfjs-node`:
|
||||||
|
|
||||||
|
All primary functionality of `Human` is available, but `human.draw` methods cannot be used as `canvas` implementation is not present
|
||||||
|
|
||||||
|
```js
|
||||||
|
const tf = require('@tensorflow/tfjs-node');
|
||||||
|
const buffer = fs.readFileSync(inputFile); // read file content into a binary buffer
|
||||||
|
const tensor = human.tf.node.decodeImage(buffer); // decode jpg/png data to raw pixels
|
||||||
|
const result = await human.detect(tensor); // perform processing
|
||||||
|
human.tf.dispose(tensor); // dispose input data, required when working with tensors
|
||||||
|
```
|
||||||
|
|
||||||
|
*Note:* For all processing, correct input tensor **shape** `[1, height, width, 3]` and **dtype** `float32`
|
||||||
|
- 1 means batch number and is a fixed value
|
||||||
|
- 3 means number of channels so 3 is used for RGB format
|
||||||
|
|
||||||
|
However `Human` will automatically convert input tensor to a correct shape
|
||||||
|
- if batch number is omitted
|
||||||
|
- if input image is 4-channels such as in **RGBA** images with alpha channel
|
||||||
|
- if input tensor is in different data type such as `int32`
|
||||||
|
|
||||||
|
|
||||||
|
### 2. Using Canvas functionality from `node-canvas`
|
||||||
|
|
||||||
|
By instructing `Human` to use 3rd party module for `canvas` operations
|
||||||
|
This method allows `Human` to use `human.draw.*` methods in **NodeJS**
|
||||||
|
|
||||||
|
```js
|
||||||
|
const canvas = require('canvas');
|
||||||
|
globalThis.Canvas = canvas.Canvas; // patch global namespace with canvas library
|
||||||
|
globalThis.ImageData = canvas.ImageData; // patch global namespace with canvas library
|
||||||
|
// human.env.Canvas = canvas.Canvas; // alternatively monkey-patch human to use external canvas library
|
||||||
|
// human.env.ImageData = canvas.ImageData; // alternatively monkey-patch human to use external canvas library
|
||||||
|
const inputImage = await canvas.loadImage(inputFile); // load image using canvas library
|
||||||
|
const inputCanvas = new canvas.Canvas(inputImage.width, inputImage.height); // create canvas
|
||||||
|
const ctx = inputCanvas.getContext('2d');
|
||||||
|
ctx.drawImage(inputImage, 0, 0); // draw input image onto canvas
|
||||||
|
const result = await human.detect(inputCanvas);
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Using Canvas functionality from `node-canvas`
|
||||||
|
|
||||||
|
Using `node-canvas` to load and decode input files only
|
||||||
|
|
||||||
|
```js
|
||||||
|
const canvas = require('canvas');
|
||||||
|
const img = await canvas.loadImage(inputFile); // read and decode image file
|
||||||
|
const canvas = canvas.createCanvas(img.width, img.height); // create canvas element
|
||||||
|
const ctx = canvas.getContext('2d');
|
||||||
|
ctx.drawImage(img, 0, 0, img.width, img.height); // draw loaded image onto canvas
|
||||||
|
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height); // read pixel data from canvas
|
||||||
|
const tensor = human.tf.tensor(imageData.data); // create tensor from pixel data
|
||||||
|
const result = await human.detect(tensor); // perform processing
|
||||||
|
human.tf.dispose(tensor); // dispose input data, required when working with tensors
|
||||||
|
```
|
||||||
14
Models.md
14
Models.md
|
|
@ -4,17 +4,17 @@
|
||||||
|
|
||||||
Default models in Human library are:
|
Default models in Human library are:
|
||||||
|
|
||||||
- **Face Detection**: MediaPipe BlazeFace - Back variation
|
- **Face Detection**: MediaPipe BlazeFace Back variation
|
||||||
- **Face Mesh**: MediaPipe FaceMesh
|
- **Face Mesh**: MediaPipe FaceMesh
|
||||||
- **Face Iris Analysis**: MediaPipe Iris
|
- **Face Iris Analysis**: MediaPipe Iris
|
||||||
- **Face Description**: HSE FaceRes
|
- **Face Description**: HSE FaceRes
|
||||||
- **Emotion Detection**: Oarriaga Emotion
|
- **Emotion Detection**: Oarriaga Emotion
|
||||||
- **Body Analysis**: MoveNet - Lightning variation
|
- **Body Analysis**: MoveNet Lightning variation
|
||||||
- **Hand Analysis**: MediaPipe Hands
|
- **Hand Analysis**: HandTrack combined with MediaPipe Hands
|
||||||
- **Body Segmentation**: Google Selfie
|
- **Object Detection**: MB3 CenterNet (not enabled by default)
|
||||||
- **Object Detection**: MB3 CenterNet
|
- **Body Segmentation**: Google Selfie (not enabled by default)
|
||||||
- **Body Segmentation**: Google Selfie
|
- **Face Anti-Spoofing**: Real-or-Fake (not enabled by default)
|
||||||
- **Face Anti-Spoofing**: Real-or-Fake
|
- **Face Live Detection**: Liveness (not enabled by default)
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
|
|
|
||||||
177
Usage.md
177
Usage.md
|
|
@ -13,62 +13,101 @@ All configuration is done in a single JSON object and all model weights are dyna
|
||||||
There is only *ONE* method you need:
|
There is only *ONE* method you need:
|
||||||
|
|
||||||
```js
|
```js
|
||||||
// 'image': can be of any type of an image object: HTMLImage, HTMLVideo, HTMLMedia, Canvas, Tensor4D
|
const human = new Human(config?) // create instance of human
|
||||||
// 'config': optional parameter used to override any options present in default configuration
|
const result = await human.detect(input) // run detection
|
||||||
// configuration is fully dynamic and can change between different calls to 'detect()'
|
|
||||||
const result = await human.detect(image, config?)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
or if you want to use promises
|
or if you want to use promises
|
||||||
|
|
||||||
```js
|
```js
|
||||||
human.detect(image, config?).then((result) => {
|
human.detect(input, config?).then((result) => {
|
||||||
// your code
|
// your code
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
## Results Smoothing
|
- [**Valid Inputs**](https://github.com/vladmandic/human/wiki/Inputs)
|
||||||
|
- [**Configuration Details**](https://github.com/vladmandic/human/wiki/Config)
|
||||||
|
|
||||||
If you're processing video input, you may want to interpolate results for smoother output
|
If no other methods are called, `Human` will
|
||||||
After calling `detect` method, simply call `next` method when you need a new interpolated frame
|
1. select best detected engine
|
||||||
|
2. use default configuration
|
||||||
|
3. load required models
|
||||||
|
4. perform warmup operations
|
||||||
|
5. preprocess input
|
||||||
|
6. perform detection
|
||||||
|
|
||||||
`Result` Parameter to `next` method is optional and if not provided, it will use last known result
|
<br>
|
||||||
|
|
||||||
Example that performs single detection and then draws new interpolated result at 50 frames per second:
|
## Results Caching and Smoothing
|
||||||
|
|
||||||
```js
|
- By default, `Human` uses frame change detection for results caching
|
||||||
const result = await human.detect(image, config?)
|
- For on-screen display best results, it is recommended to use results smoothing
|
||||||
setInterval(() => {
|
|
||||||
const interpolated = human.next(result);
|
|
||||||
human.draw.all(canvas, result);
|
|
||||||
}, 1000 / 50);
|
|
||||||
```
|
|
||||||
|
|
||||||
## Extra Properties and Methods
|
For details, see <https://github.com/vladmandic/human/wiki/Caching
|
||||||
|
|
||||||
`Human` library exposes several additional objects and methods:
|
## Demos
|
||||||
|
|
||||||
|
`Human` library comes with number of **browser-based** and **nodejs-based** demo apps in `/demo` folder
|
||||||
|
For details, see <https://github.com/vladmandic/human/wiki/Demos>
|
||||||
|
|
||||||
|
<br>
|
||||||
|
|
||||||
|
## Human Properties
|
||||||
|
|
||||||
|
`Human` library exposes several dynamically generated properties:
|
||||||
|
|
||||||
```js
|
```js
|
||||||
human.version // string containing version of human library
|
human.version // string containing version of human library
|
||||||
human.config // access to configuration object, normally set as parameter to detect()
|
human.config // access to current configuration object
|
||||||
|
// normally set during call to constructor or as parameter to detect()
|
||||||
human.result // access to last known result object, normally returned via call to detect()
|
human.result // access to last known result object, normally returned via call to detect()
|
||||||
human.performance // access to last known performance counters
|
human.performance // access to current performance counters
|
||||||
human.state // <string> describing current operation in progress
|
human.state // <string> describing current operation in progress
|
||||||
// progresses through: 'config', 'check', 'backend', 'load', 'run:<model>', 'idle'
|
// progresses through: 'config', 'check', 'backend', 'load', 'run:<model>', 'idle'
|
||||||
human.sysinfo // object containing current client platform and agent
|
human.models // dynamically maintained list of loaded models
|
||||||
human.load(config) // explicitly call load method that loads configured models
|
human.env // detected platform capabilities
|
||||||
// if you want to pre-load them instead of on-demand loading during 'human.detect()'
|
human.events // container for events dispateched by human
|
||||||
human.image(image, config?) // runs image processing without detection and returns canvas
|
Human.defaults // static property of Human class that contains default configuration
|
||||||
human.warmup(config, image? // warms up human library for faster initial execution after loading
|
|
||||||
// if image is not provided, it will generate internal sample
|
|
||||||
human.models // dynamically maintained list of object of any loaded models
|
|
||||||
human.tf // instance of tfjs used by human
|
|
||||||
human.reset() // reset current configuration to default values
|
|
||||||
human.validate(config?) // validate configuration values
|
|
||||||
// if config is not provided it will validate current values
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Face Recognition
|
<br>
|
||||||
|
|
||||||
|
## Human Methods
|
||||||
|
|
||||||
|
### General
|
||||||
|
|
||||||
|
General purpose methods exposed by `Human`
|
||||||
|
|
||||||
|
```js
|
||||||
|
human.load(config?) // explicitly call load method that loads configured models
|
||||||
|
human.image(input, config?) // runs image processing without detection and returns canvas and tensor
|
||||||
|
human.warmup(config?) // warms up human library for faster initial execution after loading
|
||||||
|
human.next(result?) // returns time variant smoothened/interpolated result based on last known result
|
||||||
|
```
|
||||||
|
|
||||||
|
### Utility
|
||||||
|
|
||||||
|
Utility methods exposed by `Human` that can be used in advanced cases but are typically not needed
|
||||||
|
|
||||||
|
```js
|
||||||
|
human.init() // explict backend initialization
|
||||||
|
human.validate(config?) // validate configuration values
|
||||||
|
human.reset() // reset current configuration to default values
|
||||||
|
human.now() // returns platform-independent timestamp, used for performance measurements
|
||||||
|
human.profile(input, config?) // runs detection with profiling enabled and returns information on top-20 kernels
|
||||||
|
human.compare(input1, input2) // runs pixel-compare on two different inputs and returns score
|
||||||
|
// internally used to detect frame-changes and cache validations
|
||||||
|
```
|
||||||
|
|
||||||
|
### TensorFlow
|
||||||
|
|
||||||
|
`Human` internally uses `TensorFlow/JS` for all ML processing
|
||||||
|
Access to interal instance of `tfjs` used by `human` is possible via:
|
||||||
|
|
||||||
|
```js
|
||||||
|
human.tf // instance of tfjs used by human, can be embedded or externally loaded
|
||||||
|
```
|
||||||
|
### Face Recognition
|
||||||
|
|
||||||
Additional functions used for face recognition:
|
Additional functions used for face recognition:
|
||||||
For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding)
|
For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding)
|
||||||
|
|
@ -78,10 +117,13 @@ For details, see [embedding documentation](https://github.com/vladmandic/human/w
|
||||||
// vectors for source and target must be previously detected using
|
// vectors for source and target must be previously detected using
|
||||||
// face.description module
|
// face.description module
|
||||||
human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces
|
human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces
|
||||||
human.enhance(face) // returns enhanced tensor of a previously detected face that can be used for visualizations
|
human.distance(descriptor1, descriptor2) // checks algorithmic distance between two descriptors
|
||||||
|
// opposite of `similarity`
|
||||||
|
human.enhance(face) // returns enhanced tensor of a previously detected face
|
||||||
|
// that can be used for visualizations
|
||||||
```
|
```
|
||||||
|
|
||||||
## Input Segmentation and Backgroun Removal or Replacement
|
### Input Segmentation and Backgroun Removal or Replacement
|
||||||
|
|
||||||
`Human` library can attempt to detect outlines of people in provided input and either remove background from input
|
`Human` library can attempt to detect outlines of people in provided input and either remove background from input
|
||||||
or replace it with a user-provided background image
|
or replace it with a user-provided background image
|
||||||
|
|
@ -94,7 +136,7 @@ For details on parameters and return values see [API Documentation](https://vlad
|
||||||
human.segmentation(input, background);
|
human.segmentation(input, background);
|
||||||
```
|
```
|
||||||
|
|
||||||
## Draw Functions
|
### Draw Functions
|
||||||
|
|
||||||
Additional helper functions inside `human.draw`:
|
Additional helper functions inside `human.draw`:
|
||||||
|
|
||||||
|
|
@ -125,71 +167,12 @@ Style of drawing is configurable via `human.draw.options` object:
|
||||||
drawLabels: true, // draw labels with detection results
|
drawLabels: true, // draw labels with detection results
|
||||||
drawBoxes: true, // draw boxes around detected faces
|
drawBoxes: true, // draw boxes around detected faces
|
||||||
roundRect: 8, // should boxes have round corners and rounding value
|
roundRect: 8, // should boxes have round corners and rounding value
|
||||||
|
drawGestures: true, // should draw gestures in top-left part of the canvas
|
||||||
|
drawGaze: true, // should draw gaze arrows
|
||||||
drawPolygons: true, // draw polygons such as body and face mesh
|
drawPolygons: true, // draw polygons such as body and face mesh
|
||||||
fillPolygons: false, // fill polygons in face mesh
|
fillPolygons: false, // fill polygons in face mesh
|
||||||
useDepth: true, // use z-axis value when available to determine color shade
|
useDepth: true, // use z-axis value when available to determine color shade
|
||||||
useCurves: false, // draw polygons and boxes using smooth curves instead of lines
|
useCurves: false, // draw polygons and boxes using smooth curves instead of lines
|
||||||
bufferedOutput: true, // experimental: buffer and interpolate results between frames
|
|
||||||
```
|
```
|
||||||
|
|
||||||
<br>
|
<br>
|
||||||
|
|
||||||
## Example Video Processing
|
|
||||||
|
|
||||||
Example simple app that uses Human to process video input and
|
|
||||||
draw output on screen using internal draw helper functions
|
|
||||||
|
|
||||||
```js
|
|
||||||
import Human from '@vladmandic/human';
|
|
||||||
|
|
||||||
// create instance of human with simple configuration using default values
|
|
||||||
const config = { backend: 'webgl' };
|
|
||||||
const human = new Human(config);
|
|
||||||
|
|
||||||
function detectVideo() {
|
|
||||||
// select input HTMLVideoElement and output HTMLCanvasElement from page
|
|
||||||
const inputVideo = document.getElementById('video-id');
|
|
||||||
const outputCanvas = document.getElementById('canvas-id');
|
|
||||||
// perform processing using default configuration
|
|
||||||
human.detect(inputVideo).then((result) => {
|
|
||||||
// result object will contain detected details
|
|
||||||
// as well as the processed canvas itself
|
|
||||||
// so lets first draw processed frame on canvas
|
|
||||||
human.draw.canvas(result.canvas, outputCanvas);
|
|
||||||
// then draw results on the same canvas
|
|
||||||
human.draw.face(outputCanvas, result.face);
|
|
||||||
human.draw.body(outputCanvas, result.body);
|
|
||||||
human.draw.hand(outputCanvas, result.hand);
|
|
||||||
human.draw.gesture(outputCanvas, result.gesture);
|
|
||||||
// loop immediate to next frame
|
|
||||||
requestAnimationFrame(detectVideo);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
detectVideo();
|
|
||||||
```
|
|
||||||
|
|
||||||
<br>
|
|
||||||
|
|
||||||
## Example for NodeJS
|
|
||||||
|
|
||||||
Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards
|
|
||||||
Input format is `Tensor4D[1, width, height, 3]` of type `float32`
|
|
||||||
|
|
||||||
For example:
|
|
||||||
|
|
||||||
```js
|
|
||||||
const imageFile = '../assets/sample1.jpg';
|
|
||||||
const buffer = fs.readFileSync(imageFile);
|
|
||||||
const decoded = tf.node.decodeImage(buffer);
|
|
||||||
const casted = decoded.toFloat();
|
|
||||||
const image = casted.expandDims(0);
|
|
||||||
decoded.dispose();
|
|
||||||
casted.dispose();
|
|
||||||
logger.log('Processing:', image.shape);
|
|
||||||
const human = new Human.Human();
|
|
||||||
const result = await human.detect(image, config);
|
|
||||||
image.dispose();
|
|
||||||
```
|
|
||||||
|
|
||||||
<br><hr>
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue