From 2a937c42e7539b7aa077a9f41085ca573bba7578 Mon Sep 17 00:00:00 2001 From: Vladimir Mandic Date: Wed, 10 Nov 2021 11:53:01 -0500 Subject: [PATCH] update wiki pages --- Demos.md | 353 +++--------------------------------------------------- Inputs.md | 79 ++++++++++++ Models.md | 14 +-- Usage.md | 189 +++++++++++++---------------- 4 files changed, 188 insertions(+), 447 deletions(-) create mode 100644 Inputs.md diff --git a/Demos.md b/Demos.md index b376dd8..de57342 100644 --- a/Demos.md +++ b/Demos.md @@ -1,347 +1,26 @@ # Demos -Demos are included in `/demo`: +All demos are included in `/demo` and come with individual documentation per-demo ## Browser Demos -- `index.html` & `index.js`: [browser] - Main browser demo app that showcases all Human capabilities: - - Optional web workers - - Optional synchronous or asynchronous workflow - - Interpolation & smoothing - - Image processing -- `typescript`: [browser] - Simple demo in WebCam processing demo in TypeScript -- `facematch`: [browser] - Extracts faces from images, calculates face descriptors and simmilarities and matches them to known database -- `facerecognition`: [browser] - Runs multiple checks to validate webcam input before performing face match, similar to *FaceID* -- `multithread`: [browser] - Runs each `human` module in a separate web worker for highest possible performance - See for details -- `face3d`: [browser] - Uses WebCam as input and draws 3D render of face mesh using `Three.js` +- **Full** [[*Live*]](https://vladmandic.github.io/human/demo/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo): Main browser demo app that showcases all Human capabilities +- **Simple** [[*Live*]](https://vladmandic.github.io/human/demo/typescript/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/typescript): Simple demo in WebCam processing demo in TypeScript +- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and simmilarities and matches them to known database +- **Face Recognition** [[*Live*]](https://vladmandic.github.io/human/demo/facerecognition/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facerecognition): Runs multiple checks to validate webcam input before performing face match, similar to *FaceID* +- **Multi-thread** [[*Live*]](https://vladmandic.github.io/human/demo/multithread/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/multithread): Runs each `human` module in a separate web worker for highest possible performance +- **Face 3D** [[*Live*]](https://vladmandic.github.io/human/demo/face3d/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/face3d): Uses WebCam as input and draws 3D render of face mesh using `Three.js` +- **Virtual Avatar** [[*Live*]](https://vladmandic.github.io/human-vrm/src/human-vrm.html) [[*Details*]](https://github.com/vladmandic/human-vrm): VR model with head, face, eye, body and hand tracking ## NodeJS Demos -- `nodejs/node`: [nodejs] - Process images from files, folders or URLs -- `nodejs/node-canvas`: [nodejs] - Process image from file or URL and draw results to a new image file using `node-canvas` -- `nodejs/node-multiprocess` & `nodejs/node-multiprocess-worker`: [nodejs] - Parallel processing in `human` **detect** in multiple child worker processes -- `facematch/node-match` & `facematch/node-match-worker`: [nodejs] - Parallel processing of face **match** in multiple child worker threads -- `nodejs/node-video`: [nodejs] - Processing of video input using `ffmpeg` -- `nodejs/node-webcam`: [nodejs] - Processing of webcam screenshots using `fswebcam` - -## External Demos - -- `Human-VRM` - VR model with head, face, eye, body and hand tracking - Using [`three`](https://github.com/mrdoob/three.js) for 3D display and scene management and [`@pixiv/three-vrm`](https://github.com/pixiv/three-vrm) for VR model mapping - [Code](https://github.com/vladmandic/human-vrm) | [Demo](https://vladmandic.github.io/human-vrm/src/human-vrm.html) - +- **Main** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process images from files, folders or URLs using native methods +- **Canvas** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Process image from file or URL and draw results to a new image file using `node-canvas` +- **Video** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of video input using `ffmpeg` +- **WebCam** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Processing of webcam screenshots using `fswebcam` +- **Events** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Showcases usage of `Human` eventing to get notifications on processing +- **Similarity** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Compares two input images for similarity of detected faces +- **Face Match** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Parallel processing of face **match** in multiple child worker threads +- **Multiple Workers** [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/nodejs): Runs multiple parallel `human` by dispaching them to pool of pre-created worker processes

- -## Main Demo - -- `index.html`: Full demo using `Human` ESM module running in Browsers, - includes selectable backends and WebWorkers - -*You can run browser demo either live from git pages, by serving demo folder from your web server or use -included micro http2 server with source file monitoring and dynamic rebuild* - -On notes on how to use built-in micro server, see notes on [**Development Server**](https://github.com/vladmandic/human/wiki/Development-Server) - -
- -### Demo Inputs - -Demo is in `demo/index.html` loads `demo/index.js` - -Demo can process: - -- Sample images -- WebCam input -- WebRTC input - -Note that WebRTC connection requires a WebRTC server that provides a compatible media track such as H.264 video track -For such a WebRTC server implementation see project -that implements a connection to IP Security camera using RTSP protocol and transcodes it to WebRTC -ready to be consumed by a client such as `Human` - -
- -### Demo Options - -Demo implements several ways to use `Human` library, -all configurable in `browse.js:ui` configuration object and in the UI itself: - -```js -const ui = { - crop: true, // video mode crop to size or leave full frame - columns: 2, // when processing sample images create this many columns - facing: true, // camera facing front or back - useWorker: false, // use web workers for processing - worker: 'index-worker.js', - samples: ['../assets/sample6.jpg', '../assets/sample1.jpg', '../assets/sample4.jpg', '../assets/sample5.jpg', '../assets/sample3.jpg', '../assets/sample2.jpg'], - compare: '../assets/sample-me.jpg', - useWebRTC: false, // use webrtc as camera source instead of local webcam - webRTCServer: 'http://localhost:8002', - webRTCStream: 'reowhite', - console: true, // log messages to browser console - maxFPSframes: 10, // keep fps history for how many frames - modelsPreload: true, // preload human models on startup - modelsWarmup: true, // warmup human models on startup - busy: false, // internal camera busy flag - buffered: true, // should output be buffered between frames - bench: true, // show gl fps benchmark window -}; -``` - -Additionally, some parameters are held inside `Human` instance: - -```ts -human.draw.options = { - color: 'rgba(173, 216, 230, 0.3)', // 'lightblue' with light alpha channel - labelColor: 'rgba(173, 216, 230, 1)', // 'lightblue' with dark alpha channel - shadowColor: 'black', - font: 'small-caps 16px "Segoe UI"', - lineHeight: 20, - lineWidth: 6, - pointSize: 2, - roundRect: 28, - drawPoints: false, - drawLabels: true, - drawBoxes: true, - drawPolygons: true, - fillPolygons: false, - useDepth: true, - useCurves: false, - bufferedOutput: true, - useRawBoxes: false, -}; -``` - -Demo app can use URL parameters to override configuration values -For example: - -- Force using `WASM` as backend: -- Enable `WebWorkers`: -- Skip pre-loading and warming up: - -

- -## Face 3D Rendering using OpenGL - -- `face3d`: Demo for Browsers that uses `Three.js` for 3D OpenGL rendering of a detected face - -

- -## Face Recognition Demo - -- `demo/facematch`: Demo for Browsers that uses all face description and embedding features to -detect, extract and identify all faces plus calculate simmilarity between them - -It highlights functionality such as: - -- Loading images -- Extracting faces from images -- Calculating face embedding descriptors -- Finding face similarity and sorting them by similarity -- Finding best face match based on a known list of faces and printing matches - -

- -## Multithreaded Demo - -- `demo/multithread`: Demo for Browsers that uses multiple worker processes - -- Showcases maximum performance sinch each `human` model runs in a separate worker thread -while main thread just combines and draws results -- Achieves 20+ detection frame rate and 60+ refresh frame rate on a medium hardware - -

- -## NodeJS Demo - -- `nodejs/node.js`: Demo using NodeJS with CommonJS module - Simple demo that can process any input image - -Note that you can run demo as-is and it will perform detection on provided sample images, -or you can pass a path to image to analyze, either on local filesystem or using URL - -```shell -node demo/nodejs/node.js -``` - -```json -2021-06-01 08:52:15 INFO: @vladmandic/human version 2.0.0 -2021-06-01 08:52:15 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0 -2021-06-01 08:52:15 INFO: Current folder: /home/vlado/dev/human -2021-06-01 08:52:15 INFO: Human: 2.0.0 -2021-06-01 08:52:15 INFO: Active Configuration { - backend: 'tensorflow', - modelBasePath: 'file://models/', - wasmPath: '../node_modules/@tensorflow/tfjs-backend-wasm/dist/', - debug: true, - async: false, - warmup: 'full', - cacheSensitivity: 0.75, - filter: { - enabled: true, - width: 0, - height: 0, - flip: true, - return: true, - brightness: 0, - contrast: 0, - sharpness: 0, - blur: 0, - saturation: 0, - hue: 0, - negative: false, - sepia: false, - vintage: false, - kodachrome: false, - technicolor: false, - polaroid: false, - pixelate: 0 - }, - gesture: { enabled: true }, - face: { - enabled: true, - detector: { modelPath: 'blazeface.json', rotation: false, maxDetected: 10, skipFrames: 15, minConfidence: 0.2, iouThreshold: 0.1, return: false, enabled: true }, - mesh: { enabled: true, modelPath: 'facemesh.json' }, - iris: { enabled: true, modelPath: 'iris.json' }, - description: { enabled: true, modelPath: 'faceres.json', skipFrames: 16, minConfidence: 0.1 }, - emotion: { enabled: true, minConfidence: 0.1, skipFrames: 17, modelPath: 'emotion.json' } - }, - body: { enabled: true, modelPath: 'movenet-lightning.json', maxDetected: 1, minConfidence: 0.2 }, - hand: { - enabled: true, - rotation: true, - skipFrames: 18, - minConfidence: 0.1, - iouThreshold: 0.1, - maxDetected: 2, - landmarks: true, - detector: { modelPath: 'handdetect.json' }, - skeleton: { modelPath: 'handskeleton.json' } - }, - object: { enabled: true, modelPath: 'mb3-centernet.json', minConfidence: 0.2, iouThreshold: 0.4, maxDetected: 10, skipFrames: 19 } -} -08:52:15.673 Human: version: 2.0.0 -08:52:15.674 Human: tfjs version: 3.6.0 -08:52:15.674 Human: platform: linux x64 -08:52:15.674 Human: agent: NodeJS v16.0.0 -08:52:15.674 Human: setting backend: tensorflow -08:52:15.710 Human: load model: file://models/blazeface.json -08:52:15.743 Human: load model: file://models/facemesh.json -08:52:15.744 Human: load model: file://models/iris.json -08:52:15.760 Human: load model: file://models/emotion.json -08:52:15.847 Human: load model: file://models/handdetect.json -08:52:15.847 Human: load model: file://models/handskeleton.json -08:52:15.914 Human: load model: file://models/movenet-lightning.json -08:52:15.957 Human: load model: file://models/mb3-centernet.json -08:52:16.015 Human: load model: file://models/faceres.json -08:52:16.015 Human: tf engine state: 50796152 bytes 1318 tensors -2021-06-01 08:52:16 INFO: Loaded: [ 'face', 'movenet', 'handpose', 'emotion', 'centernet', 'faceres', [length]: 6 ] -2021-06-01 08:52:16 INFO: Memory state: { unreliable: true, numTensors: 1318, numDataBuffers: 1318, numBytes: 50796152 } -2021-06-01 08:52:16 INFO: Loading image: private/daz3d/daz3d-kiaria-02.jpg -2021-06-01 08:52:16 STATE: Processing: [ 1, 1300, 1000, 3, [length]: 4 ] -2021-06-01 08:52:17 DATA: Results: -2021-06-01 08:52:17 DATA: Face: #0 boxScore:0.88 faceScore:1 age:16.3 genderScore:0.97 gender:female emotionScore:0.85 emotion:happy iris:61.05 -2021-06-01 08:52:17 DATA: Body: #0 score:0.82 keypoints:17 -2021-06-01 08:52:17 DATA: Hand: #0 score:0.89 -2021-06-01 08:52:17 DATA: Hand: #1 score:0.97 -2021-06-01 08:52:17 DATA: Gesture: face#0 gesture:facing left -2021-06-01 08:52:17 DATA: Gesture: body#0 gesture:leaning right -2021-06-01 08:52:17 DATA: Gesture: hand#0 gesture:pinky forward middlefinger up -2021-06-01 08:52:17 DATA: Gesture: hand#1 gesture:pinky forward middlefinger up -2021-06-01 08:52:17 DATA: Gesture: iris#0 gesture:looking left -2021-06-01 08:52:17 DATA: Object: #0 score:0.55 label:person -2021-06-01 08:52:17 DATA: Object: #1 score:0.23 label:bottle -2021-06-01 08:52:17 DATA: Persons: -2021-06-01 08:52:17 DATA: #0: Face:score:1 age:16.3 gender:female iris:61.05 Body:score:0.82 keypoints:17 LeftHand:no RightHand:yes Gestures:4 -``` - -

- -## NodeJS Multi-process Demo - -- `nodejs/node-multiprocess.js` and `nodejs/node-multiprocess-worker.js`: Demo using NodeJS with CommonJS module - Demo that starts n child worker processes for parallel execution - -```shell -node demo/nodejs/node-multiprocess.js -``` - -```json -2021-06-01 08:54:19 INFO: @vladmandic/human version 2.0.0 -2021-06-01 08:54:19 INFO: User: vlado Platform: linux Arch: x64 Node: v16.0.0 -2021-06-01 08:54:19 INFO: FaceAPI multi-process test -2021-06-01 08:54:19 STATE: Enumerated images: ./assets 15 -2021-06-01 08:54:19 STATE: Main: started worker: 130362 -2021-06-01 08:54:19 STATE: Main: started worker: 130363 -2021-06-01 08:54:19 STATE: Main: started worker: 130369 -2021-06-01 08:54:19 STATE: Main: started worker: 130370 -2021-06-01 08:54:20 STATE: Worker: PID: 130370 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow -2021-06-01 08:54:20 STATE: Worker: PID: 130362 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow -2021-06-01 08:54:20 STATE: Worker: PID: 130369 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow -2021-06-01 08:54:20 STATE: Worker: PID: 130363 TensorFlow/JS 3.6.0 Human 2.0.0 Backend: tensorflow -2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130370 -2021-06-01 08:54:21 INFO: Latency: worker initializtion: 1348 message round trip: 0 -2021-06-01 08:54:21 DATA: Worker received message: 130370 { test: true } -2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130362 -2021-06-01 08:54:21 DATA: Worker received message: 130362 { image: 'samples/ai-face.jpg' } -2021-06-01 08:54:21 DATA: Worker received message: 130370 { image: 'samples/ai-body.jpg' } -2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130369 -2021-06-01 08:54:21 STATE: Main: dispatching to worker: 130363 -2021-06-01 08:54:21 DATA: Worker received message: 130369 { image: 'assets/human-sample-upper.jpg' } -2021-06-01 08:54:21 DATA: Worker received message: 130363 { image: 'assets/sample-me.jpg' } -2021-06-01 08:54:24 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1 -2021-06-01 08:54:24 STATE: Main: dispatching to worker: 130362 -2021-06-01 08:54:24 DATA: Worker received message: 130362 { image: 'assets/sample1.jpg' } -2021-06-01 08:54:25 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 0 objects: 1 -2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130369 -2021-06-01 08:54:25 DATA: Main: worker finished: 130370 detected faces: 1 bodies: 1 hands: 0 objects: 1 -2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130370 -2021-06-01 08:54:25 DATA: Worker received message: 130369 { image: 'assets/sample2.jpg' } -2021-06-01 08:54:25 DATA: Main: worker finished: 130363 detected faces: 1 bodies: 1 hands: 0 objects: 2 -2021-06-01 08:54:25 STATE: Main: dispatching to worker: 130363 -2021-06-01 08:54:25 DATA: Worker received message: 130370 { image: 'assets/sample3.jpg' } -2021-06-01 08:54:25 DATA: Worker received message: 130363 { image: 'assets/sample4.jpg' } -2021-06-01 08:54:30 DATA: Main: worker finished: 130362 detected faces: 3 bodies: 1 hands: 0 objects: 7 -2021-06-01 08:54:30 STATE: Main: dispatching to worker: 130362 -2021-06-01 08:54:30 DATA: Worker received message: 130362 { image: 'assets/sample5.jpg' } -2021-06-01 08:54:31 DATA: Main: worker finished: 130369 detected faces: 3 bodies: 1 hands: 0 objects: 5 -2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130369 -2021-06-01 08:54:31 DATA: Worker received message: 130369 { image: 'assets/sample6.jpg' } -2021-06-01 08:54:31 DATA: Main: worker finished: 130363 detected faces: 4 bodies: 1 hands: 2 objects: 2 -2021-06-01 08:54:31 STATE: Main: dispatching to worker: 130363 -2021-06-01 08:54:39 STATE: Main: worker exit: 130370 0 -2021-06-01 08:54:39 DATA: Main: worker finished: 130362 detected faces: 1 bodies: 1 hands: 0 objects: 1 -2021-06-01 08:54:39 DATA: Main: worker finished: 130369 detected faces: 1 bodies: 1 hands: 1 objects: 3 -2021-06-01 08:54:39 STATE: Main: worker exit: 130362 0 -2021-06-01 08:54:39 STATE: Main: worker exit: 130369 0 -2021-06-01 08:54:41 DATA: Main: worker finished: 130363 detected faces: 9 bodies: 1 hands: 0 objects: 10 -2021-06-01 08:54:41 STATE: Main: worker exit: 130363 0 -2021-06-01 08:54:41 INFO: Processed: 15 images in total: 22006 ms working: 20658 ms average: 1377 ms -``` - -

- -## NodeJS Demo for Video Input Processing - -- `nodejs/node-video.js`: Demo that uses `ffmpeg` to decode video input (can be a file, stream or device such as webcam) and output results in a pipe that are captured by demo app as frames and processed by Human library - -

- -## NodeJS Demo for WebCam Screenshot Processing - -- `nodejs/node-webcam.js`: Demo that uses `fswebcam` to connect to web cam and take screenshots at regular interval which are then processed by Human library diff --git a/Inputs.md b/Inputs.md new file mode 100644 index 0000000..fe7001c --- /dev/null +++ b/Inputs.md @@ -0,0 +1,79 @@ +# Inputs + +- In **Browser** environments, actual browser provides functionality of decoding inputs (e.g. JPG image or MP4 video) +- In **NodeJS** environments, decoding must be performed manually + +
+ +## Valid Inputs in Browser Environments + +`Human` allows input to be in many different formats and will perform automatic processing of inputs to interally required format + +```ts +type Input = Tensor | AnyCanvas | AnyImage | AnyVideo | ImageObjects | ExternalCanvas; +type AnyCanvas = HTMLCanvasElement | OffscreenCanvas; +type AnyImage = HTMLImageElement | typeof Image +type AnyVideo = HTMLMediaElement | HTMLVideoElement +type ImageObjects = ImageData | ImageBitmap +type ExternalCanvas = typeof env.Canvas | typeof globalThis.Canvas; +``` + +
+ +## Examples of Input processing in NodeJS + +### 1. Using decode functionality from `tfjs-node`: + +All primary functionality of `Human` is available, but `human.draw` methods cannot be used as `canvas` implementation is not present + +```js +const tf = require('@tensorflow/tfjs-node'); +const buffer = fs.readFileSync(inputFile); // read file content into a binary buffer +const tensor = human.tf.node.decodeImage(buffer); // decode jpg/png data to raw pixels +const result = await human.detect(tensor); // perform processing +human.tf.dispose(tensor); // dispose input data, required when working with tensors +``` + +*Note:* For all processing, correct input tensor **shape** `[1, height, width, 3]` and **dtype** `float32` +- 1 means batch number and is a fixed value +- 3 means number of channels so 3 is used for RGB format + +However `Human` will automatically convert input tensor to a correct shape +- if batch number is omitted +- if input image is 4-channels such as in **RGBA** images with alpha channel +- if input tensor is in different data type such as `int32` + + +### 2. Using Canvas functionality from `node-canvas` + +By instructing `Human` to use 3rd party module for `canvas` operations +This method allows `Human` to use `human.draw.*` methods in **NodeJS** + +```js +const canvas = require('canvas'); +globalThis.Canvas = canvas.Canvas; // patch global namespace with canvas library +globalThis.ImageData = canvas.ImageData; // patch global namespace with canvas library +// human.env.Canvas = canvas.Canvas; // alternatively monkey-patch human to use external canvas library +// human.env.ImageData = canvas.ImageData; // alternatively monkey-patch human to use external canvas library +const inputImage = await canvas.loadImage(inputFile); // load image using canvas library +const inputCanvas = new canvas.Canvas(inputImage.width, inputImage.height); // create canvas +const ctx = inputCanvas.getContext('2d'); +ctx.drawImage(inputImage, 0, 0); // draw input image onto canvas +const result = await human.detect(inputCanvas); +``` + +### 3. Using Canvas functionality from `node-canvas` + +Using `node-canvas` to load and decode input files only + +```js +const canvas = require('canvas'); +const img = await canvas.loadImage(inputFile); // read and decode image file +const canvas = canvas.createCanvas(img.width, img.height); // create canvas element +const ctx = canvas.getContext('2d'); +ctx.drawImage(img, 0, 0, img.width, img.height); // draw loaded image onto canvas +const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height); // read pixel data from canvas +const tensor = human.tf.tensor(imageData.data); // create tensor from pixel data +const result = await human.detect(tensor); // perform processing +human.tf.dispose(tensor); // dispose input data, required when working with tensors +``` diff --git a/Models.md b/Models.md index d10ed0b..32712b5 100644 --- a/Models.md +++ b/Models.md @@ -4,17 +4,17 @@ Default models in Human library are: -- **Face Detection**: MediaPipe BlazeFace - Back variation +- **Face Detection**: MediaPipe BlazeFace Back variation - **Face Mesh**: MediaPipe FaceMesh - **Face Iris Analysis**: MediaPipe Iris - **Face Description**: HSE FaceRes - **Emotion Detection**: Oarriaga Emotion -- **Body Analysis**: MoveNet - Lightning variation -- **Hand Analysis**: MediaPipe Hands -- **Body Segmentation**: Google Selfie -- **Object Detection**: MB3 CenterNet -- **Body Segmentation**: Google Selfie -- **Face Anti-Spoofing**: Real-or-Fake +- **Body Analysis**: MoveNet Lightning variation +- **Hand Analysis**: HandTrack combined with MediaPipe Hands +- **Object Detection**: MB3 CenterNet (not enabled by default) +- **Body Segmentation**: Google Selfie (not enabled by default) +- **Face Anti-Spoofing**: Real-or-Fake (not enabled by default) +- **Face Live Detection**: Liveness (not enabled by default)
diff --git a/Usage.md b/Usage.md index 103ebd7..f4484f9 100644 --- a/Usage.md +++ b/Usage.md @@ -13,75 +13,117 @@ All configuration is done in a single JSON object and all model weights are dyna There is only *ONE* method you need: ```js - // 'image': can be of any type of an image object: HTMLImage, HTMLVideo, HTMLMedia, Canvas, Tensor4D - // 'config': optional parameter used to override any options present in default configuration - // configuration is fully dynamic and can change between different calls to 'detect()' - const result = await human.detect(image, config?) + const human = new Human(config?) // create instance of human + const result = await human.detect(input) // run detection ``` or if you want to use promises ```js - human.detect(image, config?).then((result) => { + human.detect(input, config?).then((result) => { // your code }) ``` -## Results Smoothing +- [**Valid Inputs**](https://github.com/vladmandic/human/wiki/Inputs) +- [**Configuration Details**](https://github.com/vladmandic/human/wiki/Config) -If you're processing video input, you may want to interpolate results for smoother output -After calling `detect` method, simply call `next` method when you need a new interpolated frame +If no other methods are called, `Human` will +1. select best detected engine +2. use default configuration +3. load required models +4. perform warmup operations +5. preprocess input +6. perform detection -`Result` Parameter to `next` method is optional and if not provided, it will use last known result +
-Example that performs single detection and then draws new interpolated result at 50 frames per second: +## Results Caching and Smoothing + +- By default, `Human` uses frame change detection for results caching +- For on-screen display best results, it is recommended to use results smoothing + +For details, see + +
+ +## Human Properties + +`Human` library exposes several dynamically generated properties: ```js - const result = await human.detect(image, config?) - setInterval(() => { - const interpolated = human.next(result); - human.draw.all(canvas, result); - }, 1000 / 50); +human.version // string containing version of human library +human.config // access to current configuration object + // normally set during call to constructor or as parameter to detect() +human.result // access to last known result object, normally returned via call to detect() +human.performance // access to current performance counters +human.state // describing current operation in progress + // progresses through: 'config', 'check', 'backend', 'load', 'run:', 'idle' +human.models // dynamically maintained list of loaded models +human.env // detected platform capabilities +human.events // container for events dispateched by human +Human.defaults // static property of Human class that contains default configuration ``` -## Extra Properties and Methods +
-`Human` library exposes several additional objects and methods: +## Human Methods + +### General + +General purpose methods exposed by `Human` ```js - human.version // string containing version of human library - human.config // access to configuration object, normally set as parameter to detect() - human.result // access to last known result object, normally returned via call to detect() - human.performance // access to last known performance counters - human.state // describing current operation in progress - // progresses through: 'config', 'check', 'backend', 'load', 'run:', 'idle' - human.sysinfo // object containing current client platform and agent - human.load(config) // explicitly call load method that loads configured models - // if you want to pre-load them instead of on-demand loading during 'human.detect()' - human.image(image, config?) // runs image processing without detection and returns canvas - human.warmup(config, image? // warms up human library for faster initial execution after loading - // if image is not provided, it will generate internal sample - human.models // dynamically maintained list of object of any loaded models - human.tf // instance of tfjs used by human - human.reset() // reset current configuration to default values - human.validate(config?) // validate configuration values - // if config is not provided it will validate current values +human.load(config?) // explicitly call load method that loads configured models +human.image(input, config?) // runs image processing without detection and returns canvas and tensor +human.warmup(config?) // warms up human library for faster initial execution after loading +human.next(result?) // returns time variant smoothened/interpolated result based on last known result ``` -## Face Recognition +### Utility + +Utility methods exposed by `Human` that can be used in advanced cases but are typically not needed + +```js +human.init() // explict backend initialization +human.validate(config?) // validate configuration values +human.reset() // reset current configuration to default values +human.now() // returns platform-independent timestamp, used for performance measurements +human.profile(input, config?) // runs detection with profiling enabled and returns information on top-20 kernels +human.compare(input1, input2) // runs pixel-compare on two different inputs and returns score + // internally used to detect frame-changes and cache validations +``` + +### TensorFlow + +`Human` internally uses `TensorFlow/JS` for all ML processing +Access to interal instance of `tfjs` used by `human` is possible via: + +```js +human.tf // instance of tfjs used by human, can be embedded or externally loaded +``` +### Face Recognition Additional functions used for face recognition: For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding) ```js - human.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors - // vectors for source and target must be previously detected using - // face.description module - human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces - human.enhance(face) // returns enhanced tensor of a previously detected face that can be used for visualizations +human.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors + // vectors for source and target must be previously detected using + // face.description module +human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces +human.distance(descriptor1, descriptor2) // checks algorithmic distance between two descriptors + // opposite of `similarity` +human.enhance(face) // returns enhanced tensor of a previously detected face + // that can be used for visualizations ``` -## Input Segmentation and Backgroun Removal or Replacement +### Input Segmentation and Backgroun Removal or Replacement `Human` library can attempt to detect outlines of people in provided input and either remove background from input or replace it with a user-provided background image @@ -94,7 +136,7 @@ For details on parameters and return values see [API Documentation](https://vlad human.segmentation(input, background); ``` -## Draw Functions +### Draw Functions Additional helper functions inside `human.draw`: @@ -125,71 +167,12 @@ Style of drawing is configurable via `human.draw.options` object: drawLabels: true, // draw labels with detection results drawBoxes: true, // draw boxes around detected faces roundRect: 8, // should boxes have round corners and rounding value + drawGestures: true, // should draw gestures in top-left part of the canvas + drawGaze: true, // should draw gaze arrows drawPolygons: true, // draw polygons such as body and face mesh fillPolygons: false, // fill polygons in face mesh useDepth: true, // use z-axis value when available to determine color shade useCurves: false, // draw polygons and boxes using smooth curves instead of lines - bufferedOutput: true, // experimental: buffer and interpolate results between frames ```
- -## Example Video Processing - -Example simple app that uses Human to process video input and -draw output on screen using internal draw helper functions - -```js -import Human from '@vladmandic/human'; - -// create instance of human with simple configuration using default values -const config = { backend: 'webgl' }; -const human = new Human(config); - -function detectVideo() { - // select input HTMLVideoElement and output HTMLCanvasElement from page - const inputVideo = document.getElementById('video-id'); - const outputCanvas = document.getElementById('canvas-id'); - // perform processing using default configuration - human.detect(inputVideo).then((result) => { - // result object will contain detected details - // as well as the processed canvas itself - // so lets first draw processed frame on canvas - human.draw.canvas(result.canvas, outputCanvas); - // then draw results on the same canvas - human.draw.face(outputCanvas, result.face); - human.draw.body(outputCanvas, result.body); - human.draw.hand(outputCanvas, result.hand); - human.draw.gesture(outputCanvas, result.gesture); - // loop immediate to next frame - requestAnimationFrame(detectVideo); - }); -} - -detectVideo(); -``` - -
- -## Example for NodeJS - -Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards -Input format is `Tensor4D[1, width, height, 3]` of type `float32` - -For example: - -```js - const imageFile = '../assets/sample1.jpg'; - const buffer = fs.readFileSync(imageFile); - const decoded = tf.node.decodeImage(buffer); - const casted = decoded.toFloat(); - const image = casted.expandDims(0); - decoded.dispose(); - casted.dispose(); - logger.log('Processing:', image.shape); - const human = new Human.Human(); - const result = await human.detect(image, config); - image.dispose(); -``` - -