mirror of https://github.com/vladmandic/human
update wiki
Vladimir Mandic
parent
93e58e16b5
commit
6ea5ea911d
2
Demos.md
2
Demos.md
|
|
@ -9,7 +9,7 @@ All demos are included in `/demo` and come with individual documentation per-dem
|
|||
- **Full** [[*Live*]](https://vladmandic.github.io/human/demo/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo): Main browser demo app that showcases all Human capabilities
|
||||
- **Simple** [[*Live*]](https://vladmandic.github.io/human/demo/typescript/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/typescript): Simple demo in WebCam processing demo in TypeScript
|
||||
- **Embedded** [[*Live*]](https://vladmandic.github.io/human/demo/video/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/video/index.html): Even simpler demo with tiny code embedded in HTML file
|
||||
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and simmilarities and matches them to known database
|
||||
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and similarities and matches them to known database
|
||||
- **Face ID** [[*Live*]](https://vladmandic.github.io/human/demo/faceid/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/faceid): Runs multiple checks to validate webcam input before performing face match to faces in IndexDB
|
||||
- **Multi-thread** [[*Live*]](https://vladmandic.github.io/human/demo/multithread/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/multithread): Runs each Human module in a separate web worker for highest possible performance
|
||||
- **NextJS** [[*Live*]](https://vladmandic.github.io/human-next/out/index.html) [[*Details*]](https://github.com/vladmandic/human-next): Use Human with TypeScript, NextJS and ReactJS
|
||||
|
|
|
|||
23
Embedding.md
23
Embedding.md
|
|
@ -38,8 +38,8 @@ const myConfig = {
|
|||
const human = new Human(myConfig);
|
||||
const firstResult = await human.detect(firstImage);
|
||||
const secondResult = await human.detect(secondImage);
|
||||
const similarity = human.similarity(firstResult.face[0].embedding, secondResult.face[0].embedding);
|
||||
console.log(`faces are ${100 * similarity}% simmilar`);
|
||||
const similarity = human.match.similarity(firstResult.face[0].embedding, secondResult.face[0].embedding);
|
||||
console.log(`faces are ${100 * similarity}% similar`);
|
||||
```
|
||||
|
||||
If the image or video frame have multiple faces and you want to match all of them, simply loop through all `results.face`
|
||||
|
|
@ -47,8 +47,8 @@ If the image or video frame have multiple faces and you want to match all of the
|
|||
```js
|
||||
for (let i = 0; i < currentResult.face.length; i++) {
|
||||
const currentEmbedding = currentResult.face[i].embedding;
|
||||
const similarity = human.similarity(referenceEmbedding, currentEmbedding);
|
||||
console.log(`face ${i} is ${100 * similarity}% simmilar`);
|
||||
const similarity = human.match.similarity(referenceEmbedding, currentEmbedding);
|
||||
console.log(`face ${i} is ${100 * similarity}% similar`);
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -61,15 +61,6 @@ const myConfig = {
|
|||
};
|
||||
```
|
||||
|
||||
Additional helper function is `human.enhance(face)` which returns an enhanced tensor
|
||||
of a face image that can be further visualized with
|
||||
|
||||
```js
|
||||
const enhanced = human.enhance(face);
|
||||
const canvas = document.getElementById('orig');
|
||||
human.tf.browser.toPixels(enhanced.squeeze(), canvas);
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
## Face Descriptor
|
||||
|
|
@ -91,8 +82,8 @@ Changing `order` can make similarity matching more or less sensitive (default or
|
|||
For example, those will produce slighly different results:
|
||||
|
||||
```js
|
||||
const similarity2ndOrder = human.similarity(firstEmbedding, secondEmbedding, { order = 2 });
|
||||
const similarity3rdOrder = human.similarity(firstEmbedding, secondEmbedding, { order = 3 });
|
||||
const similarity2ndOrder = human.match.similarity(firstEmbedding, secondEmbedding, { order = 2 });
|
||||
const similarity3rdOrder = human.match.similarity(firstEmbedding, secondEmbedding, { order = 3 });
|
||||
```
|
||||
|
||||
<br>
|
||||
|
|
@ -116,7 +107,7 @@ To find the best match, simply use `match` method while providing embedding desc
|
|||
|
||||
```js
|
||||
const embeddingArray = db.map((record) => record.embedding); // build array with just embeddings
|
||||
const best = human.match(embedding, embeddingArray); // return is object: { index: number, similarity: number, distance: number }
|
||||
const best = human.match.find(embedding, embeddingArray); // return is object: { index: number, similarity: number, distance: number }
|
||||
const label = embeddingArray[best.index].label;
|
||||
console.log({ name, similarity: best.similarity });
|
||||
```
|
||||
|
|
|
|||
2
Home.md
2
Home.md
|
|
@ -68,7 +68,7 @@
|
|||
- **Full** [[*Live*]](https://vladmandic.github.io/human/demo/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo): Main browser demo app that showcases all Human capabilities
|
||||
- **Simple** [[*Live*]](https://vladmandic.github.io/human/demo/typescript/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/typescript): Simple demo in WebCam processing demo in TypeScript
|
||||
- **Embedded** [[*Live*]](https://vladmandic.github.io/human/demo/video/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/video/index.html): Even simpler demo with tiny code embedded in HTML file
|
||||
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and simmilarities and matches them to known database
|
||||
- **Face Match** [[*Live*]](https://vladmandic.github.io/human/demo/facematch/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/facematch): Extract faces from images, calculates face descriptors and similarities and matches them to known database
|
||||
- **Face ID** [[*Live*]](https://vladmandic.github.io/human/demo/faceid/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/faceid): Runs multiple checks to validate webcam input before performing face match to faces in IndexDB
|
||||
- **Multi-thread** [[*Live*]](https://vladmandic.github.io/human/demo/multithread/index.html) [[*Details*]](https://github.com/vladmandic/human/tree/main/demo/multithread): Runs each Human module in a separate web worker for highest possible performance
|
||||
- **NextJS** [[*Live*]](https://vladmandic.github.io/human-next/out/index.html) [[*Details*]](https://github.com/vladmandic/human-next): Use Human with TypeScript, NextJS and ReactJS
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@ Switching model also automatically switches implementation used inside `Human` s
|
|||
- `Age Detection`: SSR-Net Age IMDB
|
||||
- `Face Embedding`: BecauseofAI MobileFace Embedding
|
||||
|
||||
**Object detection** can be switched from `mb3-centernet` to `nanodet`
|
||||
**Object detection** can be switched from `centernet` to `nanodet`
|
||||
|
||||
**Hand destection** can be switched from `handdetect` to `handtrack`
|
||||
|
||||
|
|
|
|||
219
Usage.md
219
Usage.md
|
|
@ -8,49 +8,108 @@ All configuration is done in a single JSON object and all model weights are dyna
|
|||
|
||||
<br>
|
||||
|
||||
## Detect
|
||||
## Basics
|
||||
|
||||
There is only *ONE* method you need:
|
||||
|
||||
```js
|
||||
const human = new Human(config?) // create instance of human
|
||||
const result = await human.detect(input) // run detection
|
||||
const human = new Human(config?) // create instance of human
|
||||
const result = await human.detect(input, config?) // run single detection
|
||||
```
|
||||
|
||||
or if you want to use promises
|
||||
or
|
||||
|
||||
```js
|
||||
human.detect(input, config?).then((result) => {
|
||||
// your code
|
||||
})
|
||||
const human = new Human(config?) // create instance of human
|
||||
await human.video(input, config?) // run detection loop on input video
|
||||
// last known results are available in human.results
|
||||
```
|
||||
|
||||
Notes:
|
||||
- [**Valid Inputs**](https://github.com/vladmandic/human/wiki/Inputs)
|
||||
- [**Configuration Details**](https://github.com/vladmandic/human/wiki/Config)
|
||||
|
||||
If no other methods are called, `Human` will
|
||||
1. select best detected engine
|
||||
2. use default configuration
|
||||
3. load required models
|
||||
4. perform warmup operations
|
||||
5. preprocess input
|
||||
6. perform detection
|
||||
Standard workflow:
|
||||
|
||||
1. Select best detected engine
|
||||
2. Use default configuration
|
||||
3. Load & compile required models
|
||||
4. Perform warmup operations
|
||||
5. Preprocess input
|
||||
6. Perform detection
|
||||
|
||||
|
||||
<br>
|
||||
|
||||
## Results Caching and Smoothing
|
||||
## Human Methods
|
||||
|
||||
- By default, `Human` uses frame change detection for results caching
|
||||
- For on-screen display best results, it is recommended to use results smoothing
|
||||
### Main
|
||||
|
||||
For details, see <https://github.com/vladmandic/human/wiki/Caching
|
||||
Methods exported by `Human` library:
|
||||
|
||||
## Demos
|
||||
```js
|
||||
human.detect(input, config?); // run processing on single image/frame
|
||||
human.video(input, config?); // run processing continously on video
|
||||
human.next(result?); // returns time-variant smoothened/interpolated result based on last known result
|
||||
```
|
||||
|
||||
`Human` library comes with number of **browser-based** and **nodejs-based** demo apps in `/demo` folder
|
||||
For details, see <https://github.com/vladmandic/human/wiki/Demos>
|
||||
### Additional
|
||||
|
||||
<br>
|
||||
Methods used for **face recognition** and **face matching**:
|
||||
For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding)
|
||||
|
||||
```js
|
||||
human.match.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors
|
||||
// vectors for source and target must be previously detected using
|
||||
// face.description module
|
||||
human.match.find(descriptor, descriptors) // finds best match for current face in a provided list of faces
|
||||
```
|
||||
|
||||
Methods used for **body segmentation**, **background removal** or **background replacement**
|
||||
For details, see [segmentation documentation](https://vladmandic.github.io/human/typedoc/classes/Human.html#segmentation)
|
||||
|
||||
```js
|
||||
human.segmentation(input, config?) // runs body segmentation and returns processed image tensor
|
||||
// which can be foreground-only, alpha-only or blended image
|
||||
```
|
||||
|
||||
### Helpers
|
||||
|
||||
Additiona helper namespaces that can be used to reduce amount of manual code that needs to be written, but do not have to be used
|
||||
For details, see:
|
||||
- [Draw methods documentation](https://github.com/vladmandic/human/wiki/Draw) | [Draw options](https://vladmandic.github.io/human/typedoc/interfaces/DrawOptions.html)
|
||||
- [WebCam API specs](https://vladmandic.github.io/human/typedoc/classes/WebCam.html)
|
||||
|
||||
```js
|
||||
human.webcam.* // helper methods to control webcam, main properties are `start`, `stop`, `play`, `pause`
|
||||
human.draw.* // helper methods to draw detected results to canvas, main options are `options`, `canvas`, `all`
|
||||
```
|
||||
|
||||
### Implicit
|
||||
|
||||
Methods that are typically called as part of standard workflow and do not need to be called manually
|
||||
|
||||
```js
|
||||
human.validate(config?); // validate human configuration
|
||||
human.init(config?); // initialize human and processing backend
|
||||
human.load(config?); // load configured models
|
||||
human.warmup(config?); // warms up human library for faster initial execution after loading
|
||||
human.image(input); // process input without detection and returns canvas or tensor
|
||||
human.models.* // namespace that serves current library ml models
|
||||
// main options are `load`, `list`, `loaded`, `reset`, `stats`, `validate`
|
||||
```
|
||||
|
||||
Utility methods that are typically not directly used except in advanced or troubleshooting cases
|
||||
|
||||
```js
|
||||
human.analyze(); // check for memory leaks
|
||||
human.compare(); // compare two images for pixel similarity
|
||||
human.now(); // utility wrapper for performance timestamp
|
||||
human.profile(); // run function via profiler
|
||||
human.reset(); // reset configuration
|
||||
human.sleep(); // utility wrapper for sleep function
|
||||
human.emit(); // internal event emitter
|
||||
```
|
||||
|
||||
## Human Properties
|
||||
|
||||
|
|
@ -65,114 +124,30 @@ human.performance // access to current performance counters
|
|||
human.state // <string> describing current operation in progress
|
||||
// progresses through: 'config', 'check', 'backend', 'load', 'run:<model>', 'idle'
|
||||
human.models // dynamically maintained list of loaded models
|
||||
human.env // detected platform capabilities
|
||||
human.env // detected platform environment and capabilities
|
||||
human.events // container for events dispateched by human
|
||||
Human.defaults // static property of Human class that contains default configuration
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
## Human Methods
|
||||
|
||||
### General
|
||||
|
||||
General purpose methods exposed by `Human`
|
||||
|
||||
```js
|
||||
human.load(config?) // explicitly call load method that loads configured models
|
||||
human.image(input, config?) // runs image processing without detection and returns canvas and tensor
|
||||
human.warmup(config?) // warms up human library for faster initial execution after loading
|
||||
human.next(result?) // returns time variant smoothened/interpolated result based on last known result
|
||||
```
|
||||
|
||||
### Utility
|
||||
|
||||
Utility methods exposed by `Human` that can be used in advanced cases but are typically not needed
|
||||
|
||||
```js
|
||||
human.init() // explict backend initialization
|
||||
human.validate(config?) // validate configuration values
|
||||
human.reset() // reset current configuration to default values
|
||||
human.now() // returns platform-independent timestamp, used for performance measurements
|
||||
human.profile(input, config?) // runs detection with profiling enabled and returns information on top-20 kernels
|
||||
human.compare(input1, input2) // runs pixel-compare on two different inputs and returns score
|
||||
// internally used to detect frame-changes and cache validations
|
||||
```
|
||||
|
||||
### TensorFlow
|
||||
## TensorFlow
|
||||
|
||||
`Human` internally uses `TensorFlow/JS` for all ML processing
|
||||
Access to interal instance of `tfjs` used by `human` is possible via:
|
||||
Access to namespace of an interal instance of `tfjs` used by `human` is possible via:
|
||||
|
||||
```js
|
||||
human.tf // instance of tfjs used by human, can be embedded or externally loaded
|
||||
```
|
||||
### Face Recognition
|
||||
|
||||
Additional functions used for face recognition:
|
||||
For details, see [embedding documentation](https://github.com/vladmandic/human/wiki/Embedding)
|
||||
|
||||
```js
|
||||
human.similarity(descriptor1, descriptor2) // runs similarity calculation between two provided embedding vectors
|
||||
// vectors for source and target must be previously detected using
|
||||
// face.description module
|
||||
human.match(descriptor, descriptors) // finds best match for current face in a provided list of faces
|
||||
human.distance(descriptor1, descriptor2) // checks algorithmic distance between two descriptors
|
||||
// opposite of `similarity`
|
||||
human.enhance(face) // returns enhanced tensor of a previously detected face
|
||||
// that can be used for visualizations
|
||||
```
|
||||
|
||||
### Input Segmentation and Backgroun Removal or Replacement
|
||||
|
||||
`Human` library can attempt to detect outlines of people in provided input and either remove background from input
|
||||
or replace it with a user-provided background image
|
||||
|
||||
For details on parameters and return values see [API Documentation](https://vladmandic.github.io/human/typedoc/classes/Human.html#segmentation)
|
||||
|
||||
```js
|
||||
const input = document.getElementById('my-canvas);
|
||||
const background = document.getElementById('my-background);
|
||||
human.segmentation(input, background);
|
||||
```
|
||||
|
||||
### Draw Functions
|
||||
|
||||
Additional helper functions inside `human.draw`:
|
||||
|
||||
```js
|
||||
human.draw.all(canvas, result) // interpolates results for smoother operations
|
||||
// and triggers each individual draw operation
|
||||
human.draw.person(canvas, result) // triggers unified person analysis and draws bounding box
|
||||
human.draw.canvas(inCanvas, outCanvas) // simply copies one canvas to another,
|
||||
// can be used to draw results.canvas to user canvas on page
|
||||
human.draw.face(canvas, results.face) // draw face detection results to canvas
|
||||
human.draw.body(canvas, results.body) // draw body detection results to canvas
|
||||
human.draw.hand(canvas, result.hand) // draw hand detection results to canvas
|
||||
human.draw.object(canvas, result.object) // draw object detection results to canvas
|
||||
human.draw.gesture(canvas, result.gesture) // draw detected gesture results to canvas
|
||||
```
|
||||
|
||||
Style of drawing is configurable via `human.draw.options` object:
|
||||
|
||||
```js
|
||||
color: 'rgba(173, 216, 230, 0.3)', // 'lightblue' with light alpha channel
|
||||
labelColor: 'rgba(173, 216, 230, 1)', // 'lightblue' with dark alpha channel
|
||||
shadowColor: 'black', // draw shadows underneath labels, set to blank to disable
|
||||
font: 'small-caps 16px "Segoe UI"', // font used for labels
|
||||
lineHeight: 20, // spacing between lines for multi-line labels
|
||||
lineWidth: 6, // line width of drawn polygons
|
||||
drawPoints: true, // draw detected points in all objects
|
||||
pointSize: 2, // size of points
|
||||
drawLabels: true, // draw labels with detection results
|
||||
drawBoxes: true, // draw boxes around detected faces
|
||||
roundRect: 8, // should boxes have round corners and rounding value
|
||||
drawGestures: true, // should draw gestures in top-left part of the canvas
|
||||
drawGaze: true, // should draw gaze arrows
|
||||
drawPolygons: true, // draw polygons such as body and face mesh
|
||||
fillPolygons: false, // fill polygons in face mesh
|
||||
useDepth: true, // use z-axis value when available to determine color shade
|
||||
useCurves: false, // draw polygons and boxes using smooth curves instead of lines
|
||||
human.tf // instance of tfjs used by human, can be embedded or externally loaded
|
||||
```
|
||||
|
||||
<br>
|
||||
|
||||
## Results Caching and Smoothing
|
||||
|
||||
- By default, `Human` uses frame change detection for results caching
|
||||
- For on-screen display best results, it is recommended to use results smoothing
|
||||
|
||||
For details, see <https://github.com/vladmandic/human/wiki/Caching
|
||||
|
||||
## Demos
|
||||
|
||||
`Human` library comes with number of **browser-based** and **nodejs-based** demo apps in `/demo` folder
|
||||
For details, see <https://github.com/vladmandic/human/wiki/Demos>
|
||||
|
|
|
|||
Loading…
Reference in New Issue