ambiser
/
human
mirror of https://github.com/vladmandic/human
9
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

new module: face description

master
Vladimir Mandic 2021-03-21 14:18:59 -04:00
parent f18a5e77f9
commit ad7e00dab1
6 changed files with 52 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
Configuration.md

@ -25,7 +25,7 @@ With **config.face** having several subsections:
- **config.face.age**: controls age prediction
- **config.face.gender**: controls gender prediction
- **config.face.emotion**: controls emotion prediction
- **config.face.embedding**: controls generation of face embedding data used for face simmilarity checks
- **config.face.embedding**: controls generation of face embedding data used for face similarity checks
<br>
@ -36,7 +36,7 @@ Note that user object and default configuration are merged using deep-merge, so
All configuration details can be changed in real-time!
```js
config = {
const config: Config = {
backend: 'webgl', // select tfjs backend to use
// can be 'webgl', 'wasm', 'cpu', or 'humangl' which is a custom version of webgl
// leave as empty string to continue using default backend
@ -133,21 +133,14 @@ config = {
modelPath: '../models/iris.json',
},
age: {
enabled: true,
modelPath: '../models/age.json',
description: {
enabled: true, // to improve accuracy of face embedding extraction it is
// recommended to enable detector.rotation and mesh.enabled
modelPath: '../models/faceres.json',
skipFrames: 31, // how many frames to go without re-running the detector
// only used for video inputs
},
gender: {
enabled: true,
minConfidence: 0.1, // threshold for discarding a prediction
modelPath: '../models/gender.json',
skipFrames: 32, // how many frames to go without re-running the detector
// only used for video inputs
},
emotion: {
enabled: true,
minConfidence: 0.1, // threshold for discarding a prediction
@ -155,9 +148,23 @@ config = {
modelPath: '../models/emotion.json',
},
age: {
enabled: false, // obsolete, replaced by description module
modelPath: '../models/age.json',
skipFrames: 31, // how many frames to go without re-running the detector
// only used for video inputs
},
gender: {
enabled: false, // obsolete, replaced by description module
minConfidence: 0.1, // threshold for discarding a prediction
modelPath: '../models/gender.json',
skipFrames: 32, // how many frames to go without re-running the detector
// only used for video inputs
},
embedding: {
enabled: false, // to improve accuracy of face embedding extraction it is
// highly recommended to enable detector.rotation and mesh.enabled
enabled: false, // obsolete, replaced by description module
modelPath: '../models/mobileface.json',
},
},

2
Demos.md

@ -55,7 +55,7 @@ It highlights functionality such as:
- Loading images
- Extracting faces from images
- Calculating face embedding descriptors
- Finding face simmilarity and sorting them by simmilarity
- Finding face similarity and sorting them by similarity
- Finding best face match based on a known list of faces and printing matches
<br><hr><br>

39
Embedding.md

@ -1,16 +1,14 @@
# Face Feature Embedding and Simmilarity Compare
# Face Feature Embedding and Similarity Compare
<br>
## Demo
To see a demo of all all face embedding features, see `/demo/embedding.js`
It highlights functionality such as:
- Loading images
- Extracting faces from images
- Calculating face embedding descriptors
- Finding face simmilarity and sorting them by simmilarity
- Finding face similarity and sorting them by similarity
- Finding best face match based on a known list of faces and printing matches
<br>
@ -23,7 +21,7 @@ and calculate embedding vectors for both first and second image you want to comp
To achieve quality results, it is also highly recommended to have `face.mesh` and `face.detection.rotation`
enabled as calculating feature vectors on non-quality inputs can lead to false results
Simmilarity match above 75% is considered good match while simmilarity match above 60% is considered best-guess
Similarity match above 65% is considered good match while similarity match above 55% is considered best-guess
For example,
@ -33,7 +31,8 @@ const myConfig = {
enabled: true,
detector: { rotation: true, return: true },
mesh: { enabled: true },
embedding: { enabled: true },
description: { enabled: true },
// embedding: { enabled: true }, // alternative you can use embedding module instead of description
},
};
@ -42,9 +41,9 @@ const human = new Human(myConfig);
const firstResult = await human.detect(firstImage);
const secondResult = await human.detect(secondImage);
const simmilarity = human.simmilarity(firstResult.face[0].embedding, secondResult.face[0].embedding);
const similarity = human.similarity(firstResult.face[0].embedding, secondResult.face[0].embedding);
console.log(`faces are ${100 * simmilarity}% simmilar`);
console.log(`faces are ${100 * similarity}% simmilar`);
```
If the image or video frame have multiple faces and you want to match all of them, simply loop through all `results.face`
@ -52,8 +51,8 @@ If the image or video frame have multiple faces and you want to match all of the
```js
for (let i = 0; i < secondResult.face.length; i++) {
const secondEmbedding = secondResult.face[i].embedding;
const simmilarity = human.simmilarity(firstEmbedding, secondEmbedding);
console.log(`face ${i} is ${100 * simmilarity}% simmilar`);
const similarity = human.similarity(firstEmbedding, secondEmbedding);
console.log(`face ${i} is ${100 * similarity}% simmilar`);
}
```
@ -76,28 +75,28 @@ They can be stored as normal arrays and reused as needed
<br>
## Face Simmilarity
## Face Similarity
Simmilarity function is based on general *Minkowski distance* between all points in vector
Similarity function is based on general *Minkowski distance* between all points in vector
*[Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) is a nth root of sum of nth powers of distances between each point (each value in 192-member array)*
*Default is Eucliean distance which is a limited case of Minkowski distance with order of 2*
Changing `order` can make simmilarity matching more or less sensitive (default order is 2nd order)
Changing `order` can make similarity matching more or less sensitive (default order is 2nd order)
For example, those will produce slighly different results:
```js
const simmilarity2ndOrder = human.simmilarity(firstEmbedding, secondEmbedding, 2);
const simmilarity3rdOrder = human.simmilarity(firstEmbedding, secondEmbedding, 3);
const similarity2ndOrder = human.similarity(firstEmbedding, secondEmbedding, 2);
const similarity3rdOrder = human.similarity(firstEmbedding, secondEmbedding, 3);
```
How simmilarity is calculated:
How similarity is calculated:
```js
const distance = ((firstEmbedding.map((val, i) => (val - secondEmbedding[i])).reduce((dist, diff) => dist + (diff ** order), 0) ** (1 / order)));
```
*Once embedding values are calculated and stored, if you want to use stored embedding values without requiring `Human` library you can use above formula to calculate simmilarity on the fly*
*Once embedding values are calculated and stored, if you want to use stored embedding values without requiring `Human` library you can use above formula to calculate similarity on the fly*
<br>
@ -125,12 +124,12 @@ Last parameter is optional and notes a minimal threshold for a match
```js
const best = human.match(current.embedding, db, 0)
// return is object: { name: 'person a', simmilarity '0.99', source 'some-image-file' }
// return is object: { name: 'person a', similarity '0.99', source 'some-image-file' }
```
Database can be further stored in a JS or JSON file and retrieved when needed to have
a permanent database of faces that can be expanded over time to cover any number of known faces
For example, see `/demo/embedding.js`:
For example, see `/demo/embedding.js` and example database `/demo/faces.json`:
```js
// download db with known faces
@ -158,6 +157,6 @@ To achieve optimal result, `Human` performs following operations on an image bef
`Human` contains a demo that enumerates number of images,
extracts all faces from them, processed them and then allows
for a selection of any face which sorts faces by simmilarity
for a selection of any face which sorts faces by similarity
Demo is available in `demo/embedding.html` which uses `demo/embedding.js` as JavaSript module

12
Models.md

@ -7,11 +7,9 @@ Default models in Human library are:
- **Face Detection**: MediaPipe BlazeFace-Back
- **Face Mesh**: MediaPipe FaceMesh
- **Face Iris Analysis**: MediaPipe Iris
- **Face Description**: HSE FaceRes
- **Emotion Detection**: Oarriaga Emotion
- **Gender Detection**: Oarriaga Gender
- **Age Detection**: SSR-Net Age IMDB
- **Body Analysis**: PoseNet
- **Face Embedding**: BecauseofAI MobileFace Embedding
- **Object Detection**: NanoDet
## Notes
@ -24,9 +22,11 @@ Default models in Human library are:
But if conditions are met, it returns far more details (39 vs 17 keypoints) and is far more accurate
Furthermore, it returns 3D approximation of each point instead of 2D
<br>
**Face description** can be switched from default combined model `FaceRes` to individual models
**Gender detection** can be switched from the default model to `SSR-Net` trained on IMDB dataset
- `Gender Detection`: Oarriaga Gender
- `Age Detection`: SSR-Net Age IMDB
- `Face Embedding`: BecauseofAI MobileFace Embedding
<br><hr><br>
@ -52,6 +52,7 @@ Default models in Human library are:
| BecauseofAI MobileFace | 33K | mobileface.json | 2.1M | mobileface.bin | 75 |
| FaceBoxes | 212K | faceboxes.json | 2.0M | faceboxes.bin | N/A |
| NanoDet | 255K | nanodet.json | 1.9M | nanodet.bin | 524 |
| FaceRes | 70K | faceres.json | 6.7M | faceres.bin | 524 |
<br>
@ -64,6 +65,7 @@ Default models in Human library are:
- Face Detection: [**MediaPipe BlazeFace**](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view)
- Facial Spacial Geometry: [**MediaPipe FaceMesh**](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view)
- Eye Iris Details: [**MediaPipe Iris**](https://drive.google.com/file/d/1bsWbokp9AklH2ANjCfmjqEzzxO1CNbMu/view)
- Face Description: [**HSE-FaceRes**](https://github.com/HSE-asavchenko/HSE_FaceRec_tf)
- Hand Detection & Skeleton: [**MediaPipe HandPose**](https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view)
- Body Pose Detection: [**BlazePose**](https://drive.google.com/file/d/10IU-DRP2ioSNjKFdiGbmmQX81xAYj88s/view)
- Body Pose Detection: [**PoseNet**](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5)

2
Outputs.md

@ -21,7 +21,7 @@ result = {
iris, // <number> relative distance of iris to camera, multiple by focal lenght to get actual distance
age, // <number> estimated age
gender, // <string> 'male', 'female'
embedding, // <array>[float] vector of 192 values used for face simmilarity compare
embedding, // <array>[float] vector of 192 values used for face similarity compare
angle: // 3d face rotation values in radians in range of -pi/2 to pi/2 which is -90 to +90 degrees
{
roll, // roll is face lean left/right, value of 0 means center

4
Usage.md

@ -39,7 +39,7 @@ Additionally, `Human` library exposes several objects and methods:
human.image(image, config?) // runs image processing without detection and returns canvas
human.warmup(config, image? // warms up human library for faster initial execution after loading
// if image is not provided, it will generate internal sample
human.simmilarity(embedding1, embedding2) // runs simmilarity calculation between two provided embedding vectors
human.similarity(embedding1, embedding2) // runs similarity calculation between two provided embedding vectors
// vectors for source and target must be previously detected using
// face.embedding module
human.enhance(face) // returns enhanced tensor of a previously detected face that can be used for visualizations
@ -50,7 +50,7 @@ For details, see [embedding documentation](https://github.com/vladmandic/human/w
```js
human.simmilarity(embedding1, embedding2) // runs simmilarity calculation between two provided embedding vectors
human.similarity(embedding1, embedding2) // runs similarity calculation between two provided embedding vectors
// vectors for source and target must be previously detected using
// face.embedding module
human.match(embedding, db, threshold) // finds best match for current face in a provided list of faces