diff --git a/Configuration.md b/Configuration.md index 564e8d0..29f995e 100644 --- a/Configuration.md +++ b/Configuration.md @@ -25,7 +25,7 @@ With **config.face** having several subsections: - **config.face.age**: controls age prediction - **config.face.gender**: controls gender prediction - **config.face.emotion**: controls emotion prediction -- **config.face.embedding**: controls generation of face embedding data used for face simmilarity checks +- **config.face.embedding**: controls generation of face embedding data used for face similarity checks
@@ -36,7 +36,7 @@ Note that user object and default configuration are merged using deep-merge, so All configuration details can be changed in real-time! ```js -config = { +const config: Config = { backend: 'webgl', // select tfjs backend to use // can be 'webgl', 'wasm', 'cpu', or 'humangl' which is a custom version of webgl // leave as empty string to continue using default backend @@ -133,21 +133,14 @@ config = { modelPath: '../models/iris.json', }, - age: { - enabled: true, - modelPath: '../models/age.json', + description: { + enabled: true, // to improve accuracy of face embedding extraction it is + // recommended to enable detector.rotation and mesh.enabled + modelPath: '../models/faceres.json', skipFrames: 31, // how many frames to go without re-running the detector // only used for video inputs }, - gender: { - enabled: true, - minConfidence: 0.1, // threshold for discarding a prediction - modelPath: '../models/gender.json', - skipFrames: 32, // how many frames to go without re-running the detector - // only used for video inputs - }, - emotion: { enabled: true, minConfidence: 0.1, // threshold for discarding a prediction @@ -155,9 +148,23 @@ config = { modelPath: '../models/emotion.json', }, + age: { + enabled: false, // obsolete, replaced by description module + modelPath: '../models/age.json', + skipFrames: 31, // how many frames to go without re-running the detector + // only used for video inputs + }, + + gender: { + enabled: false, // obsolete, replaced by description module + minConfidence: 0.1, // threshold for discarding a prediction + modelPath: '../models/gender.json', + skipFrames: 32, // how many frames to go without re-running the detector + // only used for video inputs + }, + embedding: { - enabled: false, // to improve accuracy of face embedding extraction it is - // highly recommended to enable detector.rotation and mesh.enabled + enabled: false, // obsolete, replaced by description module modelPath: '../models/mobileface.json', }, }, diff --git a/Demos.md b/Demos.md index 64a9942..9e2bae0 100644 --- a/Demos.md +++ b/Demos.md @@ -55,7 +55,7 @@ It highlights functionality such as: - Loading images - Extracting faces from images - Calculating face embedding descriptors -- Finding face simmilarity and sorting them by simmilarity +- Finding face similarity and sorting them by similarity - Finding best face match based on a known list of faces and printing matches


diff --git a/Embedding.md b/Embedding.md index 417ffeb..d6539bc 100644 --- a/Embedding.md +++ b/Embedding.md @@ -1,16 +1,14 @@ -# Face Feature Embedding and Simmilarity Compare +# Face Feature Embedding and Similarity Compare
-## Demo - To see a demo of all all face embedding features, see `/demo/embedding.js` It highlights functionality such as: - Loading images - Extracting faces from images - Calculating face embedding descriptors -- Finding face simmilarity and sorting them by simmilarity +- Finding face similarity and sorting them by similarity - Finding best face match based on a known list of faces and printing matches
@@ -23,7 +21,7 @@ and calculate embedding vectors for both first and second image you want to comp To achieve quality results, it is also highly recommended to have `face.mesh` and `face.detection.rotation` enabled as calculating feature vectors on non-quality inputs can lead to false results -Simmilarity match above 75% is considered good match while simmilarity match above 60% is considered best-guess +Similarity match above 65% is considered good match while similarity match above 55% is considered best-guess For example, @@ -33,7 +31,8 @@ const myConfig = { enabled: true, detector: { rotation: true, return: true }, mesh: { enabled: true }, - embedding: { enabled: true }, + description: { enabled: true }, + // embedding: { enabled: true }, // alternative you can use embedding module instead of description }, }; @@ -42,9 +41,9 @@ const human = new Human(myConfig); const firstResult = await human.detect(firstImage); const secondResult = await human.detect(secondImage); -const simmilarity = human.simmilarity(firstResult.face[0].embedding, secondResult.face[0].embedding); +const similarity = human.similarity(firstResult.face[0].embedding, secondResult.face[0].embedding); -console.log(`faces are ${100 * simmilarity}% simmilar`); +console.log(`faces are ${100 * similarity}% simmilar`); ``` If the image or video frame have multiple faces and you want to match all of them, simply loop through all `results.face` @@ -52,8 +51,8 @@ If the image or video frame have multiple faces and you want to match all of the ```js for (let i = 0; i < secondResult.face.length; i++) { const secondEmbedding = secondResult.face[i].embedding; - const simmilarity = human.simmilarity(firstEmbedding, secondEmbedding); - console.log(`face ${i} is ${100 * simmilarity}% simmilar`); + const similarity = human.similarity(firstEmbedding, secondEmbedding); + console.log(`face ${i} is ${100 * similarity}% simmilar`); } ``` @@ -76,28 +75,28 @@ They can be stored as normal arrays and reused as needed
-## Face Simmilarity +## Face Similarity -Simmilarity function is based on general *Minkowski distance* between all points in vector +Similarity function is based on general *Minkowski distance* between all points in vector *[Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) is a nth root of sum of nth powers of distances between each point (each value in 192-member array)* *Default is Eucliean distance which is a limited case of Minkowski distance with order of 2* -Changing `order` can make simmilarity matching more or less sensitive (default order is 2nd order) +Changing `order` can make similarity matching more or less sensitive (default order is 2nd order) For example, those will produce slighly different results: ```js - const simmilarity2ndOrder = human.simmilarity(firstEmbedding, secondEmbedding, 2); - const simmilarity3rdOrder = human.simmilarity(firstEmbedding, secondEmbedding, 3); + const similarity2ndOrder = human.similarity(firstEmbedding, secondEmbedding, 2); + const similarity3rdOrder = human.similarity(firstEmbedding, secondEmbedding, 3); ``` -How simmilarity is calculated: +How similarity is calculated: ```js const distance = ((firstEmbedding.map((val, i) => (val - secondEmbedding[i])).reduce((dist, diff) => dist + (diff ** order), 0) ** (1 / order))); ``` -*Once embedding values are calculated and stored, if you want to use stored embedding values without requiring `Human` library you can use above formula to calculate simmilarity on the fly* +*Once embedding values are calculated and stored, if you want to use stored embedding values without requiring `Human` library you can use above formula to calculate similarity on the fly*
@@ -125,12 +124,12 @@ Last parameter is optional and notes a minimal threshold for a match ```js const best = human.match(current.embedding, db, 0) - // return is object: { name: 'person a', simmilarity '0.99', source 'some-image-file' } + // return is object: { name: 'person a', similarity '0.99', source 'some-image-file' } ``` Database can be further stored in a JS or JSON file and retrieved when needed to have a permanent database of faces that can be expanded over time to cover any number of known faces -For example, see `/demo/embedding.js`: +For example, see `/demo/embedding.js` and example database `/demo/faces.json`: ```js // download db with known faces @@ -158,6 +157,6 @@ To achieve optimal result, `Human` performs following operations on an image bef `Human` contains a demo that enumerates number of images, extracts all faces from them, processed them and then allows -for a selection of any face which sorts faces by simmilarity +for a selection of any face which sorts faces by similarity Demo is available in `demo/embedding.html` which uses `demo/embedding.js` as JavaSript module diff --git a/Models.md b/Models.md index 9ae968f..53a9d3b 100644 --- a/Models.md +++ b/Models.md @@ -7,11 +7,9 @@ Default models in Human library are: - **Face Detection**: MediaPipe BlazeFace-Back - **Face Mesh**: MediaPipe FaceMesh - **Face Iris Analysis**: MediaPipe Iris +- **Face Description**: HSE FaceRes - **Emotion Detection**: Oarriaga Emotion -- **Gender Detection**: Oarriaga Gender -- **Age Detection**: SSR-Net Age IMDB - **Body Analysis**: PoseNet -- **Face Embedding**: BecauseofAI MobileFace Embedding - **Object Detection**: NanoDet ## Notes @@ -24,9 +22,11 @@ Default models in Human library are: But if conditions are met, it returns far more details (39 vs 17 keypoints) and is far more accurate Furthermore, it returns 3D approximation of each point instead of 2D -
+**Face description** can be switched from default combined model `FaceRes` to individual models -**Gender detection** can be switched from the default model to `SSR-Net` trained on IMDB dataset +- `Gender Detection`: Oarriaga Gender +- `Age Detection`: SSR-Net Age IMDB +- `Face Embedding`: BecauseofAI MobileFace Embedding


@@ -52,6 +52,7 @@ Default models in Human library are: | BecauseofAI MobileFace | 33K | mobileface.json | 2.1M | mobileface.bin | 75 | | FaceBoxes | 212K | faceboxes.json | 2.0M | faceboxes.bin | N/A | | NanoDet | 255K | nanodet.json | 1.9M | nanodet.bin | 524 | +| FaceRes | 70K | faceres.json | 6.7M | faceres.bin | 524 |
@@ -64,6 +65,7 @@ Default models in Human library are: - Face Detection: [**MediaPipe BlazeFace**](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view) - Facial Spacial Geometry: [**MediaPipe FaceMesh**](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view) - Eye Iris Details: [**MediaPipe Iris**](https://drive.google.com/file/d/1bsWbokp9AklH2ANjCfmjqEzzxO1CNbMu/view) +- Face Description: [**HSE-FaceRes**](https://github.com/HSE-asavchenko/HSE_FaceRec_tf) - Hand Detection & Skeleton: [**MediaPipe HandPose**](https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view) - Body Pose Detection: [**BlazePose**](https://drive.google.com/file/d/10IU-DRP2ioSNjKFdiGbmmQX81xAYj88s/view) - Body Pose Detection: [**PoseNet**](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5) diff --git a/Outputs.md b/Outputs.md index 5e8b564..8131b0c 100644 --- a/Outputs.md +++ b/Outputs.md @@ -21,7 +21,7 @@ result = { iris, // relative distance of iris to camera, multiple by focal lenght to get actual distance age, // estimated age gender, // 'male', 'female' - embedding, // [float] vector of 192 values used for face simmilarity compare + embedding, // [float] vector of 192 values used for face similarity compare angle: // 3d face rotation values in radians in range of -pi/2 to pi/2 which is -90 to +90 degrees { roll, // roll is face lean left/right, value of 0 means center diff --git a/Usage.md b/Usage.md index 68fbe83..e5c330b 100644 --- a/Usage.md +++ b/Usage.md @@ -39,7 +39,7 @@ Additionally, `Human` library exposes several objects and methods: human.image(image, config?) // runs image processing without detection and returns canvas human.warmup(config, image? // warms up human library for faster initial execution after loading // if image is not provided, it will generate internal sample - human.simmilarity(embedding1, embedding2) // runs simmilarity calculation between two provided embedding vectors + human.similarity(embedding1, embedding2) // runs similarity calculation between two provided embedding vectors // vectors for source and target must be previously detected using // face.embedding module human.enhance(face) // returns enhanced tensor of a previously detected face that can be used for visualizations @@ -50,7 +50,7 @@ For details, see [embedding documentation](https://github.com/vladmandic/human/w ```js - human.simmilarity(embedding1, embedding2) // runs simmilarity calculation between two provided embedding vectors + human.similarity(embedding1, embedding2) // runs similarity calculation between two provided embedding vectors // vectors for source and target must be previously detected using // face.embedding module human.match(embedding, db, threshold) // finds best match for current face in a provided list of faces