ambiser
/
human
mirror of https://github.com/vladmandic/human
9
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

breaking change: convert to object class

pull/50/head
Vladimir Mandic 2020-10-19 11:03:48 -04:00
parent 15db4b4b46
commit 7e59258fd7
19 changed files with 1194 additions and 1138 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

74
README.md
View File

@ -70,23 +70,46 @@ Simply download `dist/human.js`, include it in your `HTML` file & it's ready to
<script src="dist/human.js"><script> <script src="dist/human.js"><script>
``` ```
IIFE script auto-registers global namespace `human` within global `Window` object IIFE script auto-registers global namespace `Human` within global `Window` object
Which you can use to create instance of `human` library:
```js
const human = new Human();
```
This way you can also use `Human` library within embbedded `<script>` tag within your `html` page for all-in-one approach This way you can also use `Human` library within embbedded `<script>` tag within your `html` page for all-in-one approach
### 2. [ESM](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import) module ### 2. [ESM](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import) module
*Recommended for usage within `Browser`* *Recommended for usage within `Browser`*
#### 2.1 With Bundler #### **2.1 Using Script Module**
You could use same syntax within your main `JS` file if it's imported with `<script type="module">`
If you're using bundler *(such as rollup, webpack, esbuild)* to package your client application, you can import ESM version of `Human` library which supports full tree shaking ```html
<script src="./index.js" type="module">
```
and then in your `index.js`
```js
import Human from 'dist/human.esm.js'; // for direct import must use path to module, not package name
const human = new Human();
```
#### **2.2 With Bundler**
If you're using bundler *(such as rollup, webpack, parcel, browserify, esbuild)* to package your client application,
you can import ESM version of `Human` library which supports full tree shaking
Install with: Install with:
```shell ```shell
npm install @vladmandic/human npm install @vladmandic/human
``` ```
```js ```js
import human from '@vladmandic/human'; // points to @vladmandic/human/dist/human.esm.js import Human from '@vladmandic/human'; // points to @vladmandic/human/dist/human.esm.js
// you can also force-load specific version
// for example: `@vladmandic/human/dist/human.esm.js`
const human = new Human();
``` ```
Or if you prefer to package your version of `tfjs`, you can use `nobundle` version Or if you prefer to package your version of `tfjs`, you can use `nobundle` version
@ -97,20 +120,8 @@ Install with:
``` ```
```js ```js
import tf from '@tensorflow/tfjs' import tf from '@tensorflow/tfjs'
import human from '@vladmandic/human/dist/human.esm-nobundle.js'; // same functionality as default import, but without tfjs bundled import Human from '@vladmandic/human/dist/human.esm-nobundle.js'; // same functionality as default import, but without tfjs bundled
``` const human = new Human();
#### 2.2 Using Script Module
You could use same syntax within your main `JS` file if it's imported with `<script type="module">`
```html
<script src="./index.js" type="module">
```
and then in your `index.js`
```js
import * as tf from `https://cdnjs.cloudflare.com/ajax/libs/tensorflow/2.6.0/tf.es2017.min.js`; // load tfjs directly from CDN link
import human from 'dist/human.esm.js'; // for direct import must use path to module, not package name
``` ```
### 3. [NPM](https://www.npmjs.com/) module ### 3. [NPM](https://www.npmjs.com/) module
@ -127,7 +138,8 @@ Install with:
And then use with: And then use with:
```js ```js
const tf = require('@tensorflow/tfjs-node'); // can also use '@tensorflow/tfjs-node-gpu' if you have environment with CUDA extensions const tf = require('@tensorflow/tfjs-node'); // can also use '@tensorflow/tfjs-node-gpu' if you have environment with CUDA extensions
const human = require('@vladmandic/human'); // points to @vladmandic/human/dist/human.cjs const Human = require('@vladmandic/human').default; // points to @vladmandic/human/dist/human.cjs
const human = new Human();
``` ```
Since NodeJS projects load `weights` from local filesystem instead of using `http` calls, you must modify default configuration to include correct paths with `file://` prefix Since NodeJS projects load `weights` from local filesystem instead of using `http` calls, you must modify default configuration to include correct paths with `file://` prefix
@ -198,13 +210,20 @@ Additionally, `Human` library exposes several objects and methods:
``` ```
Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards Note that when using `Human` library in `NodeJS`, you must load and parse the image *before* you pass it for detection and dispose it afterwards
Input format is `Tensor4D[1, width, height, 3]` of type `float32`
For example: For example:
```js ```js
const imageFile = '../assets/sample1.jpg'; const imageFile = '../assets/sample1.jpg';
const buffer = fs.readFileSync(imageFile); const buffer = fs.readFileSync(imageFile);
const image = tf.node.decodeImage(buffer); const decoded = tf.node.decodeImage(buffer);
const result = human.detect(image, config); const casted = decoded.toFloat();
const image = casted.expandDims(0);
decoded.dispose();
casted.dispose();
logger.log('Processing:', image.shape);
const human = new Human.Human();
const result = await human.detect(image, config);
image.dispose(); image.dispose();
``` ```
@ -414,15 +433,15 @@ Development dependencies are [eslint](https://github.com/eslint) used for code l
Performance will vary depending on your hardware, but also on number of resolution of input video/image, enabled modules as well as their parameters Performance will vary depending on your hardware, but also on number of resolution of input video/image, enabled modules as well as their parameters
For example, on a desktop with a low-end nVidia GTX1050 it can perform multiple face detections at 60+ FPS, but drops to 10 FPS on a medium complex images if all modules are enabled For example, on a desktop with a low-end nVidia GTX1050 it can perform multiple face detections at 60+ FPS, but drops to ~15 FPS on a medium complex images if all modules are enabled
Performance per module: Performance per module:
- Enabled all: 10 FPS - Enabled all: 15 FPS
- Image filters: 80 FPS (standalone) - Image filters: 80 FPS (standalone)
- Face Detect: 80 FPS (standalone) - Face Detect: 80 FPS (standalone)
- Face Geometry: 30 FPS (includes face detect) - Face Geometry: 30 FPS (includes face detect)
- Face Iris: 25 FPS (includes face detect and face geometry) - Face Iris: 30 FPS (includes face detect and face geometry)
- Age: 60 FPS (includes face detect) - Age: 60 FPS (includes face detect)
- Gender: 60 FPS (includes face detect) - Gender: 60 FPS (includes face detect)
- Emotion: 60 FPS (includes face detect) - Emotion: 60 FPS (includes face detect)
@ -437,8 +456,11 @@ For performance details, see output of `result.performance` object during runtim
`Human` library can be used in any modern Browser or NodeJS environment, but there are several items to be aware of: `Human` library can be used in any modern Browser or NodeJS environment, but there are several items to be aware of:
- **NodeJS**: Due to a missing feature in `tfjs-node`, only some models are available <https://github.com/tensorflow/tfjs/issues/4066> - **NodeJS**: Due to a missing feature in `tfjs-node`, only some models are available
- **Browser**: `filters` module cannot be used when using web workers <https://github.com/phoboslab/WebGLImageFilter/issues/27> For unsupported models, error is: `TypeError: forwardFunc is not a function`
<https://github.com/tensorflow/tfjs/issues/4066>
- **Browser**: Module `filters` cannot be used when using web workers
<https://github.com/phoboslab/WebGLImageFilter/issues/27>
<hr> <hr>

4
demo/browser.js
View File

@ -1,7 +1,9 @@
import human from '../dist/human.esm.js'; import Human from '../dist/human.esm.js';
import draw from './draw.js'; import draw from './draw.js';
import Menu from './menu.js'; import Menu from './menu.js';
const human = new Human();
// ui options // ui options
const ui = { const ui = {
baseColor: 'rgba(173, 216, 230, 0.3)', // this is 'lightblue', just with alpha channel baseColor: 'rgba(173, 216, 230, 0.3)', // this is 'lightblue', just with alpha channel

10
demo/node.js
View File

@ -2,7 +2,7 @@ const tf = require('@tensorflow/tfjs-node');
const fs = require('fs'); const fs = require('fs');
const process = require('process'); const process = require('process');
const console = require('console'); const console = require('console');
const human = require('..'); // this resolves to project root which is '@vladmandic/human' const Human = require('..').default; // this resolves to project root which is '@vladmandic/human'
const logger = new console.Console({ const logger = new console.Console({
stdout: process.stdout, stdout: process.stdout,
@ -26,6 +26,7 @@ const logger = new console.Console({
const config = { const config = {
backend: 'tensorflow', backend: 'tensorflow',
console: true, console: true,
videoOptimized: false,
face: { face: {
detector: { modelPath: 'file://models/blazeface/back/model.json' }, detector: { modelPath: 'file://models/blazeface/back/model.json' },
mesh: { modelPath: 'file://models/facemesh/model.json' }, mesh: { modelPath: 'file://models/facemesh/model.json' },
@ -47,8 +48,13 @@ async function detect(input, output) {
logger.info('TFJS Flags:', tf.env().features); logger.info('TFJS Flags:', tf.env().features);
logger.log('Loading:', input); logger.log('Loading:', input);
const buffer = fs.readFileSync(input); const buffer = fs.readFileSync(input);
const image = tf.node.decodeImage(buffer); const decoded = tf.node.decodeImage(buffer);
const casted = decoded.toFloat();
const image = casted.expandDims(0);
decoded.dispose();
casted.dispose();
logger.log('Processing:', image.shape); logger.log('Processing:', image.shape);
const human = new Human();
const result = await human.detect(image, config); const result = await human.detect(image, config);
image.dispose(); image.dispose();
logger.log(result); logger.log(result);

3
demo/worker.js
View File

@ -1,7 +1,8 @@
import human from '../dist/human.esm.js'; import Human from '../dist/human.esm.js';
let config; let config;
let busy = false; let busy = false;
const human = new Human();
const log = (...msg) => { const log = (...msg) => {
// eslint-disable-next-line no-console // eslint-disable-next-line no-console

493
dist/human.cjs vendored
View File

@ -76,17 +76,17 @@ var require_blazeface = __commonJS((exports2) => {
}); });
} }
class BlazeFaceModel { class BlazeFaceModel {
constructor(model, config2) { constructor(model, config) {
this.blazeFaceModel = model; this.blazeFaceModel = model;
this.width = config2.detector.inputSize; this.width = config.detector.inputSize;
this.height = config2.detector.inputSize; this.height = config.detector.inputSize;
this.maxFaces = config2.detector.maxFaces; this.maxFaces = config.detector.maxFaces;
this.anchorsData = generateAnchors(config2.detector.inputSize); this.anchorsData = generateAnchors(config.detector.inputSize);
this.anchors = tf2.tensor2d(this.anchorsData); this.anchors = tf2.tensor2d(this.anchorsData);
this.inputSize = tf2.tensor1d([this.width, this.height]); this.inputSize = tf2.tensor1d([this.width, this.height]);
this.iouThreshold = config2.detector.iouThreshold; this.iouThreshold = config.detector.iouThreshold;
this.scaleFaces = 0.8; this.scaleFaces = 0.8;
this.scoreThreshold = config2.detector.scoreThreshold; this.scoreThreshold = config.detector.scoreThreshold;
} }
async getBoundingBoxes(inputImage) { async getBoundingBoxes(inputImage) {
if (!inputImage || inputImage.isDisposedInternal || inputImage.shape.length !== 4 || inputImage.shape[1] < 1 || inputImage.shape[2] < 1) if (!inputImage || inputImage.isDisposedInternal || inputImage.shape.length !== 4 || inputImage.shape[1] < 1 || inputImage.shape[2] < 1)
@ -168,12 +168,12 @@ var require_blazeface = __commonJS((exports2) => {
})); }));
} }
} }
async function load2(config2) { async function load(config) {
const blazeface = await tf2.loadGraphModel(config2.detector.modelPath, {fromTFHub: config2.detector.modelPath.includes("tfhub.dev")}); const blazeface = await tf2.loadGraphModel(config.detector.modelPath, {fromTFHub: config.detector.modelPath.includes("tfhub.dev")});
const model = new BlazeFaceModel(blazeface, config2); const model = new BlazeFaceModel(blazeface, config);
return model; return model;
} }
exports2.load = load2; exports2.load = load;
exports2.BlazeFaceModel = BlazeFaceModel; exports2.BlazeFaceModel = BlazeFaceModel;
exports2.disposeBox = disposeBox; exports2.disposeBox = disposeBox;
}); });
@ -442,16 +442,16 @@ var require_pipeline = __commonJS((exports2) => {
} }
} }
class Pipeline { class Pipeline {
constructor(boundingBoxDetector, meshDetector, irisModel, config2) { constructor(boundingBoxDetector, meshDetector, irisModel, config) {
this.regionsOfInterest = []; this.regionsOfInterest = [];
this.runsWithoutFaceDetector = 0; this.runsWithoutFaceDetector = 0;
this.boundingBoxDetector = boundingBoxDetector; this.boundingBoxDetector = boundingBoxDetector;
this.meshDetector = meshDetector; this.meshDetector = meshDetector;
this.irisModel = irisModel; this.irisModel = irisModel;
this.meshWidth = config2.mesh.inputSize; this.meshWidth = config.mesh.inputSize;
this.meshHeight = config2.mesh.inputSize; this.meshHeight = config.mesh.inputSize;
this.irisSize = config2.iris.inputSize; this.irisSize = config.iris.inputSize;
this.irisEnlarge = config2.iris.enlargeFactor; this.irisEnlarge = config.iris.enlargeFactor;
} }
transformRawCoords(rawCoords, box, angle, rotationMatrix) { transformRawCoords(rawCoords, box, angle, rotationMatrix) {
const boxSize = bounding.getBoxSize({startPoint: box.startPoint, endPoint: box.endPoint}); const boxSize = bounding.getBoxSize({startPoint: box.startPoint, endPoint: box.endPoint});
@ -522,9 +522,9 @@ var require_pipeline = __commonJS((exports2) => {
return [coord[0], coord[1], z]; return [coord[0], coord[1], z];
}); });
} }
async predict(input, config2) { async predict(input, config) {
this.skipFrames = config2.detector.skipFrames; this.skipFrames = config.detector.skipFrames;
this.maxFaces = config2.detector.maxFaces; this.maxFaces = config.detector.maxFaces;
this.runsWithoutFaceDetector++; this.runsWithoutFaceDetector++;
if (this.shouldUpdateRegionsOfInterest()) { if (this.shouldUpdateRegionsOfInterest()) {
const detector = await this.boundingBoxDetector.getBoundingBoxes(input); const detector = await this.boundingBoxDetector.getBoundingBoxes(input);
@ -574,7 +574,7 @@ var require_pipeline = __commonJS((exports2) => {
const [, flag, coords] = this.meshDetector.predict(face); const [, flag, coords] = this.meshDetector.predict(face);
const coordsReshaped = tf2.reshape(coords, [-1, 3]); const coordsReshaped = tf2.reshape(coords, [-1, 3]);
let rawCoords = coordsReshaped.arraySync(); let rawCoords = coordsReshaped.arraySync();
if (config2.iris.enabled) { if (config.iris.enabled) {
const {box: leftEyeBox, boxSize: leftEyeBoxSize, crop: leftEyeCrop} = this.getEyeBox(rawCoords, face, LEFT_EYE_BOUNDS[0], LEFT_EYE_BOUNDS[1], true); const {box: leftEyeBox, boxSize: leftEyeBoxSize, crop: leftEyeCrop} = this.getEyeBox(rawCoords, face, LEFT_EYE_BOUNDS[0], LEFT_EYE_BOUNDS[1], true);
const {box: rightEyeBox, boxSize: rightEyeBoxSize, crop: rightEyeCrop} = this.getEyeBox(rawCoords, face, RIGHT_EYE_BOUNDS[0], RIGHT_EYE_BOUNDS[1]); const {box: rightEyeBox, boxSize: rightEyeBoxSize, crop: rightEyeCrop} = this.getEyeBox(rawCoords, face, RIGHT_EYE_BOUNDS[0], RIGHT_EYE_BOUNDS[1]);
const eyePredictions = this.irisModel.predict(tf2.concat([leftEyeCrop, rightEyeCrop])); const eyePredictions = this.irisModel.predict(tf2.concat([leftEyeCrop, rightEyeCrop]));
@ -602,7 +602,7 @@ var require_pipeline = __commonJS((exports2) => {
const landmarksBox = bounding.enlargeBox(this.calculateLandmarksBoundingBox(transformedCoordsData)); const landmarksBox = bounding.enlargeBox(this.calculateLandmarksBoundingBox(transformedCoordsData));
const confidence = flag.squeeze(); const confidence = flag.squeeze();
tf2.dispose(flag); tf2.dispose(flag);
if (config2.mesh.enabled) { if (config.mesh.enabled) {
const transformedCoords = tf2.tensor2d(transformedCoordsData); const transformedCoords = tf2.tensor2d(transformedCoordsData);
this.regionsOfInterest[i] = {...landmarksBox, landmarks: transformedCoords.arraySync()}; this.regionsOfInterest[i] = {...landmarksBox, landmarks: transformedCoords.arraySync()};
const prediction2 = { const prediction2 = {
@ -3804,15 +3804,15 @@ var require_facemesh = __commonJS((exports2) => {
const uv_coords = require_uvcoords(); const uv_coords = require_uvcoords();
const triangulation = require_triangulation().default; const triangulation = require_triangulation().default;
class MediaPipeFaceMesh { class MediaPipeFaceMesh {
constructor(blazeFace, blazeMeshModel, irisModel, config2) { constructor(blazeFace, blazeMeshModel, irisModel, config) {
this.pipeline = new pipe.Pipeline(blazeFace, blazeMeshModel, irisModel, config2); this.pipeline = new pipe.Pipeline(blazeFace, blazeMeshModel, irisModel, config);
if (config2) if (config)
this.config = config2; this.config = config;
} }
async estimateFaces(input, config2) { async estimateFaces(input, config) {
if (config2) if (config)
this.config = config2; this.config = config;
const predictions = await this.pipeline.predict(input, config2); const predictions = await this.pipeline.predict(input, config);
const results = []; const results = [];
for (const prediction of predictions || []) { for (const prediction of predictions || []) {
if (prediction.isDisposedInternal) if (prediction.isDisposedInternal)
@ -3846,16 +3846,16 @@ var require_facemesh = __commonJS((exports2) => {
return results; return results;
} }
} }
async function load2(config2) { async function load(config) {
const models2 = await Promise.all([ const models = await Promise.all([
blazeface.load(config2), blazeface.load(config),
tf2.loadGraphModel(config2.mesh.modelPath, {fromTFHub: config2.mesh.modelPath.includes("tfhub.dev")}), tf2.loadGraphModel(config.mesh.modelPath, {fromTFHub: config.mesh.modelPath.includes("tfhub.dev")}),
tf2.loadGraphModel(config2.iris.modelPath, {fromTFHub: config2.iris.modelPath.includes("tfhub.dev")}) tf2.loadGraphModel(config.iris.modelPath, {fromTFHub: config.iris.modelPath.includes("tfhub.dev")})
]); ]);
const faceMesh = new MediaPipeFaceMesh(models2[0], models2[1], models2[2], config2); const faceMesh = new MediaPipeFaceMesh(models[0], models[1], models[2], config);
return faceMesh; return faceMesh;
} }
exports2.load = load2; exports2.load = load;
exports2.MediaPipeFaceMesh = MediaPipeFaceMesh; exports2.MediaPipeFaceMesh = MediaPipeFaceMesh;
exports2.uv_coords = uv_coords; exports2.uv_coords = uv_coords;
exports2.triangulation = triangulation; exports2.triangulation = triangulation;
@ -3864,35 +3864,35 @@ var require_facemesh = __commonJS((exports2) => {
// src/ssrnet/ssrnet.js // src/ssrnet/ssrnet.js
var require_ssrnet = __commonJS((exports2) => { var require_ssrnet = __commonJS((exports2) => {
const tf2 = require("@tensorflow/tfjs"); const tf2 = require("@tensorflow/tfjs");
const models2 = {}; const models = {};
let last = {age: 0, gender: ""}; let last = {age: 0, gender: ""};
let frame = 0; let frame = 0;
async function loadAge(config2) { async function loadAge(config) {
if (!models2.age) if (!models.age)
models2.age = await tf2.loadGraphModel(config2.face.age.modelPath); models.age = await tf2.loadGraphModel(config.face.age.modelPath);
return models2.age; return models.age;
} }
async function loadGender(config2) { async function loadGender(config) {
if (!models2.gender) if (!models.gender)
models2.gender = await tf2.loadGraphModel(config2.face.gender.modelPath); models.gender = await tf2.loadGraphModel(config.face.gender.modelPath);
return models2.gender; return models.gender;
} }
async function predict(image, config2) { async function predict(image, config) {
if (frame < config2.face.age.skipFrames) { if (frame < config.face.age.skipFrames) {
frame += 1; frame += 1;
return last; return last;
} }
frame = 0; frame = 0;
const resize = tf2.image.resizeBilinear(image, [config2.face.age.inputSize, config2.face.age.inputSize], false); const resize = tf2.image.resizeBilinear(image, [config.face.age.inputSize, config.face.age.inputSize], false);
const enhance = tf2.mul(resize, [255]); const enhance = tf2.mul(resize, [255]);
tf2.dispose(resize); tf2.dispose(resize);
const promises = []; const promises = [];
let ageT; let ageT;
let genderT; let genderT;
if (config2.face.age.enabled) if (config.face.age.enabled)
promises.push(ageT = models2.age.predict(enhance)); promises.push(ageT = models.age.predict(enhance));
if (config2.face.gender.enabled) if (config.face.gender.enabled)
promises.push(genderT = models2.gender.predict(enhance)); promises.push(genderT = models.gender.predict(enhance));
await Promise.all(promises); await Promise.all(promises);
const obj = {}; const obj = {};
if (ageT) { if (ageT) {
@ -3903,7 +3903,7 @@ var require_ssrnet = __commonJS((exports2) => {
if (genderT) { if (genderT) {
const data = await genderT.data(); const data = await genderT.data();
const confidence = Math.trunc(Math.abs(1.9 * 100 * (data[0] - 0.5))) / 100; const confidence = Math.trunc(Math.abs(1.9 * 100 * (data[0] - 0.5))) / 100;
if (confidence > config2.face.gender.minConfidence) { if (confidence > config.face.gender.minConfidence) {
obj.gender = data[0] <= 0.5 ? "female" : "male"; obj.gender = data[0] <= 0.5 ? "female" : "male";
obj.confidence = confidence; obj.confidence = confidence;
} }
@ -3922,22 +3922,22 @@ var require_ssrnet = __commonJS((exports2) => {
var require_emotion = __commonJS((exports2) => { var require_emotion = __commonJS((exports2) => {
const tf2 = require("@tensorflow/tfjs"); const tf2 = require("@tensorflow/tfjs");
const annotations = ["angry", "discust", "fear", "happy", "sad", "surpise", "neutral"]; const annotations = ["angry", "discust", "fear", "happy", "sad", "surpise", "neutral"];
const models2 = {}; const models = {};
let last = []; let last = [];
let frame = 0; let frame = 0;
const multiplier = 1.5; const multiplier = 1.5;
async function load2(config2) { async function load(config) {
if (!models2.emotion) if (!models.emotion)
models2.emotion = await tf2.loadGraphModel(config2.face.emotion.modelPath); models.emotion = await tf2.loadGraphModel(config.face.emotion.modelPath);
return models2.emotion; return models.emotion;
} }
async function predict(image, config2) { async function predict(image, config) {
if (frame < config2.face.emotion.skipFrames) { if (frame < config.face.emotion.skipFrames) {
frame += 1; frame += 1;
return last; return last;
} }
frame = 0; frame = 0;
const resize = tf2.image.resizeBilinear(image, [config2.face.emotion.inputSize, config2.face.emotion.inputSize], false); const resize = tf2.image.resizeBilinear(image, [config.face.emotion.inputSize, config.face.emotion.inputSize], false);
const [red, green, blue] = tf2.split(resize, 3, 3); const [red, green, blue] = tf2.split(resize, 3, 3);
resize.dispose(); resize.dispose();
const redNorm = tf2.mul(red, [0.2989]); const redNorm = tf2.mul(red, [0.2989]);
@ -3951,11 +3951,11 @@ var require_emotion = __commonJS((exports2) => {
greenNorm.dispose(); greenNorm.dispose();
blueNorm.dispose(); blueNorm.dispose();
const obj = []; const obj = [];
if (config2.face.emotion.enabled) { if (config.face.emotion.enabled) {
const emotionT = await models2.emotion.predict(grayscale); const emotionT = await models.emotion.predict(grayscale);
const data = await emotionT.data(); const data = await emotionT.data();
for (let i = 0; i < data.length; i++) { for (let i = 0; i < data.length; i++) {
if (multiplier * data[i] > config2.face.emotion.minConfidence) if (multiplier * data[i] > config.face.emotion.minConfidence)
obj.push({score: Math.min(0.99, Math.trunc(100 * multiplier * data[i]) / 100), emotion: annotations[i]}); obj.push({score: Math.min(0.99, Math.trunc(100 * multiplier * data[i]) / 100), emotion: annotations[i]});
} }
obj.sort((a, b) => b.score - a.score); obj.sort((a, b) => b.score - a.score);
@ -3966,7 +3966,7 @@ var require_emotion = __commonJS((exports2) => {
return obj; return obj;
} }
exports2.predict = predict; exports2.predict = predict;
exports2.load = load2; exports2.load = load;
}); });
// src/posenet/modelBase.js // src/posenet/modelBase.js
@ -4446,19 +4446,19 @@ var require_modelPoseNet = __commonJS((exports2) => {
constructor(net) { constructor(net) {
this.baseModel = net; this.baseModel = net;
} }
async estimatePoses(input, config2) { async estimatePoses(input, config) {
const outputStride = config2.outputStride; const outputStride = config.outputStride;
const height = input.shape[1]; const height = input.shape[1];
const width = input.shape[2]; const width = input.shape[2];
const resized = util.resizeTo(input, [config2.inputResolution, config2.inputResolution]); const resized = util.resizeTo(input, [config.inputResolution, config.inputResolution]);
const {heatmapScores, offsets, displacementFwd, displacementBwd} = this.baseModel.predict(resized); const {heatmapScores, offsets, displacementFwd, displacementBwd} = this.baseModel.predict(resized);
const allTensorBuffers = await util.toTensorBuffers3D([heatmapScores, offsets, displacementFwd, displacementBwd]); const allTensorBuffers = await util.toTensorBuffers3D([heatmapScores, offsets, displacementFwd, displacementBwd]);
const scoresBuffer = allTensorBuffers[0]; const scoresBuffer = allTensorBuffers[0];
const offsetsBuffer = allTensorBuffers[1]; const offsetsBuffer = allTensorBuffers[1];
const displacementsFwdBuffer = allTensorBuffers[2]; const displacementsFwdBuffer = allTensorBuffers[2];
const displacementsBwdBuffer = allTensorBuffers[3]; const displacementsBwdBuffer = allTensorBuffers[3];
const poses = await decodeMultiple.decodeMultiplePoses(scoresBuffer, offsetsBuffer, displacementsFwdBuffer, displacementsBwdBuffer, outputStride, config2.maxDetections, config2.scoreThreshold, config2.nmsRadius); const poses = await decodeMultiple.decodeMultiplePoses(scoresBuffer, offsetsBuffer, displacementsFwdBuffer, displacementsBwdBuffer, outputStride, config.maxDetections, config.scoreThreshold, config.nmsRadius);
const resultPoses = util.scaleAndFlipPoses(poses, [height, width], [config2.inputResolution, config2.inputResolution]); const resultPoses = util.scaleAndFlipPoses(poses, [height, width], [config.inputResolution, config.inputResolution]);
heatmapScores.dispose(); heatmapScores.dispose();
offsets.dispose(); offsets.dispose();
displacementFwd.dispose(); displacementFwd.dispose();
@ -4471,15 +4471,15 @@ var require_modelPoseNet = __commonJS((exports2) => {
} }
} }
exports2.PoseNet = PoseNet; exports2.PoseNet = PoseNet;
async function loadMobileNet(config2) { async function loadMobileNet(config) {
const graphModel = await tf2.loadGraphModel(config2.modelPath); const graphModel = await tf2.loadGraphModel(config.modelPath);
const mobilenet = new modelMobileNet.MobileNet(graphModel, config2.outputStride); const mobilenet = new modelMobileNet.MobileNet(graphModel, config.outputStride);
return new PoseNet(mobilenet); return new PoseNet(mobilenet);
} }
async function load2(config2) { async function load(config) {
return loadMobileNet(config2); return loadMobileNet(config);
} }
exports2.load = load2; exports2.load = load;
}); });
// src/posenet/posenet.js // src/posenet/posenet.js
@ -4580,14 +4580,14 @@ var require_handdetector = __commonJS((exports2) => {
const tf2 = require("@tensorflow/tfjs"); const tf2 = require("@tensorflow/tfjs");
const bounding = require_box2(); const bounding = require_box2();
class HandDetector { class HandDetector {
constructor(model, anchors, config2) { constructor(model, anchors, config) {
this.model = model; this.model = model;
this.width = config2.inputSize; this.width = config.inputSize;
this.height = config2.inputSize; this.height = config.inputSize;
this.anchors = anchors.map((anchor) => [anchor.x_center, anchor.y_center]); this.anchors = anchors.map((anchor) => [anchor.x_center, anchor.y_center]);
this.anchorsTensor = tf2.tensor2d(this.anchors); this.anchorsTensor = tf2.tensor2d(this.anchors);
this.inputSizeTensor = tf2.tensor1d([config2.inputSize, config2.inputSize]); this.inputSizeTensor = tf2.tensor1d([config.inputSize, config.inputSize]);
this.doubleInputSizeTensor = tf2.tensor1d([config2.inputSize * 2, config2.inputSize * 2]); this.doubleInputSizeTensor = tf2.tensor1d([config.inputSize * 2, config.inputSize * 2]);
} }
normalizeBoxes(boxes) { normalizeBoxes(boxes) {
return tf2.tidy(() => { return tf2.tidy(() => {
@ -4629,10 +4629,10 @@ var require_handdetector = __commonJS((exports2) => {
toDispose.forEach((tensor) => tensor.dispose()); toDispose.forEach((tensor) => tensor.dispose());
return detectedHands; return detectedHands;
} }
async estimateHandBounds(input, config2) { async estimateHandBounds(input, config) {
this.iouThreshold = config2.iouThreshold; this.iouThreshold = config.iouThreshold;
this.scoreThreshold = config2.scoreThreshold; this.scoreThreshold = config.scoreThreshold;
this.maxHands = config2.maxHands; this.maxHands = config.maxHands;
const resized = input.resizeBilinear([this.width, this.height]); const resized = input.resizeBilinear([this.width, this.height]);
const divided = resized.div(255); const divided = resized.div(255);
const normalized = divided.sub(0.5); const normalized = divided.sub(0.5);
@ -4758,14 +4758,14 @@ var require_pipeline2 = __commonJS((exports2) => {
const PALM_LANDMARKS_INDEX_OF_PALM_BASE = 0; const PALM_LANDMARKS_INDEX_OF_PALM_BASE = 0;
const PALM_LANDMARKS_INDEX_OF_MIDDLE_FINGER_BASE = 2; const PALM_LANDMARKS_INDEX_OF_MIDDLE_FINGER_BASE = 2;
class HandPipeline { class HandPipeline {
constructor(boundingBoxDetector, meshDetector, config2) { constructor(boundingBoxDetector, meshDetector, config) {
this.regionsOfInterest = []; this.regionsOfInterest = [];
this.runsWithoutHandDetector = 0; this.runsWithoutHandDetector = 0;
this.boundingBoxDetector = boundingBoxDetector; this.boundingBoxDetector = boundingBoxDetector;
this.meshDetector = meshDetector; this.meshDetector = meshDetector;
this.meshWidth = config2.inputSize; this.meshWidth = config.inputSize;
this.meshHeight = config2.inputSize; this.meshHeight = config.inputSize;
this.enlargeFactor = config2.enlargeFactor; this.enlargeFactor = config.enlargeFactor;
} }
getBoxForPalmLandmarks(palmLandmarks, rotationMatrix) { getBoxForPalmLandmarks(palmLandmarks, rotationMatrix) {
const rotatedPalmLandmarks = palmLandmarks.map((coord) => { const rotatedPalmLandmarks = palmLandmarks.map((coord) => {
@ -4810,14 +4810,14 @@ var require_pipeline2 = __commonJS((exports2) => {
coord[2] coord[2]
]); ]);
} }
async estimateHands(image, config2) { async estimateHands(image, config) {
this.skipFrames = config2.skipFrames; this.skipFrames = config.skipFrames;
this.detectionConfidence = config2.minConfidence; this.detectionConfidence = config.minConfidence;
this.maxHands = config2.maxHands; this.maxHands = config.maxHands;
this.runsWithoutHandDetector++; this.runsWithoutHandDetector++;
const useFreshBox = this.shouldUpdateRegionsOfInterest(); const useFreshBox = this.shouldUpdateRegionsOfInterest();
if (useFreshBox === true) { if (useFreshBox === true) {
const boundingBoxPredictions = await this.boundingBoxDetector.estimateHandBounds(image, config2); const boundingBoxPredictions = await this.boundingBoxDetector.estimateHandBounds(image, config);
this.regionsOfInterest = []; this.regionsOfInterest = [];
for (const i in boundingBoxPredictions) { for (const i in boundingBoxPredictions) {
this.updateRegionsOfInterest(boundingBoxPredictions[i], true, i); this.updateRegionsOfInterest(boundingBoxPredictions[i], true, i);
@ -4846,7 +4846,7 @@ var require_pipeline2 = __commonJS((exports2) => {
handImage.dispose(); handImage.dispose();
const flagValue = flag.dataSync()[0]; const flagValue = flag.dataSync()[0];
flag.dispose(); flag.dispose();
if (flagValue < config2.minConfidence) { if (flagValue < config.minConfidence) {
keypoints.dispose(); keypoints.dispose();
this.regionsOfInterest[i] = []; this.regionsOfInterest[i] = [];
return hands; return hands;
@ -4917,11 +4917,11 @@ var require_handpose = __commonJS((exports2) => {
constructor(pipeline) { constructor(pipeline) {
this.pipeline = pipeline; this.pipeline = pipeline;
} }
async estimateHands(input, config2) { async estimateHands(input, config) {
this.skipFrames = config2.skipFrames; this.skipFrames = config.skipFrames;
this.detectionConfidence = config2.minConfidence; this.detectionConfidence = config.minConfidence;
this.maxHands = config2.maxHands; this.maxHands = config.maxHands;
const predictions = await this.pipeline.estimateHands(input, config2); const predictions = await this.pipeline.estimateHands(input, config);
const hands = []; const hands = [];
if (!predictions) if (!predictions)
return hands; return hands;
@ -4951,18 +4951,18 @@ var require_handpose = __commonJS((exports2) => {
} }
return tf2.util.fetch(url).then((d) => d.json()); return tf2.util.fetch(url).then((d) => d.json());
} }
async function load2(config2) { async function load(config) {
const [anchors, handDetectorModel, handPoseModel] = await Promise.all([ const [anchors, handDetectorModel, handPoseModel] = await Promise.all([
loadAnchors(config2.detector.anchors), loadAnchors(config.detector.anchors),
tf2.loadGraphModel(config2.detector.modelPath, {fromTFHub: config2.detector.modelPath.includes("tfhub.dev")}), tf2.loadGraphModel(config.detector.modelPath, {fromTFHub: config.detector.modelPath.includes("tfhub.dev")}),
tf2.loadGraphModel(config2.skeleton.modelPath, {fromTFHub: config2.skeleton.modelPath.includes("tfhub.dev")}) tf2.loadGraphModel(config.skeleton.modelPath, {fromTFHub: config.skeleton.modelPath.includes("tfhub.dev")})
]); ]);
const detector = new hand.HandDetector(handDetectorModel, anchors, config2); const detector = new hand.HandDetector(handDetectorModel, anchors, config);
const pipeline = new pipe.HandPipeline(detector, handPoseModel, config2); const pipeline = new pipe.HandPipeline(detector, handPoseModel, config);
const handpose2 = new HandPose(pipeline); const handpose2 = new HandPose(pipeline);
return handpose2; return handpose2;
} }
exports2.load = load2; exports2.load = load;
}); });
// src/imagefx.js // src/imagefx.js
@ -5840,6 +5840,9 @@ var require_package = __commonJS((exports2, module2) => {
}); });
// src/human.js // src/human.js
__export(exports, {
default: () => Human
});
const tf = require("@tensorflow/tfjs"); const tf = require("@tensorflow/tfjs");
const facemesh = require_facemesh(); const facemesh = require_facemesh();
const ssrnet = require_ssrnet(); const ssrnet = require_ssrnet();
@ -5849,19 +5852,6 @@ const handpose = require_handpose();
const fxImage = require_imagefx(); const fxImage = require_imagefx();
const defaults = require_config().default; const defaults = require_config().default;
const app = require_package(); const app = require_package();
let config;
let fx;
let state = "idle";
let offscreenCanvas;
const models = {
facemesh: null,
posenet: null,
handpose: null,
iris: null,
age: null,
gender: null,
emotion: null
};
const override = { const override = {
face: {detector: {skipFrames: 0}, age: {skipFrames: 0}, emotion: {skipFrames: 0}}, face: {detector: {skipFrames: 0}, age: {skipFrames: 0}, emotion: {skipFrames: 0}},
hand: {skipFrames: 0} hand: {skipFrames: 0}
@ -5871,22 +5861,6 @@ const now = () => {
return performance.now(); return performance.now();
return parseInt(Number(process.hrtime.bigint()) / 1e3 / 1e3); return parseInt(Number(process.hrtime.bigint()) / 1e3 / 1e3);
}; };
const log = (...msg) => {
if (msg && config.console)
console.log(...msg);
};
let numTensors = 0;
const analyzeMemoryLeaks = false;
const analyze = (...msg) => {
if (!analyzeMemoryLeaks)
return;
const current = tf.engine().state.numTensors;
const previous = numTensors;
numTensors = current;
const leaked = current - previous;
if (leaked !== 0)
log(...msg, leaked);
};
function mergeDeep(...objects) { function mergeDeep(...objects) {
const isObject = (obj) => obj && typeof obj === "object"; const isObject = (obj) => obj && typeof obj === "object";
return objects.reduce((prev, obj) => { return objects.reduce((prev, obj) => {
@ -5917,78 +5891,113 @@ function sanity(input) {
} }
return null; return null;
} }
async function load(userConfig) { class Human {
constructor() {
this.tf = tf;
this.version = app.version;
this.defaults = defaults;
this.config = defaults;
this.fx = tf.ENV.flags.IS_BROWSER && typeof document !== "undefined" ? new fxImage.Canvas() : null;
this.state = "idle";
this.numTensors = 0;
this.analyzeMemoryLeaks = false;
this.models = {
facemesh: null,
posenet: null,
handpose: null,
iris: null,
age: null,
gender: null,
emotion: null
};
this.facemesh = facemesh;
this.ssrnet = ssrnet;
this.emotion = emotion;
this.posenet = posenet;
this.handpose = handpose;
}
log(...msg) {
if (msg && this.config.console)
console.log(...msg);
}
analyze(...msg) {
if (!this.analyzeMemoryLeaks)
return;
const current = tf.engine().state.numTensors;
const previous = this.numTensors;
this.numTensors = current;
const leaked = current - previous;
if (leaked !== 0)
this.log(...msg, leaked);
}
async load(userConfig) {
if (userConfig) if (userConfig)
config = mergeDeep(defaults, userConfig); this.config = mergeDeep(defaults, userConfig);
if (config.face.enabled && !models.facemesh) { if (this.config.face.enabled && !this.models.facemesh) {
log("Load model: Face"); this.log("Load model: Face");
models.facemesh = await facemesh.load(config.face); this.models.facemesh = await facemesh.load(this.config.face);
} }
if (config.body.enabled && !models.posenet) { if (this.config.body.enabled && !this.models.posenet) {
log("Load model: Body"); this.log("Load model: Body");
models.posenet = await posenet.load(config.body); this.models.posenet = await posenet.load(this.config.body);
} }
if (config.hand.enabled && !models.handpose) { if (this.config.hand.enabled && !this.models.handpose) {
log("Load model: Hand"); this.log("Load model: Hand");
models.handpose = await handpose.load(config.hand); this.models.handpose = await handpose.load(this.config.hand);
} }
if (config.face.enabled && config.face.age.enabled && !models.age) { if (this.config.face.enabled && this.config.face.age.enabled && !this.models.age) {
log("Load model: Age"); this.log("Load model: Age");
models.age = await ssrnet.loadAge(config); this.models.age = await ssrnet.loadAge(this.config);
} }
if (config.face.enabled && config.face.gender.enabled && !models.gender) { if (this.config.face.enabled && this.config.face.gender.enabled && !this.models.gender) {
log("Load model: Gender"); this.log("Load model: Gender");
models.gender = await ssrnet.loadGender(config); this.models.gender = await ssrnet.loadGender(this.config);
} }
if (config.face.enabled && config.face.emotion.enabled && !models.emotion) { if (this.config.face.enabled && this.config.face.emotion.enabled && !this.models.emotion) {
log("Load model: Emotion"); this.log("Load model: Emotion");
models.emotion = await emotion.load(config); this.models.emotion = await emotion.load(this.config);
} }
} }
function tfImage(input) { tfImage(input) {
let filtered; let filtered;
if (tf.ENV.flags.IS_BROWSER && config.filter.enabled && !(input instanceof tf.Tensor)) { if (this.fx && this.config.filter.enabled && !(input instanceof tf.Tensor)) {
const width = input.naturalWidth || input.videoWidth || input.width || input.shape && input.shape[1] > 0; const width = input.naturalWidth || input.videoWidth || input.width || input.shape && input.shape[1] > 0;
const height = input.naturalHeight || input.videoHeight || input.height || input.shape && input.shape[2] > 0; const height = input.naturalHeight || input.videoHeight || input.height || input.shape && input.shape[2] > 0;
if (!offscreenCanvas) const offscreenCanvas = new OffscreenCanvas(width, height);
offscreenCanvas = new OffscreenCanvas(width, height);
const ctx = offscreenCanvas.getContext("2d"); const ctx = offscreenCanvas.getContext("2d");
if (input instanceof ImageData) if (input instanceof ImageData)
ctx.putImageData(input, 0, 0); ctx.putImageData(input, 0, 0);
else else
ctx.drawImage(input, 0, 0, width, height, 0, 0, offscreenCanvas.width, offscreenCanvas.height); ctx.drawImage(input, 0, 0, width, height, 0, 0, offscreenCanvas.width, offscreenCanvas.height);
if (!fx) this.fx.reset();
fx = new fxImage.Canvas(); this.fx.addFilter("brightness", this.config.filter.brightness);
else if (this.config.filter.contrast !== 0)
fx.reset(); this.fx.addFilter("contrast", this.config.filter.contrast);
fx.addFilter("brightness", config.filter.brightness); if (this.config.filter.sharpness !== 0)
if (config.filter.contrast !== 0) this.fx.addFilter("sharpen", this.config.filter.sharpness);
fx.addFilter("contrast", config.filter.contrast); if (this.config.filter.blur !== 0)
if (config.filter.sharpness !== 0) this.fx.addFilter("blur", this.config.filter.blur);
fx.addFilter("sharpen", config.filter.sharpness); if (this.config.filter.saturation !== 0)
if (config.filter.blur !== 0) this.fx.addFilter("saturation", this.config.filter.saturation);
fx.addFilter("blur", config.filter.blur); if (this.config.filter.hue !== 0)
if (config.filter.saturation !== 0) this.fx.addFilter("hue", this.config.filter.hue);
fx.addFilter("saturation", config.filter.saturation); if (this.config.filter.negative)
if (config.filter.hue !== 0) this.fx.addFilter("negative");
fx.addFilter("hue", config.filter.hue); if (this.config.filter.sepia)
if (config.filter.negative) this.fx.addFilter("sepia");
fx.addFilter("negative"); if (this.config.filter.vintage)
if (config.filter.sepia) this.fx.addFilter("brownie");
fx.addFilter("sepia"); if (this.config.filter.sepia)
if (config.filter.vintage) this.fx.addFilter("sepia");
fx.addFilter("brownie"); if (this.config.filter.kodachrome)
if (config.filter.sepia) this.fx.addFilter("kodachrome");
fx.addFilter("sepia"); if (this.config.filter.technicolor)
if (config.filter.kodachrome) this.fx.addFilter("technicolor");
fx.addFilter("kodachrome"); if (this.config.filter.polaroid)
if (config.filter.technicolor) this.fx.addFilter("polaroid");
fx.addFilter("technicolor"); if (this.config.filter.pixelate !== 0)
if (config.filter.polaroid) this.fx.addFilter("pixelate", this.config.filter.pixelate);
fx.addFilter("polaroid"); filtered = this.fx.apply(offscreenCanvas);
if (config.filter.pixelate !== 0)
fx.addFilter("pixelate", config.filter.pixelate);
filtered = fx.apply(offscreenCanvas);
} }
let tensor; let tensor;
if (input instanceof tf.Tensor) { if (input instanceof tf.Tensor) {
@ -6000,83 +6009,83 @@ function tfImage(input) {
pixels.dispose(); pixels.dispose();
casted.dispose(); casted.dispose();
} }
return {tensor, canvas: config.filter.return ? filtered : null}; return {tensor, canvas: this.config.filter.return ? filtered : null};
} }
async function detect(input, userConfig = {}) { async detect(input, userConfig = {}) {
state = "config"; this.state = "config";
const perf = {}; const perf = {};
let timeStamp; let timeStamp;
timeStamp = now(); timeStamp = now();
config = mergeDeep(defaults, userConfig); this.config = mergeDeep(defaults, userConfig);
if (!config.videoOptimized) if (!this.config.videoOptimized)
config = mergeDeep(config, override); this.config = mergeDeep(this.config, override);
perf.config = Math.trunc(now() - timeStamp); perf.config = Math.trunc(now() - timeStamp);
timeStamp = now(); timeStamp = now();
state = "check"; this.state = "check";
const error = sanity(input); const error = sanity(input);
if (error) { if (error) {
log(error, input); this.log(error, input);
return {error}; return {error};
} }
perf.sanity = Math.trunc(now() - timeStamp); perf.sanity = Math.trunc(now() - timeStamp);
return new Promise(async (resolve) => { return new Promise(async (resolve) => {
const timeStart = now(); const timeStart = now();
timeStamp = now(); timeStamp = now();
if (tf.getBackend() !== config.backend) { if (tf.getBackend() !== this.config.backend) {
state = "backend"; this.state = "backend";
log("Human library setting backend:", config.backend); this.log("Human library setting backend:", this.config.backend);
await tf.setBackend(config.backend); await tf.setBackend(this.config.backend);
await tf.ready(); await tf.ready();
} }
perf.backend = Math.trunc(now() - timeStamp); perf.backend = Math.trunc(now() - timeStamp);
const loadedModels = Object.values(models).filter((a) => a).length; const loadedModels = Object.values(this.models).filter((a) => a).length;
if (loadedModels === 0) { if (loadedModels === 0) {
log("Human library starting"); this.log("Human library starting");
log("Configuration:", config); this.log("Configuration:", this.config);
log("Flags:", tf.ENV.flags); this.log("Flags:", tf.ENV.flags);
} }
timeStamp = now(); timeStamp = now();
state = "load"; this.state = "load";
await load(); await this.load();
perf.load = Math.trunc(now() - timeStamp); perf.load = Math.trunc(now() - timeStamp);
if (config.scoped) if (this.config.scoped)
tf.engine().startScope(); tf.engine().startScope();
analyze("Start Detect:"); this.analyze("Start Detect:");
timeStamp = now(); timeStamp = now();
const image = tfImage(input); const image = this.tfImage(input);
perf.image = Math.trunc(now() - timeStamp); perf.image = Math.trunc(now() - timeStamp);
const imageTensor = image.tensor; const imageTensor = image.tensor;
state = "run:body"; this.state = "run:body";
timeStamp = now(); timeStamp = now();
analyze("Start PoseNet"); this.analyze("Start PoseNet");
const poseRes = config.body.enabled ? await models.posenet.estimatePoses(imageTensor, config.body) : []; const poseRes = this.config.body.enabled ? await this.models.posenet.estimatePoses(imageTensor, this.config.body) : [];
analyze("End PoseNet:"); this.analyze("End PoseNet:");
perf.body = Math.trunc(now() - timeStamp); perf.body = Math.trunc(now() - timeStamp);
state = "run:hand"; this.state = "run:hand";
timeStamp = now(); timeStamp = now();
analyze("Start HandPose:"); this.analyze("Start HandPose:");
const handRes = config.hand.enabled ? await models.handpose.estimateHands(imageTensor, config.hand) : []; const handRes = this.config.hand.enabled ? await this.models.handpose.estimateHands(imageTensor, this.config.hand) : [];
analyze("End HandPose:"); this.analyze("End HandPose:");
perf.hand = Math.trunc(now() - timeStamp); perf.hand = Math.trunc(now() - timeStamp);
const faceRes = []; const faceRes = [];
if (config.face.enabled) { if (this.config.face.enabled) {
state = "run:face"; this.state = "run:face";
timeStamp = now(); timeStamp = now();
analyze("Start FaceMesh:"); this.analyze("Start FaceMesh:");
const faces = await models.facemesh.estimateFaces(imageTensor, config.face); const faces = await this.models.facemesh.estimateFaces(imageTensor, this.config.face);
perf.face = Math.trunc(now() - timeStamp); perf.face = Math.trunc(now() - timeStamp);
for (const face of faces) { for (const face of faces) {
if (!face.image || face.image.isDisposedInternal) { if (!face.image || face.image.isDisposedInternal) {
log("face object is disposed:", face.image); this.log("face object is disposed:", face.image);
continue; continue;
} }
state = "run:agegender"; this.state = "run:agegender";
timeStamp = now(); timeStamp = now();
const ssrData = config.face.age.enabled || config.face.gender.enabled ? await ssrnet.predict(face.image, config) : {}; const ssrData = this.config.face.age.enabled || this.config.face.gender.enabled ? await ssrnet.predict(face.image, this.config) : {};
perf.agegender = Math.trunc(now() - timeStamp); perf.agegender = Math.trunc(now() - timeStamp);
state = "run:emotion"; this.state = "run:emotion";
timeStamp = now(); timeStamp = now();
const emotionData = config.face.emotion.enabled ? await emotion.predict(face.image, config) : {}; const emotionData = this.config.face.emotion.enabled ? await emotion.predict(face.image, this.config) : {};
perf.emotion = Math.trunc(now() - timeStamp); perf.emotion = Math.trunc(now() - timeStamp);
face.image.dispose(); face.image.dispose();
const iris = face.annotations.leftEyeIris && face.annotations.rightEyeIris ? Math.max(face.annotations.leftEyeIris[3][0] - face.annotations.leftEyeIris[1][0], face.annotations.rightEyeIris[3][0] - face.annotations.rightEyeIris[1][0]) : 0; const iris = face.annotations.leftEyeIris && face.annotations.rightEyeIris ? Math.max(face.annotations.leftEyeIris[3][0] - face.annotations.leftEyeIris[1][0], face.annotations.rightEyeIris[3][0] - face.annotations.rightEyeIris[1][0]) : 0;
@ -6091,27 +6100,17 @@ async function detect(input, userConfig = {}) {
emotion: emotionData, emotion: emotionData,
iris: iris !== 0 ? Math.trunc(100 * 11.7 / iris) / 100 : 0 iris: iris !== 0 ? Math.trunc(100 * 11.7 / iris) / 100 : 0
}); });
analyze("End FaceMesh:"); this.analyze("End FaceMesh:");
} }
} }
imageTensor.dispose(); imageTensor.dispose();
state = "idle"; this.state = "idle";
if (config.scoped) if (this.config.scoped)
tf.engine().endScope(); tf.engine().endScope();
analyze("End Scope:"); this.analyze("End Scope:");
perf.total = Math.trunc(now() - timeStart); perf.total = Math.trunc(now() - timeStart);
resolve({face: faceRes, body: poseRes, hand: handRes, performance: perf, canvas: image.canvas}); resolve({face: faceRes, body: poseRes, hand: handRes, performance: perf, canvas: image.canvas});
}); });
}
} }
exports.detect = detect;
exports.defaults = defaults;
exports.config = config;
exports.models = models;
exports.facemesh = facemesh;
exports.ssrnet = ssrnet;
exports.posenet = posenet;
exports.handpose = handpose;
exports.tf = tf;
exports.version = app.version;
exports.state = state;
//# sourceMappingURL=human.cjs.map //# sourceMappingURL=human.cjs.map

29
dist/human.cjs.json vendored
View File

@ -116,7 +116,7 @@
"imports": [] "imports": []
}, },
"src/human.js": { "src/human.js": {
"bytes": 10488, "bytes": 11405,
"imports": [ "imports": [
{ {
"path": "src/facemesh/facemesh.js" "path": "src/facemesh/facemesh.js"
@ -260,13 +260,13 @@
"dist/human.cjs.map": { "dist/human.cjs.map": {
"imports": [], "imports": [],
"inputs": {}, "inputs": {},
"bytes": 250765 "bytes": 264025
}, },
"dist/human.cjs": { "dist/human.cjs": {
"imports": [], "imports": [],
"inputs": { "inputs": {
"src/facemesh/blazeface.js": { "src/facemesh/blazeface.js": {
"bytesInOutput": 7138 "bytesInOutput": 7125
}, },
"src/facemesh/keypoints.js": { "src/facemesh/keypoints.js": {
"bytesInOutput": 2771 "bytesInOutput": 2771
@ -278,7 +278,7 @@
"bytesInOutput": 3027 "bytesInOutput": 3027
}, },
"src/facemesh/pipeline.js": { "src/facemesh/pipeline.js": {
"bytesInOutput": 13270 "bytesInOutput": 13260
}, },
"src/facemesh/uvcoords.js": { "src/facemesh/uvcoords.js": {
"bytesInOutput": 20586 "bytesInOutput": 20586
@ -287,13 +287,13 @@
"bytesInOutput": 23311 "bytesInOutput": 23311
}, },
"src/facemesh/facemesh.js": { "src/facemesh/facemesh.js": {
"bytesInOutput": 2687 "bytesInOutput": 2666
}, },
"src/ssrnet/ssrnet.js": { "src/ssrnet/ssrnet.js": {
"bytesInOutput": 1768 "bytesInOutput": 1748
}, },
"src/emotion/emotion.js": { "src/emotion/emotion.js": {
"bytesInOutput": 1736 "bytesInOutput": 1721
}, },
"src/posenet/modelBase.js": { "src/posenet/modelBase.js": {
"bytesInOutput": 1120 "bytesInOutput": 1120
@ -323,7 +323,7 @@
"bytesInOutput": 2404 "bytesInOutput": 2404
}, },
"src/posenet/modelPoseNet.js": { "src/posenet/modelPoseNet.js": {
"bytesInOutput": 1955 "bytesInOutput": 1939
}, },
"src/posenet/posenet.js": { "src/posenet/posenet.js": {
"bytesInOutput": 917 "bytesInOutput": 917
@ -332,7 +332,7 @@
"bytesInOutput": 2813 "bytesInOutput": 2813
}, },
"src/handpose/handdetector.js": { "src/handpose/handdetector.js": {
"bytesInOutput": 4130 "bytesInOutput": 4119
}, },
"src/handpose/keypoints.js": { "src/handpose/keypoints.js": {
"bytesInOutput": 265 "bytesInOutput": 265
@ -341,10 +341,10 @@
"bytesInOutput": 2671 "bytesInOutput": 2671
}, },
"src/handpose/pipeline.js": { "src/handpose/pipeline.js": {
"bytesInOutput": 7616 "bytesInOutput": 7606
}, },
"src/handpose/handpose.js": { "src/handpose/handpose.js": {
"bytesInOutput": 2288 "bytesInOutput": 2273
}, },
"src/imagefx.js": { "src/imagefx.js": {
"bytesInOutput": 20197 "bytesInOutput": 20197
@ -356,10 +356,13 @@
"bytesInOutput": 2777 "bytesInOutput": 2777
}, },
"src/human.js": { "src/human.js": {
"bytesInOutput": 9246 "bytesInOutput": 47
},
"src/human.js": {
"bytesInOutput": 10244
} }
}, },
"bytes": 153698 "bytes": 154612
} }
} }
} }

6
dist/human.cjs.map vendored
View File

File diff suppressed because one or more lines are too long

16
dist/human.esm-nobundle.js vendored
View File

File diff suppressed because one or more lines are too long

4
dist/human.esm-nobundle.js.map vendored
View File

File diff suppressed because one or more lines are too long

23
dist/human.esm-nobundle.json vendored
View File

@ -116,7 +116,7 @@
"imports": [] "imports": []
}, },
"src/human.js": { "src/human.js": {
"bytes": 10488, "bytes": 11405,
"imports": [ "imports": [
{ {
"path": "src/facemesh/facemesh.js" "path": "src/facemesh/facemesh.js"
@ -260,7 +260,7 @@
"dist/human.esm-nobundle.js.map": { "dist/human.esm-nobundle.js.map": {
"imports": [], "imports": [],
"inputs": {}, "inputs": {},
"bytes": 226578 "bytes": 228083
}, },
"dist/human.esm-nobundle.js": { "dist/human.esm-nobundle.js": {
"imports": [], "imports": [],
@ -272,10 +272,10 @@
"bytesInOutput": 1950 "bytesInOutput": 1950
}, },
"src/facemesh/box.js": { "src/facemesh/box.js": {
"bytesInOutput": 1033 "bytesInOutput": 1026
}, },
"src/facemesh/util.js": { "src/facemesh/util.js": {
"bytesInOutput": 1183 "bytesInOutput": 1176
}, },
"src/facemesh/pipeline.js": { "src/facemesh/pipeline.js": {
"bytesInOutput": 5576 "bytesInOutput": 5576
@ -287,7 +287,7 @@
"bytesInOutput": 9995 "bytesInOutput": 9995
}, },
"src/facemesh/facemesh.js": { "src/facemesh/facemesh.js": {
"bytesInOutput": 1264 "bytesInOutput": 1259
}, },
"src/ssrnet/ssrnet.js": { "src/ssrnet/ssrnet.js": {
"bytesInOutput": 934 "bytesInOutput": 934
@ -314,7 +314,7 @@
"bytesInOutput": 612 "bytesInOutput": 612
}, },
"src/posenet/decodePose.js": { "src/posenet/decodePose.js": {
"bytesInOutput": 1028 "bytesInOutput": 1021
}, },
"src/posenet/decodeMultiple.js": { "src/posenet/decodeMultiple.js": {
"bytesInOutput": 608 "bytesInOutput": 608
@ -338,10 +338,10 @@
"bytesInOutput": 160 "bytesInOutput": 160
}, },
"src/handpose/util.js": { "src/handpose/util.js": {
"bytesInOutput": 984 "bytesInOutput": 977
}, },
"src/handpose/pipeline.js": { "src/handpose/pipeline.js": {
"bytesInOutput": 3216 "bytesInOutput": 3207
}, },
"src/handpose/handpose.js": { "src/handpose/handpose.js": {
"bytesInOutput": 1211 "bytesInOutput": 1211
@ -356,10 +356,13 @@
"bytesInOutput": 2304 "bytesInOutput": 2304
}, },
"src/human.js": { "src/human.js": {
"bytesInOutput": 5180 "bytesInOutput": 6417
},
"src/human.js": {
"bytesInOutput": 0
} }
}, },
"bytes": 80684 "bytes": 81881
} }
} }
} }

500
dist/human.esm.js vendored
View File

File diff suppressed because one or more lines are too long

4
dist/human.esm.js.map vendored
View File

File diff suppressed because one or more lines are too long

11
dist/human.esm.json vendored
View File

@ -291,7 +291,7 @@
"imports": [] "imports": []
}, },
"src/human.js": { "src/human.js": {
"bytes": 10488, "bytes": 11405,
"imports": [ "imports": [
{ {
"path": "node_modules/@tensorflow/tfjs/dist/tf.node.js" "path": "node_modules/@tensorflow/tfjs/dist/tf.node.js"
@ -468,7 +468,7 @@
"dist/human.esm.js.map": { "dist/human.esm.js.map": {
"imports": [], "imports": [],
"inputs": {}, "inputs": {},
"bytes": 4987630 "bytes": 4989135
}, },
"dist/human.esm.js": { "dist/human.esm.js": {
"imports": [], "imports": [],
@ -621,10 +621,13 @@
"bytesInOutput": 2305 "bytesInOutput": 2305
}, },
"src/human.js": { "src/human.js": {
"bytesInOutput": 5335 "bytesInOutput": 6434
},
"src/human.js": {
"bytesInOutput": 0
} }
}, },
"bytes": 1117630 "bytes": 1118731
} }
} }
} }

500
dist/human.js vendored
View File

File diff suppressed because one or more lines are too long

4
dist/human.js.map vendored
View File

File diff suppressed because one or more lines are too long

8
dist/human.json vendored
View File

@ -291,7 +291,7 @@
"imports": [] "imports": []
}, },
"src/human.js": { "src/human.js": {
"bytes": 10488, "bytes": 11405,
"imports": [ "imports": [
{ {
"path": "node_modules/@tensorflow/tfjs/dist/tf.node.js" "path": "node_modules/@tensorflow/tfjs/dist/tf.node.js"
@ -468,7 +468,7 @@
"dist/human.js.map": { "dist/human.js.map": {
"imports": [], "imports": [],
"inputs": {}, "inputs": {},
"bytes": 4987630 "bytes": 4989136
}, },
"dist/human.js": { "dist/human.js": {
"imports": [], "imports": [],
@ -621,10 +621,10 @@
"bytesInOutput": 2305 "bytesInOutput": 2305
}, },
"src/human.js": { "src/human.js": {
"bytesInOutput": 5335 "bytesInOutput": 6476
} }
}, },
"bytes": 1117639 "bytes": 1118780
} }
} }
} }

2
package.json
View File

@ -39,7 +39,7 @@
"scripts": { "scripts": {
"start": "node --trace-warnings --unhandled-rejections=strict --trace-uncaught --no-deprecation src/node.js", "start": "node --trace-warnings --unhandled-rejections=strict --trace-uncaught --no-deprecation src/node.js",
"lint": "eslint src/*.js demo/*.js", "lint": "eslint src/*.js demo/*.js",
"build-iife": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=iife --minify --external:fs --global-name=human --metafile=dist/human.json --outfile=dist/human.js src/human.js", "build-iife": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=iife --minify --external:fs --global-name=Human --metafile=dist/human.json --outfile=dist/human.js src/human.js",
"build-esm-bundle": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=esm --minify --external:fs --metafile=dist/human.esm.json --outfile=dist/human.esm.js src/human.js", "build-esm-bundle": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=esm --minify --external:fs --metafile=dist/human.esm.json --outfile=dist/human.esm.js src/human.js",
"build-esm-nobundle": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=esm --minify --external:@tensorflow --external:fs --metafile=dist/human.esm-nobundle.json --outfile=dist/human.esm-nobundle.js src/human.js", "build-esm-nobundle": "esbuild --bundle --platform=browser --sourcemap --target=esnext --format=esm --minify --external:@tensorflow --external:fs --metafile=dist/human.esm-nobundle.json --outfile=dist/human.esm-nobundle.js src/human.js",
"build-node": "esbuild --bundle --platform=node --sourcemap --target=esnext --format=cjs --external:@tensorflow --metafile=dist/human.cjs.json --outfile=dist/human.cjs src/human.js", "build-node": "esbuild --bundle --platform=node --sourcemap --target=esnext --format=cjs --external:@tensorflow --metafile=dist/human.cjs.json --outfile=dist/human.cjs src/human.js",

276
src/human.js
View File

@ -8,22 +8,7 @@ const fxImage = require('./imagefx.js');
const defaults = require('../config.js').default; const defaults = require('../config.js').default;
const app = require('../package.json'); const app = require('../package.json');
let config; // static config override for non-video detection
let fx;
let state = 'idle';
let offscreenCanvas;
// object that contains all initialized models
const models = {
facemesh: null,
posenet: null,
handpose: null,
iris: null,
age: null,
gender: null,
emotion: null,
};
const override = { const override = {
face: { detector: { skipFrames: 0 }, age: { skipFrames: 0 }, emotion: { skipFrames: 0 } }, face: { detector: { skipFrames: 0 }, age: { skipFrames: 0 }, emotion: { skipFrames: 0 } },
hand: { skipFrames: 0 }, hand: { skipFrames: 0 },
@ -35,24 +20,6 @@ const now = () => {
return parseInt(Number(process.hrtime.bigint()) / 1000 / 1000); return parseInt(Number(process.hrtime.bigint()) / 1000 / 1000);
}; };
// helper function: wrapper around console output
const log = (...msg) => {
// eslint-disable-next-line no-console
if (msg && config.console) console.log(...msg);
};
// helper function: measure tensor leak
let numTensors = 0;
const analyzeMemoryLeaks = false;
const analyze = (...msg) => {
if (!analyzeMemoryLeaks) return;
const current = tf.engine().state.numTensors;
const previous = numTensors;
numTensors = current;
const leaked = current - previous;
if (leaked !== 0) log(...msg, leaked);
};
// helper function: perform deep merge of multiple objects so it allows full inheriance with overrides // helper function: perform deep merge of multiple objects so it allows full inheriance with overrides
function mergeDeep(...objects) { function mergeDeep(...objects) {
const isObject = (obj) => obj && typeof obj === 'object'; const isObject = (obj) => obj && typeof obj === 'object';
@ -85,68 +52,104 @@ function sanity(input) {
return null; return null;
} }
async function load(userConfig) { class Human {
if (userConfig) config = mergeDeep(defaults, userConfig); constructor() {
if (config.face.enabled && !models.facemesh) { this.tf = tf;
log('Load model: Face'); this.version = app.version;
models.facemesh = await facemesh.load(config.face); this.defaults = defaults;
this.config = defaults;
this.fx = (tf.ENV.flags.IS_BROWSER && (typeof document !== 'undefined')) ? new fxImage.Canvas() : null;
this.state = 'idle';
this.numTensors = 0;
this.analyzeMemoryLeaks = false;
// object that contains all initialized models
this.models = {
facemesh: null,
posenet: null,
handpose: null,
iris: null,
age: null,
gender: null,
emotion: null,
};
// export raw access to underlying models
this.facemesh = facemesh;
this.ssrnet = ssrnet;
this.emotion = emotion;
this.posenet = posenet;
this.handpose = handpose;
} }
if (config.body.enabled && !models.posenet) {
log('Load model: Body');
models.posenet = await posenet.load(config.body);
}
if (config.hand.enabled && !models.handpose) {
log('Load model: Hand');
models.handpose = await handpose.load(config.hand);
}
if (config.face.enabled && config.face.age.enabled && !models.age) {
log('Load model: Age');
models.age = await ssrnet.loadAge(config);
}
if (config.face.enabled && config.face.gender.enabled && !models.gender) {
log('Load model: Gender');
models.gender = await ssrnet.loadGender(config);
}
if (config.face.enabled && config.face.emotion.enabled && !models.emotion) {
log('Load model: Emotion');
models.emotion = await emotion.load(config);
}
}
function tfImage(input) { // helper function: wrapper around console output
log(...msg) {
// eslint-disable-next-line no-console
if (msg && this.config.console) console.log(...msg);
}
// helper function: measure tensor leak
analyze(...msg) {
if (!this.analyzeMemoryLeaks) return;
const current = tf.engine().state.numTensors;
const previous = this.numTensors;
this.numTensors = current;
const leaked = current - previous;
if (leaked !== 0) this.log(...msg, leaked);
}
async load(userConfig) {
if (userConfig) this.config = mergeDeep(defaults, userConfig);
if (this.config.face.enabled && !this.models.facemesh) {
this.log('Load model: Face');
this.models.facemesh = await facemesh.load(this.config.face);
}
if (this.config.body.enabled && !this.models.posenet) {
this.log('Load model: Body');
this.models.posenet = await posenet.load(this.config.body);
}
if (this.config.hand.enabled && !this.models.handpose) {
this.log('Load model: Hand');
this.models.handpose = await handpose.load(this.config.hand);
}
if (this.config.face.enabled && this.config.face.age.enabled && !this.models.age) {
this.log('Load model: Age');
this.models.age = await ssrnet.loadAge(this.config);
}
if (this.config.face.enabled && this.config.face.gender.enabled && !this.models.gender) {
this.log('Load model: Gender');
this.models.gender = await ssrnet.loadGender(this.config);
}
if (this.config.face.enabled && this.config.face.emotion.enabled && !this.models.emotion) {
this.log('Load model: Emotion');
this.models.emotion = await emotion.load(this.config);
}
}
tfImage(input) {
// let imageData; // let imageData;
let filtered; let filtered;
if (tf.ENV.flags.IS_BROWSER && config.filter.enabled && !(input instanceof tf.Tensor)) { if (this.fx && this.config.filter.enabled && !(input instanceof tf.Tensor)) {
const width = input.naturalWidth || input.videoWidth || input.width || (input.shape && (input.shape[1] > 0)); const width = input.naturalWidth || input.videoWidth || input.width || (input.shape && (input.shape[1] > 0));
const height = input.naturalHeight || input.videoHeight || input.height || (input.shape && (input.shape[2] > 0)); const height = input.naturalHeight || input.videoHeight || input.height || (input.shape && (input.shape[2] > 0));
if (!offscreenCanvas) offscreenCanvas = new OffscreenCanvas(width, height); const offscreenCanvas = new OffscreenCanvas(width, height);
/*
if (!offscreenCanvas) {
offscreenCanvas = document.createElement('canvas');
offscreenCanvas.width = width;
offscreenCanvas.height = height;
}
*/
const ctx = offscreenCanvas.getContext('2d'); const ctx = offscreenCanvas.getContext('2d');
if (input instanceof ImageData) ctx.putImageData(input, 0, 0); if (input instanceof ImageData) ctx.putImageData(input, 0, 0);
else ctx.drawImage(input, 0, 0, width, height, 0, 0, offscreenCanvas.width, offscreenCanvas.height); else ctx.drawImage(input, 0, 0, width, height, 0, 0, offscreenCanvas.width, offscreenCanvas.height);
if (!fx) fx = new fxImage.Canvas(); this.fx.reset();
else fx.reset(); this.fx.addFilter('brightness', this.config.filter.brightness); // must have at least one filter enabled
fx.addFilter('brightness', config.filter.brightness); // must have at least one filter enabled if (this.config.filter.contrast !== 0) this.fx.addFilter('contrast', this.config.filter.contrast);
if (config.filter.contrast !== 0) fx.addFilter('contrast', config.filter.contrast); if (this.config.filter.sharpness !== 0) this.fx.addFilter('sharpen', this.config.filter.sharpness);
if (config.filter.sharpness !== 0) fx.addFilter('sharpen', config.filter.sharpness); if (this.config.filter.blur !== 0) this.fx.addFilter('blur', this.config.filter.blur);
if (config.filter.blur !== 0) fx.addFilter('blur', config.filter.blur); if (this.config.filter.saturation !== 0) this.fx.addFilter('saturation', this.config.filter.saturation);
if (config.filter.saturation !== 0) fx.addFilter('saturation', config.filter.saturation); if (this.config.filter.hue !== 0) this.fx.addFilter('hue', this.config.filter.hue);
if (config.filter.hue !== 0) fx.addFilter('hue', config.filter.hue); if (this.config.filter.negative) this.fx.addFilter('negative');
if (config.filter.negative) fx.addFilter('negative'); if (this.config.filter.sepia) this.fx.addFilter('sepia');
if (config.filter.sepia) fx.addFilter('sepia'); if (this.config.filter.vintage) this.fx.addFilter('brownie');
if (config.filter.vintage) fx.addFilter('brownie'); if (this.config.filter.sepia) this.fx.addFilter('sepia');
if (config.filter.sepia) fx.addFilter('sepia'); if (this.config.filter.kodachrome) this.fx.addFilter('kodachrome');
if (config.filter.kodachrome) fx.addFilter('kodachrome'); if (this.config.filter.technicolor) this.fx.addFilter('technicolor');
if (config.filter.technicolor) fx.addFilter('technicolor'); if (this.config.filter.polaroid) this.fx.addFilter('polaroid');
if (config.filter.polaroid) fx.addFilter('polaroid'); if (this.config.filter.pixelate !== 0) this.fx.addFilter('pixelate', this.config.filter.pixelate);
if (config.filter.pixelate !== 0) fx.addFilter('pixelate', config.filter.pixelate); filtered = this.fx.apply(offscreenCanvas);
filtered = fx.apply(offscreenCanvas);
} }
let tensor; let tensor;
if (input instanceof tf.Tensor) { if (input instanceof tf.Tensor) {
@ -158,25 +161,25 @@ function tfImage(input) {
pixels.dispose(); pixels.dispose();
casted.dispose(); casted.dispose();
} }
return { tensor, canvas: config.filter.return ? filtered : null }; return { tensor, canvas: this.config.filter.return ? filtered : null };
} }
async function detect(input, userConfig = {}) { async detect(input, userConfig = {}) {
state = 'config'; this.state = 'config';
const perf = {}; const perf = {};
let timeStamp; let timeStamp;
timeStamp = now(); timeStamp = now();
config = mergeDeep(defaults, userConfig); this.config = mergeDeep(defaults, userConfig);
if (!config.videoOptimized) config = mergeDeep(config, override); if (!this.config.videoOptimized) this.config = mergeDeep(this.config, override);
perf.config = Math.trunc(now() - timeStamp); perf.config = Math.trunc(now() - timeStamp);
// sanity checks // sanity checks
timeStamp = now(); timeStamp = now();
state = 'check'; this.state = 'check';
const error = sanity(input); const error = sanity(input);
if (error) { if (error) {
log(error, input); this.log(error, input);
return { error }; return { error };
} }
perf.sanity = Math.trunc(now() - timeStamp); perf.sanity = Math.trunc(now() - timeStamp);
@ -187,76 +190,76 @@ async function detect(input, userConfig = {}) {
// configure backend // configure backend
timeStamp = now(); timeStamp = now();
if (tf.getBackend() !== config.backend) { if (tf.getBackend() !== this.config.backend) {
state = 'backend'; this.state = 'backend';
log('Human library setting backend:', config.backend); this.log('Human library setting backend:', this.config.backend);
await tf.setBackend(config.backend); await tf.setBackend(this.config.backend);
await tf.ready(); await tf.ready();
} }
perf.backend = Math.trunc(now() - timeStamp); perf.backend = Math.trunc(now() - timeStamp);
// check number of loaded models // check number of loaded models
const loadedModels = Object.values(models).filter((a) => a).length; const loadedModels = Object.values(this.models).filter((a) => a).length;
if (loadedModels === 0) { if (loadedModels === 0) {
log('Human library starting'); this.log('Human library starting');
log('Configuration:', config); this.log('Configuration:', this.config);
log('Flags:', tf.ENV.flags); this.log('Flags:', tf.ENV.flags);
} }
// load models if enabled // load models if enabled
timeStamp = now(); timeStamp = now();
state = 'load'; this.state = 'load';
await load(); await this.load();
perf.load = Math.trunc(now() - timeStamp); perf.load = Math.trunc(now() - timeStamp);
if (config.scoped) tf.engine().startScope(); if (this.config.scoped) tf.engine().startScope();
analyze('Start Detect:'); this.analyze('Start Detect:');
timeStamp = now(); timeStamp = now();
const image = tfImage(input); const image = this.tfImage(input);
perf.image = Math.trunc(now() - timeStamp); perf.image = Math.trunc(now() - timeStamp);
const imageTensor = image.tensor; const imageTensor = image.tensor;
// run posenet // run posenet
state = 'run:body'; this.state = 'run:body';
timeStamp = now(); timeStamp = now();
analyze('Start PoseNet'); this.analyze('Start PoseNet');
const poseRes = config.body.enabled ? await models.posenet.estimatePoses(imageTensor, config.body) : []; const poseRes = this.config.body.enabled ? await this.models.posenet.estimatePoses(imageTensor, this.config.body) : [];
analyze('End PoseNet:'); this.analyze('End PoseNet:');
perf.body = Math.trunc(now() - timeStamp); perf.body = Math.trunc(now() - timeStamp);
// run handpose // run handpose
state = 'run:hand'; this.state = 'run:hand';
timeStamp = now(); timeStamp = now();
analyze('Start HandPose:'); this.analyze('Start HandPose:');
const handRes = config.hand.enabled ? await models.handpose.estimateHands(imageTensor, config.hand) : []; const handRes = this.config.hand.enabled ? await this.models.handpose.estimateHands(imageTensor, this.config.hand) : [];
analyze('End HandPose:'); this.analyze('End HandPose:');
perf.hand = Math.trunc(now() - timeStamp); perf.hand = Math.trunc(now() - timeStamp);
// run facemesh, includes blazeface and iris // run facemesh, includes blazeface and iris
const faceRes = []; const faceRes = [];
if (config.face.enabled) { if (this.config.face.enabled) {
state = 'run:face'; this.state = 'run:face';
timeStamp = now(); timeStamp = now();
analyze('Start FaceMesh:'); this.analyze('Start FaceMesh:');
const faces = await models.facemesh.estimateFaces(imageTensor, config.face); const faces = await this.models.facemesh.estimateFaces(imageTensor, this.config.face);
perf.face = Math.trunc(now() - timeStamp); perf.face = Math.trunc(now() - timeStamp);
for (const face of faces) { for (const face of faces) {
// is something went wrong, skip the face // is something went wrong, skip the face
if (!face.image || face.image.isDisposedInternal) { if (!face.image || face.image.isDisposedInternal) {
log('face object is disposed:', face.image); this.log('face object is disposed:', face.image);
continue; continue;
} }
// run ssr-net age & gender, inherits face from blazeface // run ssr-net age & gender, inherits face from blazeface
state = 'run:agegender'; this.state = 'run:agegender';
timeStamp = now(); timeStamp = now();
const ssrData = (config.face.age.enabled || config.face.gender.enabled) ? await ssrnet.predict(face.image, config) : {}; const ssrData = (this.config.face.age.enabled || this.config.face.gender.enabled) ? await ssrnet.predict(face.image, this.config) : {};
perf.agegender = Math.trunc(now() - timeStamp); perf.agegender = Math.trunc(now() - timeStamp);
// run emotion, inherits face from blazeface // run emotion, inherits face from blazeface
state = 'run:emotion'; this.state = 'run:emotion';
timeStamp = now(); timeStamp = now();
const emotionData = config.face.emotion.enabled ? await emotion.predict(face.image, config) : {}; const emotionData = this.config.face.emotion.enabled ? await emotion.predict(face.image, this.config) : {};
perf.emotion = Math.trunc(now() - timeStamp); perf.emotion = Math.trunc(now() - timeStamp);
// dont need face anymore // dont need face anymore
@ -277,31 +280,20 @@ async function detect(input, userConfig = {}) {
emotion: emotionData, emotion: emotionData,
iris: (iris !== 0) ? Math.trunc(100 * 11.7 /* human iris size in mm */ / iris) / 100 : 0, iris: (iris !== 0) ? Math.trunc(100 * 11.7 /* human iris size in mm */ / iris) / 100 : 0,
}); });
analyze('End FaceMesh:'); this.analyze('End FaceMesh:');
} }
} }
imageTensor.dispose(); imageTensor.dispose();
state = 'idle'; this.state = 'idle';
if (config.scoped) tf.engine().endScope(); if (this.config.scoped) tf.engine().endScope();
analyze('End Scope:'); this.analyze('End Scope:');
perf.total = Math.trunc(now() - timeStart); perf.total = Math.trunc(now() - timeStart);
resolve({ face: faceRes, body: poseRes, hand: handRes, performance: perf, canvas: image.canvas }); resolve({ face: faceRes, body: poseRes, hand: handRes, performance: perf, canvas: image.canvas });
}); });
}
} }
exports.detect = detect; export { Human as default };
exports.defaults = defaults;
exports.config = config;
exports.models = models;
exports.facemesh = facemesh;
exports.ssrnet = ssrnet;
exports.posenet = posenet;
exports.handpose = handpose;
exports.tf = tf;
exports.version = app.version;
exports.state = state;
// Error: Failed to compile fragment shader

29
src/node.js
View File

@ -1,6 +1,6 @@
const console = require('console'); const console = require('console');
const tf = require('@tensorflow/tfjs-node'); const tf = require('@tensorflow/tfjs-node');
const human = require('..'); // this resolves to project root which is '@vladmandic/human' const Human = require('..').default; // this resolves to project root which is '@vladmandic/human'
const logger = new console.Console({ const logger = new console.Console({
stdout: process.stdout, stdout: process.stdout,
@ -21,12 +21,37 @@ const logger = new console.Console({
}, },
}); });
const config = {
backend: 'tensorflow',
console: false,
videoOptimized: false,
face: {
detector: { modelPath: 'file://models/blazeface/back/model.json' },
mesh: { modelPath: 'file://models/facemesh/model.json' },
iris: { modelPath: 'file://models/iris/model.json' },
age: { modelPath: 'file://models/ssrnet-age/imdb/model.json' },
gender: { modelPath: 'file://models/ssrnet-gender/imdb/model.json' },
emotion: { modelPath: 'file://models/emotion/model.json' },
},
body: { modelPath: 'file://models/posenet/model.json' },
hand: {
detector: { anchors: 'file://models/handdetect/anchors.json', modelPath: 'file://models/handdetect/model.json' },
skeleton: { modelPath: 'file://models/handskeleton/model.json' },
},
};
async function main() { async function main() {
await tf.ready(); await tf.ready();
const human = new Human();
logger.info('Human:', human.version); logger.info('Human:', human.version);
logger.info('Default Configuration', human.defaults); logger.info('Default Configuration', human.config);
logger.info('TFJS Version:', tf.version_core, 'Backend:', tf.getBackend()); logger.info('TFJS Version:', tf.version_core, 'Backend:', tf.getBackend());
logger.info('TFJS Flags:', tf.env().features); logger.info('TFJS Flags:', tf.env().features);
logger.info('Loading models:');
await human.load(config);
for (const model of Object.keys(human.models)) logger.info(' Loaded:', model);
logger.info('Memory state:', human.tf.engine().memory());
logger.info('Test Complete');
} }
main(); main();