mirror of https://github.com/vladmandic/human
initial
Vladimir Mandic
parent
ecd2eb1730
commit
26e765023c
|
|
@ -0,0 +1,22 @@
|
||||||
|
## Build Process
|
||||||
|
|
||||||
|
If you want to modify the library and perform a full rebuild:
|
||||||
|
|
||||||
|
*clone repository, install dependencies, check for errors and run full rebuild from which creates bundles from `/src` into `/dist`:*
|
||||||
|
|
||||||
|
```shell
|
||||||
|
git clone https://github.com/vladmandic/human
|
||||||
|
cd human
|
||||||
|
npm install # installs all project dependencies
|
||||||
|
npm run lint
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
|
||||||
|
This will rebuild library itself (all variations) as well as demo
|
||||||
|
|
||||||
|
Project is written in pure `JavaScript` [ECMAScript version 2020](https://www.ecma-international.org/ecma-262/11.0/index.html)
|
||||||
|
Build target is `JavaScript` **EMCAScript version 2018**
|
||||||
|
|
||||||
|
Only project depdendency is [@tensorflow/tfjs](https://github.com/tensorflow/tfjs)
|
||||||
|
Development dependencies are [eslint](https://github.com/eslint) used for code linting and [esbuild](https://github.com/evanw/esbuild) used for IIFE and ESM script bundling
|
||||||
|
|
||||||
|
|
@ -0,0 +1,170 @@
|
||||||
|
|
||||||
|
# @vladmandic/human
|
||||||
|
|
||||||
|
Version: **0.7.3**
|
||||||
|
Description: **human: 3D Face Detection, Body Pose, Hand & Finger Tracking, Iris Tracking, Age & Gender Prediction, Emotion Prediction & Gesture Recognition**
|
||||||
|
|
||||||
|
Author: **Vladimir Mandic <mandic00@live.com>**
|
||||||
|
License: **MIT** </LICENSE>
|
||||||
|
Repository: **<git+https://github.com/vladmandic/human.git>**
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### **HEAD -> main** 2020/11/05 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.7.3** 2020/11/05 mandic00@live.com
|
||||||
|
- optimized camera and mobile layout
|
||||||
|
- fixed worker and filter compatibility
|
||||||
|
- removed test code
|
||||||
|
|
||||||
|
### **0.7.2** 2020/11/04 mandic00@live.com
|
||||||
|
- major work on handpose model
|
||||||
|
|
||||||
|
### **0.7.1** 2020/11/04 mandic00@live.com
|
||||||
|
- changed demo build process
|
||||||
|
|
||||||
|
### **0.6.7** 2020/11/04 mandic00@live.com
|
||||||
|
- implemented simple gesture recognition
|
||||||
|
|
||||||
|
### **0.6.6** 2020/11/04 mandic00@live.com
|
||||||
|
- remove debug code
|
||||||
|
|
||||||
|
### **0.6.5** 2020/11/04 mandic00@live.com
|
||||||
|
- redo hand detection
|
||||||
|
|
||||||
|
### **0.6.4** 2020/11/03 mandic00@live.com
|
||||||
|
- added manifest
|
||||||
|
|
||||||
|
### **0.6.3** 2020/11/03 mandic00@live.com
|
||||||
|
- enhanced processing resolution
|
||||||
|
- fix pause restart
|
||||||
|
- complete model refactoring
|
||||||
|
- fixed typo
|
||||||
|
|
||||||
|
### **0.6.2** 2020/11/02 mandic00@live.com
|
||||||
|
- optimized demo
|
||||||
|
- package update
|
||||||
|
|
||||||
|
### **0.6.1** 2020/11/02 mandic00@live.com
|
||||||
|
- major performance improvements for all models
|
||||||
|
- revert "optimized canvas handling"
|
||||||
|
- optimized canvas handling
|
||||||
|
- minor optimization to imagefx
|
||||||
|
- fix demo image sample
|
||||||
|
- added tfjs-vis to distribution
|
||||||
|
|
||||||
|
### **0.5.5** 2020/11/01 mandic00@live.com
|
||||||
|
- changed defaults
|
||||||
|
|
||||||
|
### **0.5.4** 2020/11/01 mandic00@live.com
|
||||||
|
- implemented memory profiler
|
||||||
|
|
||||||
|
### **0.5.3** 2020/10/30 mandic00@live.com
|
||||||
|
- improved debug logging
|
||||||
|
|
||||||
|
### **0.5.2** 2020/10/30 mandic00@live.com
|
||||||
|
- added wasm and webgpu backends
|
||||||
|
|
||||||
|
### **0.5.1** 2020/10/30 mandic00@live.com
|
||||||
|
- improve demo line continous draws
|
||||||
|
|
||||||
|
### **0.4.10** 2020/10/30 mandic00@live.com
|
||||||
|
- fix for seedrandom
|
||||||
|
|
||||||
|
### **0.4.9** 2020/10/29 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.4.8** 2020/10/28 mandic00@live.com
|
||||||
|
- revert "updated menu handler"
|
||||||
|
- fix webpack compatibility issue
|
||||||
|
|
||||||
|
### **0.4.7** 2020/10/27 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.4.6** 2020/10/27 mandic00@live.com
|
||||||
|
- fix firefox compatibility bug
|
||||||
|
|
||||||
|
### **0.4.5** 2020/10/27 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.4.4** 2020/10/27 mandic00@live.com
|
||||||
|
- implelented input resizing
|
||||||
|
|
||||||
|
### **0.4.3** 2020/10/22 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.4.2** 2020/10/20 mandic00@live.com
|
||||||
|
- log initialization
|
||||||
|
|
||||||
|
### **0.4.1** 2020/10/19 mandic00@live.com
|
||||||
|
- breaking change: convert to object class
|
||||||
|
- compatibility notes
|
||||||
|
|
||||||
|
### **0.3.9** 2020/10/18 mandic00@live.com
|
||||||
|
- implemented image filters
|
||||||
|
- pure tensor pipeline without image converts
|
||||||
|
- autodetect skipframes
|
||||||
|
|
||||||
|
### **0.3.8** 2020/10/17 mandic00@live.com
|
||||||
|
- new menu layout
|
||||||
|
|
||||||
|
### **0.3.7** 2020/10/17 mandic00@live.com
|
||||||
|
- added diagnostics output
|
||||||
|
- parallelized agegender operations
|
||||||
|
|
||||||
|
### **0.3.6** 2020/10/17 mandic00@live.com
|
||||||
|
- fixed webcam initialization
|
||||||
|
- fixed memory leaks and added scoped runs
|
||||||
|
- modularized draw
|
||||||
|
- added state handling
|
||||||
|
- refactored package file layout
|
||||||
|
|
||||||
|
### **0.3.5** 2020/10/16 mandic00@live.com
|
||||||
|
- added auto-generated changelog
|
||||||
|
|
||||||
|
### **0.3.4** 2020/10/16 mandic00@live.com
|
||||||
|
- added camera selection
|
||||||
|
- optimized blazeface anchors
|
||||||
|
- added error handling
|
||||||
|
|
||||||
|
### **0.3.3** 2020/10/15 mandic00@live.com
|
||||||
|
- added blazeface back and front models
|
||||||
|
|
||||||
|
### **0.3.2** 2020/10/15 mandic00@live.com
|
||||||
|
- reduced web worker latency
|
||||||
|
- added debugging and versioning
|
||||||
|
- optimized demos and added scoped runs
|
||||||
|
- added multi backend support
|
||||||
|
|
||||||
|
### **0.3.1** 2020/10/14 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.2.10** 2020/10/14 mandic00@live.com
|
||||||
|
- added emotion backend
|
||||||
|
- module parametrization and performance monitoring
|
||||||
|
- implemented multi-hand support
|
||||||
|
- fixed documentation typos
|
||||||
|
|
||||||
|
### **0.2.9** 2020/10/13 mandic00@live.com
|
||||||
|
- added node build and demo
|
||||||
|
|
||||||
|
### **0.2.8** 2020/10/13 mandic00@live.com
|
||||||
|
- added example image
|
||||||
|
|
||||||
|
### **0.2.7** 2020/10/13 mandic00@live.com
|
||||||
|
- new examples
|
||||||
|
|
||||||
|
### **0.2.6** 2020/10/13 mandic00@live.com
|
||||||
|
- enable all models by default
|
||||||
|
|
||||||
|
### **0.2.5** 2020/10/12 mandic00@live.com
|
||||||
|
- fixed memory leak
|
||||||
|
|
||||||
|
### **0.2.4** 2020/10/12 mandic00@live.com
|
||||||
|
- removed extra files
|
||||||
|
|
||||||
|
### **0.2.3** 2020/10/12 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.2.2** 2020/10/12 mandic00@live.com
|
||||||
|
|
||||||
|
### **tag: 1.0** 2020/10/12 mandic00@live.com
|
||||||
|
|
||||||
|
### **0.2.1** 2020/10/12 mandic00@live.com
|
||||||
|
- added sample image
|
||||||
|
- initial public commit
|
||||||
|
- initial commit
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
- Face Detection: [**MediaPipe BlazeFace**](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view)
|
||||||
|
- Facial Spacial Geometry: [**MediaPipe FaceMesh**](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view)
|
||||||
|
- Eye Iris Details: [**MediaPipe Iris**](https://drive.google.com/file/d/1bsWbokp9AklH2ANjCfmjqEzzxO1CNbMu/view)
|
||||||
|
- Hand Detection & Skeleton: [**MediaPipe HandPose**](https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view)
|
||||||
|
- Body Pose Detection: [**PoseNet**](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5)
|
||||||
|
- Age & Gender Prediction: [**SSR-Net**](https://github.com/shamangary/SSR-Net)
|
||||||
|
- Emotion Prediction: [**Oarriaga**](https://github.com/oarriaga/face_classification)
|
||||||
|
- Image Filters: [**WebGLImageFilter**](https://github.com/phoboslab/WebGLImageFilter)
|
||||||
|
- Pinto Model Zoo: [**Pinto**](https://github.com/PINTO0309/PINTO_model_zoo)
|
||||||
|
|
||||||
|
*Included models are included under license inherited from the original model source*
|
||||||
|
*Model code has substantially changed from source that it is considered a derivative work and not simple re-publishing*
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
Performance will vary depending on your hardware, but also on number of resolution of input video/image, enabled modules as well as their parameters
|
||||||
|
|
||||||
|
For example, it can perform multiple face detections at 60+ FPS, but drops to ~15 FPS on a medium complex images if all modules are enabled
|
||||||
|
|
||||||
|
### Performance per module on a **notebook** with nVidia GTX1050 GPU on a FullHD input:
|
||||||
|
|
||||||
|
- Enabled all: 15 FPS
|
||||||
|
- Image filters: 80 FPS (standalone)
|
||||||
|
- Gesture: 80 FPS (standalone)
|
||||||
|
- Face Detect: 80 FPS (standalone)
|
||||||
|
- Face Geometry: 30 FPS (includes face detect)
|
||||||
|
- Face Iris: 30 FPS (includes face detect and face geometry)
|
||||||
|
- Age: 60 FPS (includes face detect)
|
||||||
|
- Gender: 60 FPS (includes face detect)
|
||||||
|
- Emotion: 60 FPS (includes face detect)
|
||||||
|
- Hand: 40 FPS (standalone)
|
||||||
|
- Body: 50 FPS (standalone)
|
||||||
|
|
||||||
|
### Performance per module on a **smartphone** with Snapdragon 855 on a FullHD input:
|
||||||
|
|
||||||
|
- Enabled all: 5 FPS
|
||||||
|
- Image filters: 30 FPS (standalone)
|
||||||
|
- Gesture: 30 FPS (standalone)
|
||||||
|
- Face Detect: 20 FPS (standalone)
|
||||||
|
- Face Geometry: 10 FPS (includes face detect)
|
||||||
|
- Face Iris: 5 FPS (includes face detect and face geometry)
|
||||||
|
- Age: 20 FPS (includes face detect)
|
||||||
|
- Gender: 20 FPS (includes face detect)
|
||||||
|
- Emotion: 20 FPS (includes face detect)
|
||||||
|
- Hand: 40 FPS (standalone)
|
||||||
|
- Body: 10 FPS (standalone)
|
||||||
|
|
||||||
|
For performance details, see output of `result.performance` object during after running inference
|
||||||
|
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
## Profiling
|
||||||
|
|
||||||
|
If `config.profile` is enabled, call to `human.profile()` will return detailed profiling data from the last detect invokation.
|
||||||
|
|
||||||
|
example:
|
||||||
|
```js
|
||||||
|
result = {
|
||||||
|
{age: {…}, gender: {…}, emotion: {…}}
|
||||||
|
age:
|
||||||
|
timeKernelOps: 53.78892800000002
|
||||||
|
newBytes: 4
|
||||||
|
newTensors: 1
|
||||||
|
numKernelOps: 341
|
||||||
|
peakBytes: 46033948
|
||||||
|
largestKernelOps: Array(5)
|
||||||
|
0: {name: "Reshape", bytesAdded: 107648, totalBytesSnapshot: 46033948, tensorsAdded: 1, totalTensorsSnapshot: 1149, …}
|
||||||
|
1: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45818652, tensorsAdded: 1, totalTensorsSnapshot: 1147, …}
|
||||||
|
2: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45633996, tensorsAdded: 1, totalTensorsSnapshot: 1148, …}
|
||||||
|
3: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45389376, tensorsAdded: 1, totalTensorsSnapshot: 1154, …}
|
||||||
|
4: {name: "Reshape", bytesAdded: 53824, totalBytesSnapshot: 45381776, tensorsAdded: 1, totalTensorsSnapshot: 1155, …}
|
||||||
|
slowestKernelOps: Array(5)
|
||||||
|
0: {name: "_FusedMatMul", bytesAdded: 12, totalBytesSnapshot: 44802280, tensorsAdded: 1, totalTensorsSnapshot: 1156, …}
|
||||||
|
1: {name: "_FusedMatMul", bytesAdded: 4, totalBytesSnapshot: 44727564, tensorsAdded: 1, totalTensorsSnapshot: 1152, …}
|
||||||
|
2: {name: "_FusedMatMul", bytesAdded: 12, totalBytesSnapshot: 44789100, tensorsAdded: 1, totalTensorsSnapshot: 1157, …}
|
||||||
|
3: {name: "Add", bytesAdded: 4, totalBytesSnapshot: 44788748, tensorsAdded: 1, totalTensorsSnapshot: 1158, …}
|
||||||
|
4: {name: "Add", bytesAdded: 4, totalBytesSnapshot: 44788748, tensorsAdded: 1, totalTensorsSnapshot: 1158, …}
|
||||||
|
}
|
||||||
|
```
|
||||||
Loading…
Reference in New Issue