diff --git a/Build-Process.md b/Build-Process.md new file mode 100644 index 0000000..5cc8c89 --- /dev/null +++ b/Build-Process.md @@ -0,0 +1,22 @@ +## Build Process + +If you want to modify the library and perform a full rebuild: + +*clone repository, install dependencies, check for errors and run full rebuild from which creates bundles from `/src` into `/dist`:* + +```shell +git clone https://github.com/vladmandic/human +cd human +npm install # installs all project dependencies +npm run lint +npm run build +``` + +This will rebuild library itself (all variations) as well as demo + +Project is written in pure `JavaScript` [ECMAScript version 2020](https://www.ecma-international.org/ecma-262/11.0/index.html) +Build target is `JavaScript` **EMCAScript version 2018** + +Only project depdendency is [@tensorflow/tfjs](https://github.com/tensorflow/tfjs) +Development dependencies are [eslint](https://github.com/eslint) used for code linting and [esbuild](https://github.com/evanw/esbuild) used for IIFE and ESM script bundling + diff --git a/Change-Log.md b/Change-Log.md new file mode 100644 index 0000000..7924105 --- /dev/null +++ b/Change-Log.md @@ -0,0 +1,170 @@ + +# @vladmandic/human + +Version: **0.7.3** +Description: **human: 3D Face Detection, Body Pose, Hand & Finger Tracking, Iris Tracking, Age & Gender Prediction, Emotion Prediction & Gesture Recognition** + +Author: **Vladimir Mandic ** +License: **MIT** +Repository: **** + +## Changelog + +### **HEAD -> main** 2020/11/05 mandic00@live.com + +### **0.7.3** 2020/11/05 mandic00@live.com +- optimized camera and mobile layout +- fixed worker and filter compatibility +- removed test code + +### **0.7.2** 2020/11/04 mandic00@live.com +- major work on handpose model + +### **0.7.1** 2020/11/04 mandic00@live.com +- changed demo build process + +### **0.6.7** 2020/11/04 mandic00@live.com +- implemented simple gesture recognition + +### **0.6.6** 2020/11/04 mandic00@live.com +- remove debug code + +### **0.6.5** 2020/11/04 mandic00@live.com +- redo hand detection + +### **0.6.4** 2020/11/03 mandic00@live.com +- added manifest + +### **0.6.3** 2020/11/03 mandic00@live.com +- enhanced processing resolution +- fix pause restart +- complete model refactoring +- fixed typo + +### **0.6.2** 2020/11/02 mandic00@live.com +- optimized demo +- package update + +### **0.6.1** 2020/11/02 mandic00@live.com +- major performance improvements for all models +- revert "optimized canvas handling" +- optimized canvas handling +- minor optimization to imagefx +- fix demo image sample +- added tfjs-vis to distribution + +### **0.5.5** 2020/11/01 mandic00@live.com +- changed defaults + +### **0.5.4** 2020/11/01 mandic00@live.com +- implemented memory profiler + +### **0.5.3** 2020/10/30 mandic00@live.com +- improved debug logging + +### **0.5.2** 2020/10/30 mandic00@live.com +- added wasm and webgpu backends + +### **0.5.1** 2020/10/30 mandic00@live.com +- improve demo line continous draws + +### **0.4.10** 2020/10/30 mandic00@live.com +- fix for seedrandom + +### **0.4.9** 2020/10/29 mandic00@live.com + +### **0.4.8** 2020/10/28 mandic00@live.com +- revert "updated menu handler" +- fix webpack compatibility issue + +### **0.4.7** 2020/10/27 mandic00@live.com + +### **0.4.6** 2020/10/27 mandic00@live.com +- fix firefox compatibility bug + +### **0.4.5** 2020/10/27 mandic00@live.com + +### **0.4.4** 2020/10/27 mandic00@live.com +- implelented input resizing + +### **0.4.3** 2020/10/22 mandic00@live.com + +### **0.4.2** 2020/10/20 mandic00@live.com +- log initialization + +### **0.4.1** 2020/10/19 mandic00@live.com +- breaking change: convert to object class +- compatibility notes + +### **0.3.9** 2020/10/18 mandic00@live.com +- implemented image filters +- pure tensor pipeline without image converts +- autodetect skipframes + +### **0.3.8** 2020/10/17 mandic00@live.com +- new menu layout + +### **0.3.7** 2020/10/17 mandic00@live.com +- added diagnostics output +- parallelized agegender operations + +### **0.3.6** 2020/10/17 mandic00@live.com +- fixed webcam initialization +- fixed memory leaks and added scoped runs +- modularized draw +- added state handling +- refactored package file layout + +### **0.3.5** 2020/10/16 mandic00@live.com +- added auto-generated changelog + +### **0.3.4** 2020/10/16 mandic00@live.com +- added camera selection +- optimized blazeface anchors +- added error handling + +### **0.3.3** 2020/10/15 mandic00@live.com +- added blazeface back and front models + +### **0.3.2** 2020/10/15 mandic00@live.com +- reduced web worker latency +- added debugging and versioning +- optimized demos and added scoped runs +- added multi backend support + +### **0.3.1** 2020/10/14 mandic00@live.com + +### **0.2.10** 2020/10/14 mandic00@live.com +- added emotion backend +- module parametrization and performance monitoring +- implemented multi-hand support +- fixed documentation typos + +### **0.2.9** 2020/10/13 mandic00@live.com +- added node build and demo + +### **0.2.8** 2020/10/13 mandic00@live.com +- added example image + +### **0.2.7** 2020/10/13 mandic00@live.com +- new examples + +### **0.2.6** 2020/10/13 mandic00@live.com +- enable all models by default + +### **0.2.5** 2020/10/12 mandic00@live.com +- fixed memory leak + +### **0.2.4** 2020/10/12 mandic00@live.com +- removed extra files + +### **0.2.3** 2020/10/12 mandic00@live.com + +### **0.2.2** 2020/10/12 mandic00@live.com + +### **tag: 1.0** 2020/10/12 mandic00@live.com + +### **0.2.1** 2020/10/12 mandic00@live.com +- added sample image +- initial public commit +- initial commit diff --git a/Credits.md b/Credits.md new file mode 100644 index 0000000..9a49631 --- /dev/null +++ b/Credits.md @@ -0,0 +1,14 @@ +## Credits + +- Face Detection: [**MediaPipe BlazeFace**](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view) +- Facial Spacial Geometry: [**MediaPipe FaceMesh**](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view) +- Eye Iris Details: [**MediaPipe Iris**](https://drive.google.com/file/d/1bsWbokp9AklH2ANjCfmjqEzzxO1CNbMu/view) +- Hand Detection & Skeleton: [**MediaPipe HandPose**](https://drive.google.com/file/d/1sv4sSb9BSNVZhLzxXJ0jBv9DqD-4jnAz/view) +- Body Pose Detection: [**PoseNet**](https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5) +- Age & Gender Prediction: [**SSR-Net**](https://github.com/shamangary/SSR-Net) +- Emotion Prediction: [**Oarriaga**](https://github.com/oarriaga/face_classification) +- Image Filters: [**WebGLImageFilter**](https://github.com/phoboslab/WebGLImageFilter) +- Pinto Model Zoo: [**Pinto**](https://github.com/PINTO0309/PINTO_model_zoo) + +*Included models are included under license inherited from the original model source* +*Model code has substantially changed from source that it is considered a derivative work and not simple re-publishing* diff --git a/Performance.md b/Performance.md new file mode 100644 index 0000000..26da612 --- /dev/null +++ b/Performance.md @@ -0,0 +1,36 @@ +## Performance + +Performance will vary depending on your hardware, but also on number of resolution of input video/image, enabled modules as well as their parameters + +For example, it can perform multiple face detections at 60+ FPS, but drops to ~15 FPS on a medium complex images if all modules are enabled + +### Performance per module on a **notebook** with nVidia GTX1050 GPU on a FullHD input: + +- Enabled all: 15 FPS +- Image filters: 80 FPS (standalone) +- Gesture: 80 FPS (standalone) +- Face Detect: 80 FPS (standalone) +- Face Geometry: 30 FPS (includes face detect) +- Face Iris: 30 FPS (includes face detect and face geometry) +- Age: 60 FPS (includes face detect) +- Gender: 60 FPS (includes face detect) +- Emotion: 60 FPS (includes face detect) +- Hand: 40 FPS (standalone) +- Body: 50 FPS (standalone) + +### Performance per module on a **smartphone** with Snapdragon 855 on a FullHD input: + +- Enabled all: 5 FPS +- Image filters: 30 FPS (standalone) +- Gesture: 30 FPS (standalone) +- Face Detect: 20 FPS (standalone) +- Face Geometry: 10 FPS (includes face detect) +- Face Iris: 5 FPS (includes face detect and face geometry) +- Age: 20 FPS (includes face detect) +- Gender: 20 FPS (includes face detect) +- Emotion: 20 FPS (includes face detect) +- Hand: 40 FPS (standalone) +- Body: 10 FPS (standalone) + +For performance details, see output of `result.performance` object during after running inference + diff --git a/Profiling.md b/Profiling.md new file mode 100644 index 0000000..686747b --- /dev/null +++ b/Profiling.md @@ -0,0 +1,28 @@ +## Profiling + +If `config.profile` is enabled, call to `human.profile()` will return detailed profiling data from the last detect invokation. + +example: +```js + result = { + {age: {…}, gender: {…}, emotion: {…}} + age: + timeKernelOps: 53.78892800000002 + newBytes: 4 + newTensors: 1 + numKernelOps: 341 + peakBytes: 46033948 + largestKernelOps: Array(5) + 0: {name: "Reshape", bytesAdded: 107648, totalBytesSnapshot: 46033948, tensorsAdded: 1, totalTensorsSnapshot: 1149, …} + 1: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45818652, tensorsAdded: 1, totalTensorsSnapshot: 1147, …} + 2: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45633996, tensorsAdded: 1, totalTensorsSnapshot: 1148, …} + 3: {name: "Reshape", bytesAdded: 0, totalBytesSnapshot: 45389376, tensorsAdded: 1, totalTensorsSnapshot: 1154, …} + 4: {name: "Reshape", bytesAdded: 53824, totalBytesSnapshot: 45381776, tensorsAdded: 1, totalTensorsSnapshot: 1155, …} + slowestKernelOps: Array(5) + 0: {name: "_FusedMatMul", bytesAdded: 12, totalBytesSnapshot: 44802280, tensorsAdded: 1, totalTensorsSnapshot: 1156, …} + 1: {name: "_FusedMatMul", bytesAdded: 4, totalBytesSnapshot: 44727564, tensorsAdded: 1, totalTensorsSnapshot: 1152, …} + 2: {name: "_FusedMatMul", bytesAdded: 12, totalBytesSnapshot: 44789100, tensorsAdded: 1, totalTensorsSnapshot: 1157, …} + 3: {name: "Add", bytesAdded: 4, totalBytesSnapshot: 44788748, tensorsAdded: 1, totalTensorsSnapshot: 1158, …} + 4: {name: "Add", bytesAdded: 4, totalBytesSnapshot: 44788748, tensorsAdded: 1, totalTensorsSnapshot: 1158, …} + } +```