add notes on building custom modules

master
Vladimir Mandic 2021-07-31 20:39:36 -04:00
parent de0142bcee
commit bffa2cb040
2 changed files with 75 additions and 0 deletions

@ -53,6 +53,7 @@ Check out [**Live Demo**](https://vladmandic.github.io/human/demo/index.html) fo
- [**Notes on Backends**](https://github.com/vladmandic/human/wiki/Backends)
- [**Development Server**](https://github.com/vladmandic/human/wiki/Development-Server)
- [**Build Process**](https://github.com/vladmandic/human/wiki/Build-Process)
- [**Adding Custom Modules**](https://github.com/vladmandic/human/wiki/Module)
- [**Performance Notes**](https://github.com/vladmandic/human/wiki/Performance)
- [**Performance Profiling**](https://github.com/vladmandic/human/wiki/Profiling)
- [**Platform Support**](https://github.com/vladmandic/human/wiki/Platforms)

74
Module.md Normal file

@ -0,0 +1,74 @@
# Custom Modules
## Overview
Each module is fully self enclosed:
- Defined in a separate folder under `/src/`
- Performs `load` and `predict` functions
- Uses global configuration object
- Runs prediction on global input image
- Merges results into global results object
<br>
## Define module
Define module that implements `load` and `predict` async methods:
```js
export async function load(config: Config | any) { ... }
```
- loads specific model using `modelPath` configuration
- returns `tf.GraphModel`
```js
export async function predict(image: Tensor, config: Config, idx, count) { ... }
```
- input image is already preprocessed and passed to predict method as tensor
- takes input tensor and resizes and normalizes it as needed
- optionally implements `skipFrames` for caching of results
- optionally uses `idx` and `count` params to map cached objects to current objects
- uses any other optional configuration params
- returns result object
<br>
## Enable Module
in `human.ts`:
- define model placeholder (`module.load()` stores model here) in the main constructor
in `config.ts`:
- define configuration types
- set configuration defaul values
in `result.ts`:
- define results type
in `models.ts`:
- enable model loading sync and async by calling `module.load()`
<br>
## Execute Module
if model works on full image, execute from `human.ts`
if model works on face image, execute from `face.ts`
- add results placeholder
- implement calls to `module.predict()` for sync and async
- merge results object
## Examples
- For simple module that works on pre-detected face tensor,
follow existing module defintion is `emotion/emotion.ts`
- For simple module that works on a full image tensor,
follow existing module definition in `object/centernet.ts`