# Data generator

Fills a MongoDB database with analytics data that follows the data model contract of `@openvidu-dashboard/typings`: a
deterministic scenario S, M or L over a number of days ending now. The backend integration tests use it for their query
plan and latency checks, and `./dashboard.sh demo` uses it for the local demo database.

Before inserting, `src/schema.ts` creates every collection, index and TTL of the contract and gives the plain
collections the contract validator with `validationAction: error`, so a document that drifts from the contract fails
the run (`track_stats` takes no validator as a time-series collection). `src/contract.ts` turns each model document
into its stored shape: the model's bookkeeping removed, absent fields omitted and every number in the BSON type its
schema declares.

The package is also a library for the backend integration tests (`dashboard-ce/backend/test/`): `./generator`
(`generate(db, options)`), `./context` (`queryContext(db)`: the data range and sample identifiers), `./plan` (explain
summaries used by the `expectIndexBacked` harness) and `./schema` (`createCollections()`).

Requirements: the workspace installed (`./dashboard.sh install`), a reachable MongoDB (8.x; time-series
collections are used when the server accepts them) and a user allowed to create and drop the target
database. Run the commands from this directory.

## Usage

```sh
pnpm generate --uri "$MONGO_URI" --db ov_data_s --scenario S --days 30 --drop
pnpm generate --uri "$MONGO_URI" --db ov_data_m --scenario M --days 7 --drop
pnpm generate --uri "$MONGO_URI" --db ov_data_l --scenario L --days 1 --drop --stats-mode track
```

| Option                              | Default       | Meaning                                                                                                                   |
| ----------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `--uri`, `--db`                     | required      | Connection string and target database                                                                                     |
| `--scenario S\|M\|L`                | required      | Rooms per day, sessions per day and average room size (table below)                                                       |
| `--days <n>`                        | required      | Days of history ending now                                                                                                |
| `--stats-mode participant\|track`   | `participant` | One `track_stats` document per participant and direction, or per track                                                    |
| `--seed <n>`                        | `1`           | Seed of the pseudo-random generator (mulberry32); same seed, same data                                                    |
| `--retention-events-days <n>`       | `7`           | Only raw events inside this window are written (TTL index on `ts`)                                                        |
| `--retention-stats-days <n>`        | `7`           | Only stats inside this window are written (`expireAfterSeconds`)                                                          |
| `--retention-tracks-days <n>`       | `7`           | `expiresAt` of tracks and subscriptions; expired ones are not written                                                     |
| `--retention-entities-days <n>`     | `90`          | `expiresAt = closedAt + retention` on closed entities                                                                     |
| `--retention-rollups-5min-days <n>` | `7`           | Only 5-minute rollups inside this window are written (TTL index on `bucket`)                                              |
| `--drop`                            | off           | Drop the database first                                                                                                   |
| `--plain-stats`                     | off           | Force the plain-collection fallback for `track_stats`                                                                     |
| `--five-minute-rollups`             | off           | Write `analytics_5min` inside the 5-minute rollup window                                                                  |
| `--node-rollups`                    | off           | With `--media-nodes`: write the `node:<label>` scope beside `global` in every rollup                                      |
| `--room-sums`                       | off           | Write `rooms.participantsSum` and `rooms.durationMsSum` at room end                                                       |
| `--active-window <minutes>`         | off           | Rooms created in the last minutes stay active (two of three); without it, the most recent 10 % of the rooms are active    |
| `--min-active-rooms <n>`            | 0             | With `--active-window`: rooms added to the window when the hour of day leaves fewer than `n` active rooms                 |
| `--nodes-without-media-stats <n>`   | 0             | The last `n` of the three nodes report no media statistics: `mediaStats: false`, no media `track_stats`, bytes or quality |
| `--media-nodes`                     | off           | Label the nodes `media-node-10-0-0-1` to `-3` and add an earlier stopped process of the first one (a restart)             |

| Scenario | Rooms per day | Sessions per day | Average room size |
| -------- | ------------- | ---------------- | ----------------- |
| S        | 200           | 1 000            | 4                 |
| M        | 2 000         | 20 000           | 6                 |
| L        | 5 000         | 50 000           | 8                 |

What it writes (`analytics_5min` stays empty without `--five-minute-rollups`):

- `rooms`: created over the days with a diurnal curve (office-hours peak); the most recent 10 % are
  `active`, the rest `closed` with `endReason` and `closedBy`. Id arrays and counters (`participantSids`,
  `activeParticipantSids`, `identities`, `participants.*`, `trackSids`, `egressIds`, `ingressSessionIds`,
  `features`) are derived from the generated entities; participant counters count STANDARD sessions and `byKind`
  every kind, as the writer does (also in the rollups).
- `participants`: unique identities per room follow a Poisson distribution around the scenario average;
  the difference between sessions/day and rooms/day × average size is produced as reconnects
  (`DUPLICATE_IDENTITY` leave plus a new session of the same identity). Sessions last ~20 min, 3 % are
  aborted, 95 % are `STANDARD`; rooms with egress/ingress/agent/SIP get the matching participant kinds.
  Client (sdk/os/browser/device/network), connection type, connect time, disconnect reason and resumes
  follow fixed distributions.
- `tracks` (CAMERA video + MICROPHONE audio per publishing participant, with simulcast layers and final
  publish stats) and `track_subscriptions` (every participant subscribes to the tracks of the other
  identities present at the same time; 1 % failed).
- `egresses` (10 % of rooms; mostly `room_composite`; `complete`/`failed`/`aborted`, `active` in active
  rooms), `ingresses` and `ingress_sessions` (2 % of rooms).
- `events`: lifecycle events of rooms, participants, tracks, subscriptions, egresses and ingresses with
  small payloads, only inside the events retention window.
- `track_stats`: one document per 30 s per participant and direction (`participant` mode) or per track and
  direction (`track` mode), only inside the stats retention window, with plausible score/bytes/rtt/jitter.
  Time-series collection (`timeField: ts`, `metaField: meta`, `granularity: seconds`) with
  `expireAfterSeconds`; plain collection with a TTL index on `ts` when the server rejects time-series.
- `analytics_hourly` and `analytics_daily` (global scope, plus one scope per media node label with
  `--media-nodes --node-rollups`) aggregated from the generated entities; bytes and quality are spread over the
  hours each session was connected. `--room-sums` adds the room size and duration sums; `--five-minute-rollups`
  adds `analytics_5min` (lifecycle counters, the `participants.connected` gauge, bytes and quality per five minutes)
  inside the 5-minute rollup window. `analytics_daily` gets no TTL index: daily rollups are kept forever.
  `generate()` takes the same switches as `rollups: { fiveMinute, nodes, roomSums }` and `mediaNodes`.
- `nodes` (3 nodes, plus the stopped earlier process of the first one with `--media-nodes`) and `schema_meta`
  (`schema`, `retention`, `capabilities`, `config`).

Documents are inserted in batches of 5 000 with `insertMany(ordered: false)`. The output shows progress
and a final summary with documents per collection, elapsed time and inserts per second. The generator
keeps only one room in memory at a time, so scenario L fits in a few hundred MB of heap.

## Files

| File               | Role                                                                                         |
| ------------------ | -------------------------------------------------------------------------------------------- |
| `src/schema.ts`    | Collections, indexes and validators of the contract, `createCollections()` with the fallback |
| `src/contract.ts`  | Stored shape of a model document: bookkeeping removed, nulls omitted, BSON numeric types     |
| `src/model.ts`     | Scenarios, document types and the per-room entity generator                                  |
| `src/derive.ts`    | Raw events, stats samples and rollups derived from a generated room                          |
| `src/generator.ts` | `generate(db, options)` and the batched writer                                               |
| `src/generate.ts`  | Generator CLI                                                                                |
| `src/context.ts`   | `queryContext(db)`: data range and sample identifiers for query plan tests                   |
| `src/plan.ts`      | Explain summaries: plan stages, indexes, documents examined                                  |
| `src/random.ts`    | mulberry32 generator and sampling helpers                                                    |
| `src/args.ts`      | CLI argument parsing                                                                         |