# Data generator Fills a MongoDB database with analytics data that follows the data model contract of `@openvidu-dashboard/typings`: a deterministic scenario S, M or L over a number of days ending now. The backend integration tests use it for their query plan and latency checks, and `./dashboard.sh demo` uses it for the local demo database. Before inserting, `src/schema.ts` creates every collection, index and TTL of the contract and gives the plain collections the contract validator with `validationAction: error`, so a document that drifts from the contract fails the run (`track_stats` takes no validator as a time-series collection). `src/contract.ts` turns each model document into its stored shape: the model's bookkeeping removed, absent fields omitted and every number in the BSON type its schema declares. The package is also a library for the backend integration tests (`dashboard-ce/backend/test/`): `./generator` (`generate(db, options)`), `./context` (`queryContext(db)`: the data range and sample identifiers), `./plan` (explain summaries used by the `expectIndexBacked` harness) and `./schema` (`createCollections()`). Requirements: the workspace installed (`./dashboard.sh install`), a reachable MongoDB (8.x; time-series collections are used when the server accepts them) and a user allowed to create and drop the target database. Run the commands from this directory. ## Usage ```sh pnpm generate --uri "$MONGO_URI" --db ov_data_s --scenario S --days 30 --drop pnpm generate --uri "$MONGO_URI" --db ov_data_m --scenario M --days 7 --drop pnpm generate --uri "$MONGO_URI" --db ov_data_l --scenario L --days 1 --drop --stats-mode track ``` | Option | Default | Meaning | | ----------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------- | | `--uri`, `--db` | required | Connection string and target database | | `--scenario S\|M\|L` | required | Rooms per day, sessions per day and average room size (table below) | | `--days ` | required | Days of history ending now | | `--stats-mode participant\|track` | `participant` | One `track_stats` document per participant and direction, or per track | | `--seed ` | `1` | Seed of the pseudo-random generator (mulberry32); same seed, same data | | `--retention-events-days ` | `7` | Only raw events inside this window are written (TTL index on `ts`) | | `--retention-stats-days ` | `7` | Only stats inside this window are written (`expireAfterSeconds`) | | `--retention-tracks-days ` | `7` | `expiresAt` of tracks and subscriptions; expired ones are not written | | `--retention-entities-days ` | `90` | `expiresAt = closedAt + retention` on closed entities | | `--retention-rollups-5min-days ` | `7` | Only 5-minute rollups inside this window are written (TTL index on `bucket`) | | `--drop` | off | Drop the database first | | `--plain-stats` | off | Force the plain-collection fallback for `track_stats` | | `--five-minute-rollups` | off | Write `analytics_5min` inside the 5-minute rollup window | | `--node-rollups` | off | Write the `node:` scope beside `global` in every rollup | | `--room-sums` | off | Write `rooms.participantsSum` and `rooms.durationMsSum` at room end | | `--active-window ` | off | Rooms created in the last minutes stay active (two of three); without it, the most recent 10 % of the rooms are active | | `--min-active-rooms ` | 0 | With `--active-window`: rooms added to the window when the hour of day leaves fewer than `n` active rooms | | `--nodes-without-media-stats ` | 0 | The last `n` of the three nodes report no media statistics: `mediaStats: false`, no media `track_stats`, bytes or quality | | Scenario | Rooms per day | Sessions per day | Average room size | | -------- | ------------- | ---------------- | ----------------- | | S | 200 | 1 000 | 4 | | M | 2 000 | 20 000 | 6 | | L | 5 000 | 50 000 | 8 | What it writes (`analytics_5min` stays empty without `--five-minute-rollups`): - `rooms`: created over the days with a diurnal curve (office-hours peak); the most recent 10 % are `active`, the rest `closed` with `endReason` and `closedBy`. Id arrays and counters (`participantSids`, `activeParticipantSids`, `identities`, `participants.*`, `trackSids`, `egressIds`, `ingressSessionIds`, `features`) are derived from the generated entities; participant counters count STANDARD sessions and `byKind` every kind, as the writer does (also in the rollups). - `participants`: unique identities per room follow a Poisson distribution around the scenario average; the difference between sessions/day and rooms/day × average size is produced as reconnects (`DUPLICATE_IDENTITY` leave plus a new session of the same identity). Sessions last ~20 min, 3 % are aborted, 95 % are `STANDARD`; rooms with egress/ingress/agent/SIP get the matching participant kinds. Client (sdk/os/browser/device/network), connection type, connect time, disconnect reason and resumes follow fixed distributions. - `tracks` (CAMERA video + MICROPHONE audio per publishing participant, with simulcast layers and final publish stats) and `track_subscriptions` (every participant subscribes to the tracks of the other identities present at the same time; 1 % failed). - `egresses` (10 % of rooms; mostly `room_composite`; `complete`/`failed`/`aborted`, `active` in active rooms), `ingresses` and `ingress_sessions` (2 % of rooms). - `events`: lifecycle events of rooms, participants, tracks, subscriptions, egresses and ingresses with small payloads, only inside the events retention window. - `track_stats`: one document per 30 s per participant and direction (`participant` mode) or per track and direction (`track` mode), only inside the stats retention window, with plausible score/bytes/rtt/jitter. Time-series collection (`timeField: ts`, `metaField: meta`, `granularity: seconds`) with `expireAfterSeconds`; plain collection with a TTL index on `ts` when the server rejects time-series. - `analytics_hourly` and `analytics_daily` (global scope, plus one scope per node with `--node-rollups`) aggregated from the generated entities; bytes and quality are spread over the hours each session was connected. `--room-sums` adds the room size and duration sums; `--five-minute-rollups` adds `analytics_5min` (lifecycle counters, the `participants.connected` gauge, bytes and quality per five minutes) inside the 5-minute rollup window. `analytics_daily` gets no TTL index: daily rollups are kept forever. `generate()` takes the same switches as `rollups: { fiveMinute, nodes, roomSums }`. - `nodes` (3 nodes) and `schema_meta` (`schema`, `retention`, `capabilities`, `config`). Documents are inserted in batches of 5 000 with `insertMany(ordered: false)`. The output shows progress and a final summary with documents per collection, elapsed time and inserts per second. The generator keeps only one room in memory at a time, so scenario L fits in a few hundred MB of heap. ## Files | File | Role | | ------------------ | -------------------------------------------------------------------------------------------- | | `src/schema.ts` | Collections, indexes and validators of the contract, `createCollections()` with the fallback | | `src/contract.ts` | Stored shape of a model document: bookkeeping removed, nulls omitted, BSON numeric types | | `src/model.ts` | Scenarios, document types and the per-room entity generator | | `src/derive.ts` | Raw events, stats samples and rollups derived from a generated room | | `src/generator.ts` | `generate(db, options)` and the batched writer | | `src/generate.ts` | Generator CLI | | `src/context.ts` | `queryContext(db)`: data range and sample identifiers for query plan tests | | `src/plan.ts` | Explain summaries: plan stages, indexes, documents examined | | `src/random.ts` | mulberry32 generator and sampling helpers | | `src/args.ts` | CLI argument parsing |