openvidu-deployment: optimize Azure HA deployment time

Parallel master VM creation (drop chained dependsOn), publish
FINISH-MASTER-NODE from the first healthy master after its health gate
instead of unconditionally from master 4 (removes the SPOF and the fixed
sleep 150), overlap the heavy media node install with the masters (gate 1:
secrets + 4 IPs before installing; gate 2: FINISH before service start,
5s polling instead of 1s), bounded timeouts on every Key Vault poll,
Key Vault/storage wait margins 100s -> 300s, robust installer fetch
(curl --retry to file instead of sh <(curl)), apt cleanup.

Fix createUiDefinition.json: the portal 'Container Name' field emitted
'containerName' but the template parameter is 'appDataContainerName',
so the user-provided value never reached the template.

ARM JSON recompiled with bicep 0.45.15 (committed JSON was 0.44.1;
codegen noise verified to be metadata-only against the pristine bicep).

Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS.
deploy 2m11s, ready 10m21s, destroy 7m14s.
master
Piwccle 2026-07-24 14:31:46 +02:00
parent 0a587eb33e
commit 196b3f767f
4 changed files with 282 additions and 45 deletions

View File

@ -0,0 +1,181 @@
# Documentation changes for `ha-optimizations` (Azure HA deployment)
Instructions for the Claude working on the docs repo
`/home/sergio/Escritorio/openvidu/openvidu.io` (branch `next`, already up to date).
These instructions derive from the changes applied in
`openvidu-deployment/pro/ha/azure/cf-openvidu-ha.bicep`,
`openvidu-deployment/pro/ha/azure/cf-openvidu-ha.json` (recompiled) and
`openvidu-deployment/pro/ha/azure/createUiDefinition.json` on the `ha-optimizations`
branch. Functional summary of what changed in the deployment:
- The 4 master VMs are now created in parallel (the `dependsOn` chain
MasterNode2→1, 3→2, 4→3 was removed; coordination stays data-driven via Key Vault).
- Media Nodes now install (Docker + image pulls) **in parallel** with the masters,
instead of waiting idle until all masters finished before starting to install.
- The master-node-4 single point of failure was removed: readiness
(`FINISH-MASTER-NODE`) is now published by the **first healthy master**, right after
its own health check, and a fixed `sleep 150` was removed.
- Wait timeouts hardened (Key Vault availability and the blob-storage config retry
raised from 100s to 300s; the previously unbounded polls are now bounded to 30 min),
and the installer download is now retried and validated (`curl --retry 8 ... -o file`
+ non-empty check) instead of the fragile `sh <(curl ...)`.
- Bug fix in `createUiDefinition.json`: the portal now maps the "Container Name" field
to the correct template parameter (`appDataContainerName`) — see section (b).
SUMMARY: the ONLY content change required in the docs is the two deployment time
figures in Azure HA's `install.md`. Everything else (public parameters, screenshots,
the "Container Name" field description, other clouds, other deployment types) stays the
same. Read this file in full before touching anything.
---
## (a) Time figures to update — MANDATORY
File: `docs/docs/self-hosting/ha/azure/install.md`
There are TWO deployment-time figures in this file. Locate each by its literal text
(line numbers below are approximate and may have shifted). The deployment is now
faster (masters run in parallel and the media install overlaps with the masters'
install, plus the fixed 150s sleep is gone), so both figures must be re-measured.
### Occurrence 1 (section "## Deploying the stack", ~line 97)
Current literal text:
```
Whenever you are satisfied with your Template parameters, just click on _"Next"_ to trigger the validation process. If correct, click on _"Create"_ to start the deployment process (which will take about 10 to 15 minutes).
```
Proposed text (fill in `X`/`Y` with the measurement from `ov-cloud-tester`):
```
Whenever you are satisfied with your Template parameters, just click on _"Next"_ to trigger the validation process. If correct, click on _"Create"_ to start the deployment process (which will take about X to Y minutes).
```
### Occurrence 2 (section "## Configuration and administration", ~line 181)
Current literal text:
```
When your Azure stack reaches the **`Succeeded`** status, it means that all resources have been created. You will need to wait about 5 to 10 minutes for the instance to install OpenVidu. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
```
Proposed text (fill in `X`/`Y` with the measurement from `ov-cloud-tester`):
```
When your Azure stack reaches the **`Succeeded`** status, it means that all resources have been created. You will need to wait about X to Y minutes for the instance to install OpenVidu. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
```
> Placeholder to fill in: `X to Y minutes — fill in with the measurement from ov-cloud-tester`.
> The two figures are independent (occurrence 1 = full stack creation time; occurrence 2
> = extra wait after `Succeeded` for OpenVidu to install). Measure and fill each on its own.
### Do NOT touch the same figures in other files
- `"10 to 15 minutes"` appears ONLY in `docs/docs/self-hosting/ha/azure/install.md`, so
there is no ambiguity for occurrence 1.
- `"5 to 10 minutes"` also appears in OTHER documents that are OUT OF SCOPE here (they
belong to other deployment types / clouds). Do NOT modify them:
- `docs/docs/self-hosting/single-node-pro/azure/install.md`
- `docs/docs/self-hosting/single-node/azure/install.md`
- `docs/docs/self-hosting/single-node-pro/gcp/install.md`
- `docs/docs/self-hosting/single-node/gcp/install.md`
- `shared/self-hosting/gcp/deploying-stack.md`
Only change the occurrence inside `docs/docs/self-hosting/ha/azure/install.md`.
---
## (b) "Container Name" field fix — NO doc change needed
The bug: in `createUiDefinition.json` the portal emitted the output key `containerName`,
but the ARM/Bicep template parameter is `appDataContainerName`. Because the key did not
match, the value typed by the user in the "Container Name" field never reached the
template — a custom container name was silently ignored and the deployment always fell
back to the default `openvidu-appdata`. The fix renames only that output **key** to
`appDataContainerName` (the value still comes from the same UI field
`steps('parameters STORAGE').containerName`).
Impact on the docs: **NONE.** Explicit reasoning so you can confirm and move on:
- The visible UI element name and its label are unchanged — the form still shows a field
labeled **"Container Name"**. Only a hidden output-mapping key changed.
- The shared snippet `shared/self-hosting/azure/storage-account.md` (included by the
Azure HA `install.md` via `--8<--`) describes that field as:
*"**Container Name** is the name that you desire for the container ... If you leave it
blank it will create the container with name `openvidu-appdata`."* That description
now matches reality in BOTH cases (blank → `openvidu-appdata`, as before; non-blank →
the value is finally honored). The text was already written as if the field worked, so
no wording needs to change.
- The screenshot `assets/images/platform/self-hosting/shared/azure/storageaccount.png`
(referenced only from `shared/self-hosting/azure/storage-account.md`) shows the same
unchanged form, so it does NOT need to be regenerated.
Action: leave `shared/self-hosting/azure/storage-account.md` and its screenshot as they
are.
---
## (c) Public template parameters / outputs: NO CHANGES — confirmation
Do not touch parameter tables or screenshots in the parameters section.
Reason: no public parameter was added, removed or renamed. `appDataContainerName` and
`storageAccountName` already existed as template parameters; the createUiDefinition fix
only corrects an existing field mapping. The Bicep template has no `outputs` section, and
none was added. Therefore:
- Any "Parameters" tables/screenshots in `docs/docs/self-hosting/ha/azure/install.md`
and in the shared Azure snippets **stay the same**.
- There are no new fields the user must fill in the Azure portal form.
- The parameter form screenshots **do not change**.
---
## (d) Other statements in the Azure HA docs that may become outdated
The Azure HA documentation (`docs/docs/self-hosting/ha/azure/*.md`) and the shared Azure
includes (`shared/self-hosting/azure/*.md`) were reviewed against the internal changes.
Conclusions:
### d.1 — Master node ordering / parallelization: not documented
The docs never state that master nodes are created sequentially or "one by one", nor do
they describe the `dependsOn` chain. Removing it changes nothing user-facing. No text to
update.
### d.2 — `FINISH-MASTER-NODE` / master-4 SPOF / scale-in lock (`lock.txt`): not documented
The internal readiness handshake (`FINISH-MASTER-NODE`), the fact that it used to be
written only by master-4, and the `automation-locks/lock.txt` scale-in lock blob are
NOT mentioned anywhere in the docs. The scale-in behavior described in
`shared/self-hosting/azure/scale-in-config.md` (the graceful drain and its "up to 5
minutes" / Azure "15 minutes" figures) is about Azure's termination timing and is
UNCHANGED by this work — **do not touch it**.
### d.3 — Internal coordination (`ALL-SECRETS-GENERATED`, Key Vault polling): not documented
The master-1-as-leader secrets generation, the Key Vault polling, and the new bounded
timeouts are internal and not publicly documented. Nothing user-facing changes. No text
to update.
### d.4 — `admin.md`, `upgrade.md`, `index.md` (Azure HA): DO NOT TOUCH
Reviewed; none of them reference the internals that changed. No changes required.
---
## Implementation checklist
- [ ] In `docs/docs/self-hosting/ha/azure/install.md`, replace `"10 to 15 minutes"`
(~line 97) and `"5 to 10 minutes"` (~line 181) with the re-measured
`"X to Y minutes"` figures, located by their literal text.
- [ ] Measure the actual deployment times with `ov-cloud-tester` on `ha-optimizations`
and fill in `X`/`Y` for each occurrence independently.
- [ ] Confirm NO parameter tables or screenshots were touched (section c).
- [ ] Confirm `shared/self-hosting/azure/storage-account.md` and `storageaccount.png`
were left unchanged (section b).
- [ ] Confirm the `"5 to 10 minutes"` figures in single-node / single-node-pro / GCP
docs and `shared/self-hosting/gcp/deploying-stack.md` were NOT touched (section a).
- [ ] Confirm `scale-in-config.md`, `admin.md`, `upgrade.md`, `index.md` were NOT touched
(section d).

View File

@ -304,7 +304,7 @@ apt-get update && apt-get install -y \
# Wait for the keyvault availability
MAX_WAIT=100
MAX_WAIT=300
WAIT_INTERVAL=1
ELAPSED_TIME=0
set +e
@ -397,6 +397,10 @@ if [[ $MASTER_NODE_NUM -eq 1 ]] && [[ "$ALL_SECRETS_GENERATED" == "" || "$ALL_SE
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL-SECRETS-GENERATED "true")"
fi
# Wait (bounded: 360 x 5s = 30 min) until all 4 master nodes have published their private IPs
IP_WAIT_MAX_RETRIES=360
IP_WAIT_INTERVAL=5
IP_WAIT_RETRIES=0
while true; do
MASTER_NODE_1_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-1-PRIVATE-IP --query value -o tsv) || true
MASTER_NODE_2_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-2-PRIVATE-IP --query value -o tsv) || true
@ -409,11 +413,26 @@ while true; do
[[ "$MASTER_NODE_4_PRIVATE_IP" != "" ]]; then
break
fi
sleep 5
IP_WAIT_RETRIES=$((IP_WAIT_RETRIES + 1))
if [ $IP_WAIT_RETRIES -ge $IP_WAIT_MAX_RETRIES ]; then
echo "[OpenVidu] timed out after 30 min waiting for the 4 master nodes to publish their private IPs"
exit 1
fi
sleep $IP_WAIT_INTERVAL
done
# Wait until master-node-1 has generated all shared secrets before fetching them
while [[ "$(az keyvault secret show --vault-name ${keyVaultName} --name ALL-SECRETS-GENERATED --query value -o tsv 2>/dev/null)" != "true" ]]; do sleep 5; done
# Wait (bounded: 360 x 5s = 30 min) until master-node-1 has generated all shared secrets before fetching them
SECRETS_WAIT_MAX_RETRIES=360
SECRETS_WAIT_INTERVAL=5
SECRETS_WAIT_RETRIES=0
while [[ "$(az keyvault secret show --vault-name ${keyVaultName} --name ALL-SECRETS-GENERATED --query value -o tsv 2>/dev/null)" != "true" ]]; do
SECRETS_WAIT_RETRIES=$((SECRETS_WAIT_RETRIES + 1))
if [ $SECRETS_WAIT_RETRIES -ge $SECRETS_WAIT_MAX_RETRIES ]; then
echo "[OpenVidu] timed out after 30 min waiting for ALL-SECRETS-GENERATED to become true"
exit 1
fi
sleep $SECRETS_WAIT_INTERVAL
done
# Fetch the values in the keyvault
@ -448,8 +467,16 @@ fi
ENABLED_MODULES=$(az keyvault secret show --vault-name ${keyVaultName} --name ENABLED-MODULES --query value -o tsv)
# Download the installer script (retry transient curl failures; fail if empty/missing)
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh"
if [ ! -s "$INSTALLER_SCRIPT" ]; then
echo "[OpenVidu] failed to download the master node installer script"
exit 1
fi
# Base command
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)"
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
# Common arguments
COMMON_ARGS=(
@ -811,7 +838,7 @@ INSTALL_DIR="/opt/openvidu"
CLUSTER_CONFIG_DIR="${INSTALL_DIR}/config/cluster"
# Retry login + storage key fetch to allow the Contributor role assignment to propagate
MAX_WAIT=100
MAX_WAIT=300
WAIT_INTERVAL=1
ELAPSED_TIME=0
set +e
@ -1043,8 +1070,6 @@ az login --identity --allow-no-subscriptions
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
apt-get update && apt-get install -y
export HOME="/root"
# Install OpenVidu
@ -1062,20 +1087,21 @@ systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1;
# Launch on reboot
echo "@reboot /usr/local/bin/restart.sh >> /var/log/openvidu-restart.log" 2>&1 | crontab
# Wait until this master node is healthy (check_app_ready.sh caps at 1200s).
# On failure exit WITHOUT publishing readiness, so media nodes keep waiting for another master.
/usr/local/bin/check_app_ready.sh || { echo "[OpenVidu] master node did not become healthy"; exit 1; }
# Publish readiness so media nodes can start. Every healthy master writes the same value, so the
# first healthy master unblocks media and a failed master can no longer leave this unset (no SPOF).
az keyvault secret set --vault-name ${keyVaultName} --name FINISH-MASTER-NODE --value "true"
MASTER_NODE_NUM=${masterNodeNum}
if [[ $MASTER_NODE_NUM -eq 4 ]]; then
# Creating scale in lock
set +e
az storage blob upload --account-name ${storageAccountName} --container-name automation-locks --name lock.txt --file /dev/null --auth-mode key
set -e
#Finish all the nodes
az keyvault secret set --vault-name ${keyVaultName} --name FINISH-MASTER-NODE --value "true"
fi
# Wait for the app
sleep 150
/usr/local/bin/check_app_ready.sh
'''
var userDataMasterNode1 = reduce(
@ -1168,7 +1194,6 @@ resource openviduMasterNode2 'Microsoft.Compute/virtualMachines@2023-09-01' = {
}
userData: base64(userDataMasterNode2)
}
dependsOn: [openviduMasterNode1]
}
resource openviduMasterNode3 'Microsoft.Compute/virtualMachines@2023-09-01' = {
@ -1203,7 +1228,6 @@ resource openviduMasterNode3 'Microsoft.Compute/virtualMachines@2023-09-01' = {
}
userData: base64(userDataMasterNode3)
}
dependsOn: [openviduMasterNode2]
}
resource openviduMasterNode4 'Microsoft.Compute/virtualMachines@2023-09-01' = {
@ -1238,7 +1262,6 @@ resource openviduMasterNode4 'Microsoft.Compute/virtualMachines@2023-09-01' = {
}
userData: base64(userDataMasterNode4)
}
dependsOn: [openviduMasterNode3]
}
/*------------------------------------------- MEDIA NODES -------------------------------------------*/
@ -1272,23 +1295,31 @@ apt-get update && apt-get install -y \
# Get own private IP
PRIVATE_IP=$(curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance/network/interface/0/ipv4/ipAddress/0/privateIpAddress?api-version=2017-08-01&format=text")
WAIT_INTERVAL=1
MAX_WAIT=10000
ELAPSED_TIME=0
# Gate 1: wait (bounded: 360 x 5s = 30 min) until master nodes generated the shared secrets and
# published their 4 private IPs. This lets the heavy media installer run in parallel with the masters.
WAIT_INTERVAL=5
MAX_RETRIES=360
RETRIES=0
set +e
while true; do
# get secret value
FINISH_MASTER_NODE=$(az keyvault secret show --vault-name ${keyVaultName} --name FINISH-MASTER-NODE --query value -o tsv)
ALL_SECRETS_GENERATED=$(az keyvault secret show --vault-name ${keyVaultName} --name ALL-SECRETS-GENERATED --query value -o tsv 2>/dev/null)
MASTER_NODE_1_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-1-PRIVATE-IP --query value -o tsv 2>/dev/null)
MASTER_NODE_2_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-2-PRIVATE-IP --query value -o tsv 2>/dev/null)
MASTER_NODE_3_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-3-PRIVATE-IP --query value -o tsv 2>/dev/null)
MASTER_NODE_4_PRIVATE_IP=$(az keyvault secret show --vault-name ${keyVaultName} --name MASTER-NODE-4-PRIVATE-IP --query value -o tsv 2>/dev/null)
# Check if all master nodes finished
if [ "$FINISH_MASTER_NODE" == "true" ]; then
# Break once secrets are generated and all 4 master IPs are published
if [ "$ALL_SECRETS_GENERATED" == "true" ] &&
[ "$MASTER_NODE_1_PRIVATE_IP" != "" ] &&
[ "$MASTER_NODE_2_PRIVATE_IP" != "" ] &&
[ "$MASTER_NODE_3_PRIVATE_IP" != "" ] &&
[ "$MASTER_NODE_4_PRIVATE_IP" != "" ]; then
break
fi
ELAPSED_TIME=$((ELAPSED_TIME + WAIT_INTERVAL))
# Check if the maximum waiting time has been reached
if [ $ELAPSED_TIME -ge $MAX_WAIT ]; then
RETRIES=$((RETRIES + 1))
if [ $RETRIES -ge $MAX_RETRIES ]; then
echo "[OpenVidu] timed out after 30 min waiting for master nodes to generate secrets and publish their IPs"
exit 1
fi
@ -1305,8 +1336,16 @@ REDIS_PASSWORD=$(az keyvault secret show --vault-name ${keyVaultName} --name RED
ENABLED_MODULES=$(az keyvault secret show --vault-name ${keyVaultName} --name ENABLED-MODULES --query value -o tsv)
OPENVIDU_VERSION=$(az keyvault secret show --vault-name ${keyVaultName} --name OPENVIDU-VERSION --query value -o tsv)
# Download the installer script (retry transient curl failures; fail if empty/missing)
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh"
if [ ! -s "$INSTALLER_SCRIPT" ]; then
echo "[OpenVidu] failed to download the media node installer script"
exit 1
fi
# Base command
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)"
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
# Common arguments
COMMON_ARGS=(
@ -1413,8 +1452,7 @@ chmod +x /usr/local/bin/delete_media_node.sh
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
apt-get update && apt-get install -y
apt-get install -y jq
apt-get update && apt-get install -y jq
# Install azure cli
AZURE_CLI_VERSION=2.87.0
@ -1435,9 +1473,29 @@ az vmss update --resource-group $RESOURCE_GROUP_NAME --name $VM_SCALE_SET_NAME -
export HOME="/root"
# Install OpenVidu
# Install OpenVidu (heavy: docker + image pulls). Overlaps with the masters' own installs.
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/delete_media_node.sh; }
# Gate 2: wait (bounded: 360 x 5s = 30 min) for the master nodes to publish readiness before starting
WAIT_INTERVAL=5
MAX_RETRIES=360
RETRIES=0
set +e
while true; do
FINISH_MASTER_NODE=$(az keyvault secret show --vault-name ${keyVaultName} --name FINISH-MASTER-NODE --query value -o tsv 2>/dev/null)
if [ "$FINISH_MASTER_NODE" == "true" ]; then
break
fi
RETRIES=$((RETRIES + 1))
if [ $RETRIES -ge $MAX_RETRIES ]; then
echo "[OpenVidu] timed out after 30 min waiting for FINISH-MASTER-NODE"
/usr/local/bin/delete_media_node.sh
exit 1
fi
sleep $WAIT_INTERVAL
done
set -e
# Start OpenVidu
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/delete_media_node.sh; }
'''
@ -1470,6 +1528,7 @@ var userDataParamsMedia = {
base64delete_mediaNode: base64delete_mediaNode_ScriptMedia
resourceGroupName: resourceGroup().name
vmScaleSetName: '${stackName}-mediaNodeScaleSet'
keyVaultName: keyVaultName
}
var userDataMediaNode = reduce(

File diff suppressed because one or more lines are too long

View File

@ -553,7 +553,7 @@
"datetime": "[steps('parameters SCALING').datetime]",
"automationAccountName": "[steps('parameters SCALING').automationAccountName]",
"storageAccountName": "[steps('parameters STORAGE').storageAccountName]",
"containerName": "[steps('parameters STORAGE').containerName]",
"appDataContainerName": "[steps('parameters STORAGE').containerName]",
"additionalInstallFlags": "[steps('FLAGS').additionalInstallFlags]"
}
}