From 4d6b7acd7cc5e17090316c9b6c09241ed4e7d5dc Mon Sep 17 00:00:00 2001 From: Piwccle Date: Fri, 24 Jul 2026 17:11:58 +0200 Subject: [PATCH] openvidu-deployment: optimize AWS HA deployment time Parallel master creation: drop the MasterNodeN -> WaitConditionN-1 DependsOn chain. Each master gets a fixed MASTER_NODE_NUM via UserData (replaces subnet-matching detection) and publishes its private IP to its own SSM parameter (atomic per-key writes; the 4 parameters are stack resources), removing the IP exchange through the shared Secrets Manager JSON and its lost-update race. after_install.sh is now leader-only, so the shared JSON keeps a single writer with parallel masters. Ungate media nodes and the NLB from WaitCondition4: media wait for secrets + the 4 SSM IPs (bounded polls) before the heavy install, and for a healthy master (:7880/health/caddy) before starting the service. Bounded timeout on the previously infinite IP wait, robust installer fetch (curl --retry to file instead of sh <(curl)), SSM read/write IAM statements, MEET_INITIAL_API_KEY dead-code fix, SubnetProcessor Lambda cleanup (unused boto3 client and ec2:DescribeSubnets permission). Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS. deploy 5m24s (baseline ~12m), ready 21m14s (wait-ready dominated by external DNS propagation), destroy 7m39s. --- .../ha/aws/DOC-CHANGES-ha-optimizations.md | 195 +++++++++++++ .../pro/ha/aws/cf-openvidu-ha.yaml | 259 ++++++++++++------ 2 files changed, 364 insertions(+), 90 deletions(-) create mode 100644 openvidu-deployment/pro/ha/aws/DOC-CHANGES-ha-optimizations.md diff --git a/openvidu-deployment/pro/ha/aws/DOC-CHANGES-ha-optimizations.md b/openvidu-deployment/pro/ha/aws/DOC-CHANGES-ha-optimizations.md new file mode 100644 index 000000000..ef4c9cc5d --- /dev/null +++ b/openvidu-deployment/pro/ha/aws/DOC-CHANGES-ha-optimizations.md @@ -0,0 +1,195 @@ +# Documentation changes for `ha-optimizations` (AWS HA deployment) + +Instructions for the Claude working on the docs repo +`/home/sergio/Escritorio/openvidu/openvidu.io` (branch `next`, already up to date). + +These instructions derive from the changes applied in +`openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml` on the `ha-optimizations` branch. +Functional summary of what changed in the deployment: + +- The 4 master EC2 instances are now created **in parallel** (the `DependsOn` chain + MasterNode2→WaitCondition1, 3→2, 4→3 was removed; coordination stays data-driven). +- Each master now publishes its private IP to its **own** SSM Parameter Store parameter + (`/openvidu//master-node-N-private-ip`), an atomic per-key write, instead + of a racy read-modify-write into the single shared Secrets Manager JSON. The 4 + `MASTER_NODE_*_PRIVATE_IP` keys were removed from that JSON. +- Media Nodes now install (Docker + image pulls) **in parallel** with the masters. They + are no longer gated behind `MasterNodesWaitCondition4`; instead they poll for the + shared secrets and the master IPs, and wait for at least one healthy master before + starting. The Network Load Balancer is likewise no longer gated behind + `MasterNodesWaitCondition4`. +- Hardening: the previously unbounded "wait for the 4 master IPs" loop is now bounded + (30 min), and the installer download is retried and validated + (`curl --retry 8 ... -o file` + non-empty check) instead of the fragile + `sh <(curl ...)`. + +SUMMARY: **no doc content change is strictly required.** The AWS HA docs publish no +deployment-time figure and describe no internals that changed. The only *optional* +edits are (a) adding a deployment-time figure and (c.1) regenerating one screenshot. +Everything else (public parameters, the single Output, screenshots, other clouds, other +deployment types, on-node config files) stays the same. Read this file in full before +touching anything. + +--- + +## (a) Deployment-time figure — OPTIONAL (none exists today) + +Unlike the GCP and Azure HA `install.md` files, the **AWS HA `install.md` currently +publishes NO deployment-time figure at all** (no "wait about N minutes" sentence). So +there is nothing to update here. This is only a suggestion in case you want to add one +now that the deployment is faster (masters in parallel + media install overlapping the +masters' install + no IP handshake through Secrets Manager). + +File: `docs/docs/self-hosting/ha/aws/install.md` + +Two natural, OPTIONAL insertion points (locate by literal text; line numbers approximate): + +### Option 1 — end of "## Deploying the stack" (~line 155) + +Current literal text (the paragraph that ends the section): + +``` +When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_. +``` + +Proposed (append one sentence; fill in `X`/`Y` from `ov-cloud-tester`): + +``` +When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_. The stack will take about X to Y minutes to create all resources. +``` + +### Option 2 — "## Configuration and administration" (~line 190) + +Current literal text: + +``` +When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status, your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment. +``` + +Proposed (fill in `X`/`Y` from `ov-cloud-tester`): + +``` +When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status (about X to Y minutes), your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment. +``` + +> Placeholder to fill in: `X to Y minutes — fill with the ov-cloud-tester measurement`. +> If you add a figure in BOTH options, keep them consistent. Do NOT invent a number — +> leave the `X to Y` placeholder until the real measurement is available. +> This is entirely optional; skipping section (a) leaves the docs correct. + +--- + +## (b) Public template parameters / outputs: NO CHANGES — confirmation + +Do not touch parameter tables or screenshots in the parameters section. + +Reason: **no CloudFormation `Parameters` were added, removed or renamed, and the single +`Outputs` entry (`ServicesAndCredentials`) is unchanged.** The new per-master IPs live in +SSM Parameter Store parameters that are created internally by the template; they are not +template parameters and require no user input. Therefore: + +- The parameter sections of `docs/docs/self-hosting/ha/aws/install.md` + ("## CloudFormation Parameters" and its subsections, ~lines 52–137) **stay the same**. +- There are no new fields the user must fill in the CloudFormation form. +- The parameter form screenshots **do not change**. +- The `ServicesAndCredentials` output and the "Configure your application to use the + deployment" section (~lines 163–180) are unaffected. + +--- + +## (c) Other statements in the AWS HA docs that may become outdated + +The AWS HA documentation (`docs/docs/self-hosting/ha/aws/*.md`) and the shared AWS +includes (`shared/self-hosting/aws/*.md`) were reviewed against the internal changes. +Conclusions: + +### c.1 — Secrets Manager contents: `MASTER_NODE_{1..4}_PRIVATE_IP` removed from the JSON + +The 4 keys `MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` were removed from the +`OpenViduSharedInfo` Secrets Manager JSON (the masters' IPs now live in SSM Parameter +Store). The AWS HA docs **do not mention these keys by name in any text**, so there is no +prose to fix. + +- `install.md` (~line 165–177) and `admin.md` (~line 226–238) describe the + `ServicesAndCredentials` secret only generically ("contains all URLs and credentials", + "get the JSON with all the information"). The "most relevant" values are listed via the + shared includes `shared/self-hosting/aws/credentials-general.md` and + `credentials-v2compatibility.md`, neither of which lists the IP keys. No change. +- Only possible impact: the secret screenshot + `assets/images/platform/self-hosting/ha/aws/2-secrets.png` (shown in `install.md`) + COULD still display the 4 removed IP keys. This is purely cosmetic. OPTIONAL action: + if/when the screenshot is regenerated, do it against an `ha-optimizations` deployment. + This is NOT blocking. + +### c.2 — `upgrade.md`: DO NOT TOUCH + +`docs/docs/self-hosting/ha/aws/upgrade.md` (~lines 82–92) documents +`/usr/local/bin/store_secret.sh save OPENVIDU_VERSION ""` and the +`OPENVIDU_VERSION` Secrets Manager secret. **Neither `store_secret.sh` nor the +`OPENVIDU_VERSION` secret was changed** by this work. No changes required. + +### c.3 — `backup-and-restore.md`: DO NOT TOUCH + +`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (~lines 676–692) documents +`MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` inside the **on-node** +configuration file `/opt/openvidu/config/node/master-node.env`. + +IMPORTANT: **these are NOT the removed Secrets Manager keys.** They are config-file +variables that the installer still writes from the `--master-node-private-ip-list` flag +(which is still passed — now built from the SSM parameters instead of the shared JSON). +That on-node flow is unchanged. **Do not modify `backup-and-restore.md`.** + +### c.4 — Master boot order / parallelization: not documented + +The AWS HA docs never state that master nodes are created sequentially / "one by one", +nor do they describe the `DependsOn` chain, `MasterNodesWaitCondition*`, or the fact that +the Load Balancer / Media Nodes used to wait for `MasterNodesWaitCondition4`. Removing +that chain and un-gating the NLB and Media Nodes changes nothing user-facing. No text to +update. + +### c.5 — Internal coordination (`ALL_SECRETS_GENERATED`, SSM IP handshake, health gates): not documented + +The master-1-as-leader secret generation, the `ALL_SECRETS_GENERATED` flag, the new SSM +per-master IP handshake, the bounded wait loops, and the media "wait for a healthy +master" gate are all internal and not publicly documented. Nothing user-facing changes. +No text to update. `admin.md` and `index.md` were also reviewed: no references to the +changed internals. No changes required. + +--- + +## (d) New resources created by the stack: 4 SSM parameters + +The stack now creates **4 SSM Parameter Store parameters** as CloudFormation resources: + +``` +/openvidu//master-node-1-private-ip +/openvidu//master-node-2-private-ip +/openvidu//master-node-3-private-ip +/openvidu//master-node-4-private-ip +``` + +They are visible in the AWS Systems Manager → Parameter Store console and are deleted +with the stack. They are harmless and require no user interaction. + +The AWS HA docs do **not** enumerate the resources the stack creates anywhere, so there +is nothing to update. This note exists only so that, if such an enumeration is ever added +(or if a reader asks about unfamiliar parameters in the console), the 4 parameters are +accounted for and expected. + +--- + +## Implementation checklist + +- [ ] (OPTIONAL, section a) In `docs/docs/self-hosting/ha/aws/install.md`, add a + deployment-time sentence with the `X to Y minutes` placeholder at one of the two + insertion points; measure with `ov-cloud-tester` on `ha-optimizations` and fill in + `X`/`Y`. Skipping this leaves the docs correct. +- [ ] Confirm NO parameter tables or screenshots were touched (section b). +- [ ] Confirm `upgrade.md` (`store_secret.sh` / `OPENVIDU_VERSION`) was NOT touched (c.2). +- [ ] Confirm `backup-and-restore.md` (on-node `master-node.env` IP vars) was NOT touched + (c.3). +- [ ] (OPTIONAL, section c.1) Regenerate `2-secrets.png` against an `ha-optimizations` + deployment if you want it to stop showing the 4 removed `MASTER_NODE_*_PRIVATE_IP` + keys. Not blocking. +- [ ] No action needed for the 4 new SSM parameters unless a resource enumeration is + added to the docs (section d). diff --git a/openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml b/openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml index 5f7a91afc..b9b259b41 100644 --- a/openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml +++ b/openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml @@ -1242,14 +1242,39 @@ Resources: "GRAFANA_ADMIN_USERNAME": "none", "GRAFANA_ADMIN_PASSWORD": "none", "ENABLED_MODULES": "none", - "MASTER_NODE_1_PRIVATE_IP": "none", - "MASTER_NODE_2_PRIVATE_IP": "none", - "MASTER_NODE_3_PRIVATE_IP": "none", - "MASTER_NODE_4_PRIVATE_IP": "none", "OPENVIDU_VERSION": "none", "ALL_SECRETS_GENERATED": "false" } + # One SSM parameter per master: atomic per-key writes + MasterNode1PrivateIpParameter: + Type: AWS::SSM::Parameter + Properties: + Name: !Sub /openvidu/${AWS::StackName}/master-node-1-private-ip + Type: String + Value: "none" + + MasterNode2PrivateIpParameter: + Type: AWS::SSM::Parameter + Properties: + Name: !Sub /openvidu/${AWS::StackName}/master-node-2-private-ip + Type: String + Value: "none" + + MasterNode3PrivateIpParameter: + Type: AWS::SSM::Parameter + Properties: + Name: !Sub /openvidu/${AWS::StackName}/master-node-3-private-ip + Type: String + Value: "none" + + MasterNode4PrivateIpParameter: + Type: AWS::SSM::Parameter + Properties: + Name: !Sub /openvidu/${AWS::StackName}/master-node-4-private-ip + Type: String + Value: "none" + S3AppDataBucketResource: Type: 'AWS::S3::Bucket' Properties: @@ -1281,8 +1306,7 @@ Resources: Condition: CreateClusterDataBucket # ------------------------- - # Preprocess subnets to allocate Volumes and ENIs across Availability Zones - # For OpenVidu Master Nodes + # Normalize the master node subnet list to exactly 4 entries, one per OpenVidu Master Node # ------------------------- SubnetProcessorFunction: Type: AWS::Lambda::Function @@ -1293,13 +1317,11 @@ Resources: Code: ZipFile: | import cfnresponse - import boto3 def lambda_handler(event, context): try: # Process event data subnets = event['ResourceProperties']['Subnets'] - ec2 = boto3.client('ec2') # Ensure we have at least four subnets by cycling through the available subnets subnets = (subnets * 4)[:4] # Repeat the list to have at least 4 elements and then take the first 4 @@ -1355,10 +1377,6 @@ Resources: - logs:CreateLogStream - logs:PutLogEvents Resource: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/SubnetProcessor-${AWS::Region}-${AWS::StackName}:*' - - Effect: Allow - Action: - - ec2:DescribeSubnets - Resource: '*' OpenViduMasterNodeRole: Type: AWS::IAM::Role @@ -1390,6 +1408,11 @@ Resources: - secretsmanager:GetSecretValue - secretsmanager:UpdateSecret Resource: !Ref OpenViduSharedInfo + - Effect: Allow + Action: + - ssm:GetParameter + - ssm:PutParameter + Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/* - Fn::If: - CreateRecordingsBucket - Effect: Allow @@ -1474,6 +1497,10 @@ Resources: Action: - secretsmanager:GetSecretValue Resource: !Ref OpenViduSharedInfo + - Effect: Allow + Action: + - ssm:GetParameter + Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/* - Effect: Allow Action: - autoscaling:SetInstanceHealth @@ -1526,8 +1553,6 @@ Resources: OpenViduMediaInstanceProfile: Type: AWS::IAM::InstanceProfile - DependsOn: - - MasterNodesWaitCondition4 Properties: InstanceProfileName: !Sub OpenViduMediaInstanceProfile-${AWS::Region}-${AWS::StackName} Roles: @@ -1566,42 +1591,15 @@ Resources: rm -rf awscliv2.zip aws fi - # Singal to notify instance is waiting + # Signal to notify instance is waiting SIGNAL_NAME="$1" + # Exported earlier in this instance's UserData + MASTER_NODE_NUM="${!MASTER_NODE_NUM}" + # Token for IMDSv2 TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")" - # Subnets IDs - SUBNETS=( - "${SubnetProcessor.Subnet1}" - "${SubnetProcessor.Subnet2}" - "${SubnetProcessor.Subnet3}" - "${SubnetProcessor.Subnet4}" - ) - - MAC_ADDRESS="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254//latest/meta-data/mac)" - SUBNET_ID="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDRESS/subnet-id")" - - # Check master node number - MASTER_NODE_NUM=1 - for subnet in "${!SUBNETS[@]}"; do - if [[ "$subnet" == "$SUBNET_ID" ]]; then - - SHARED_SECRET=$(aws secretsmanager get-secret-value \ - --region ${AWS::Region} \ - --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \ - --query SecretString --output text || echo 'none') - - # Check if current master node is reacheable with ping - ACUTAL_MASTER_NODE_IP=$(echo "$SHARED_SECRET" | jq -r ".MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP") - if [[ "$ACUTAL_MASTER_NODE_IP" == "none" ]]; then - break - fi - fi - MASTER_NODE_NUM=$((MASTER_NODE_NUM + 1)) - done - # Get own private IP PRIVATE_IP="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/local-ipv4)" @@ -1610,8 +1608,12 @@ Resources: exit 1 fi - # Store current private IP - /usr/local/bin/store_secret.sh save MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP "${!PRIVATE_IP}" + aws ssm put-parameter \ + --region ${AWS::Region} \ + --name "/openvidu/${AWS::StackName}/master-node-${!MASTER_NODE_NUM}-private-ip" \ + --value "${!PRIVATE_IP}" \ + --type String \ + --overwrite > /dev/null SHARED_SECRET=$(aws secretsmanager get-secret-value \ --region ${AWS::Region} \ @@ -1690,39 +1692,39 @@ Resources: # sending the signal call cfn-signal -e $? --stack ${AWS::StackId} --resource "$SIGNAL_NAME" --region ${AWS::Region} - # Wait for all master nodes to store their private IPs + IP_WAIT_ATTEMPTS=0 + IP_WAIT_MAX=360 while true; do - SHARED_SECRET=$(aws secretsmanager get-secret-value \ - --region ${AWS::Region} \ - --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \ - --query SecretString --output text || echo 'none') + MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') - MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP') - MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP') - MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP') - MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP') - - # Check if all master nodes have stored their private IPs - if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && - [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && - [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && - [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]]; then + # Check if all master nodes have published their private IPs + if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then break fi + + IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1)) + if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then + echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM" + exit 1 + fi sleep 5 done + # Comma-separated 1-4: must match installer's --master-node-private-ip-list format + MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP" + + # Re-read: other (non-IP) values still come from the shared secret SHARED_SECRET=$(aws secretsmanager get-secret-value \ --region ${AWS::Region} \ --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \ --query SecretString --output text) - MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP') - MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP') - MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP') - MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP') - MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP" - DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME') OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE') OPENVIDU_RTC_ENGINE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_RTC_ENGINE') @@ -1738,18 +1740,22 @@ Resources: GRAFANA_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_PASSWORD') MEET_INITIAL_ADMIN_USER=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_USER') MEET_INITIAL_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_PASSWORD') + MEET_INITIAL_API_KEY="" if [[ "${InitialMeetApiKey}" != '' ]]; then MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY') - else - MEET_INITIAL_API_KEY="" fi - MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY') LIVEKIT_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_KEY') LIVEKIT_API_SECRET=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_SECRET') ENABLED_MODULES=$(echo "$SHARED_SECRET" | jq -r '.ENABLED_MODULES') - # Base command - INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)" + # Download first: sh <(curl ...) would silently run an empty script on a transient curl failure + INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh" + curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh" + if [ ! -s "$INSTALLER_SCRIPT" ]; then + echo "Downloaded OpenVidu master node installer is empty or missing" >&2 + exit 1 + fi + INSTALL_COMMAND="sh $INSTALLER_SCRIPT" # Common arguments COMMON_ARGS=( @@ -1837,6 +1843,10 @@ Resources: content: !Sub | #!/bin/bash set -e + # Only master 1 writes the shared secret, to avoid a lost-update race + if [[ "${!MASTER_NODE_NUM:-}" != "1" ]]; then + exit 0 + fi # Get current shared secret SHARED_SECRET=$(aws secretsmanager get-secret-value \ --region ${AWS::Region} \ @@ -2103,6 +2113,8 @@ Resources: OpenViduMasterNode1: Type: AWS::EC2::Instance + # Depends on its own SSM parameter so its put-parameter can't race the parameter's creation + DependsOn: MasterNode1PrivateIpParameter Properties: LaunchTemplate: LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate @@ -2133,6 +2145,8 @@ Resources: cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate + export MASTER_NODE_NUM=1 + # Install OpenVidu /usr/local/bin/install.sh "MasterNodesWaitCondition1" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; } @@ -2179,7 +2193,7 @@ Resources: OpenViduMasterNode2: Type: AWS::EC2::Instance - DependsOn: MasterNodesWaitCondition1 + DependsOn: MasterNode2PrivateIpParameter Properties: LaunchTemplate: LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate @@ -2210,6 +2224,8 @@ Resources: cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate + export MASTER_NODE_NUM=2 + # Install OpenVidu /usr/local/bin/install.sh "MasterNodesWaitCondition2" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; } @@ -2256,7 +2272,7 @@ Resources: OpenViduMasterNode3: Type: AWS::EC2::Instance - DependsOn: MasterNodesWaitCondition2 + DependsOn: MasterNode3PrivateIpParameter Properties: LaunchTemplate: LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate @@ -2287,6 +2303,8 @@ Resources: cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate + export MASTER_NODE_NUM=3 + # Install OpenVidu /usr/local/bin/install.sh "MasterNodesWaitCondition3" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; } @@ -2333,7 +2351,7 @@ Resources: OpenViduMasterNode4: Type: AWS::EC2::Instance - DependsOn: MasterNodesWaitCondition3 + DependsOn: MasterNode4PrivateIpParameter Properties: LaunchTemplate: LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate @@ -2364,6 +2382,8 @@ Resources: cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate + export MASTER_NODE_NUM=4 + # Install OpenVidu /usr/local/bin/install.sh "MasterNodesWaitCondition4" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; } @@ -2453,6 +2473,25 @@ Resources: exit 1 fi + SECRETS_WAIT_ATTEMPTS=0 + SECRETS_WAIT_MAX=360 + while true; do + ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED') + if [[ "$ALL_SECRETS_GENERATED" == "true" ]]; then + break + fi + SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1)) + if [[ $SECRETS_WAIT_ATTEMPTS -ge $SECRETS_WAIT_MAX ]]; then + echo "Error: timed out after 30 minutes waiting for shared secrets to be generated" + exit 1 + fi + sleep 5 + SHARED_SECRET="$(aws secretsmanager get-secret-value \ + --region ${AWS::Region} \ + --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \ + --query SecretString --output text || echo 'none')" + done + # Get OpenVidu Media Nodes version to deploy OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION') if [[ "$OPENVIDU_VERSION" == "none" ]]; then @@ -2460,20 +2499,39 @@ Resources: exit 1 fi - ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED') - if [[ "$ALL_SECRETS_GENERATED" == "none" ]]; then - echo "Error: Secrets not generated" - exit 1 - fi - MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP') - MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP') - MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP') - MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP') + IP_WAIT_ATTEMPTS=0 + IP_WAIT_MAX=360 + while true; do + MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + + if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] && + [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then + break + fi + + IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1)) + if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then + echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM" + exit 1 + fi + sleep 5 + done MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP" REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD') - # Base command - INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)" + # Download first: sh <(curl ...) would silently run an empty script on a transient curl failure + INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh" + curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh" + if [ ! -s "$INSTALLER_SCRIPT" ]; then + echo "Downloaded OpenVidu media node installer is empty or missing" >&2 + exit 1 + fi + INSTALL_COMMAND="sh $INSTALLER_SCRIPT" # Common arguments COMMON_ARGS=( @@ -2609,13 +2667,36 @@ Resources: python3 -m pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-${!CFN_BOOTSTRAP_VERSION}.tar.gz fi - cfn-init --region ${AWS::Region} --stack ${AWS::StackId} --resource OpenViduMediaNodeLaunchTemplate + cfn-init --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMediaNodeLaunchTemplate export HOME="/root" # Install OpenVidu /usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; } + MASTER_IP_1=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_IP_2=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_IP_3=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_IP_4=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none') + MASTER_HEALTHY=false + for i in $(seq 1 360); do + for MASTER_IP in "$MASTER_IP_1" "$MASTER_IP_2" "$MASTER_IP_3" "$MASTER_IP_4"; do + if [ "$MASTER_IP" != "none" ] && [ -n "$MASTER_IP" ] && curl -sf "http://$MASTER_IP:7880/health/caddy" >/dev/null 2>&1; then + MASTER_HEALTHY=true + break + fi + done + if [ "$MASTER_HEALTHY" = "true" ]; then + break + fi + sleep 5 + done + if [ "$MASTER_HEALTHY" != "true" ]; then + echo "[OpenVidu] no master node became healthy after 30 minutes" + /usr/local/bin/set_as_unhealthy.sh + exit 1 + fi + # Start OpenVidu systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; } @@ -3191,8 +3272,6 @@ Resources: LoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer - DependsOn: - - MasterNodesWaitCondition4 Properties: Name: Fn::Join: