openvidu-deployment: optimize AWS HA deployment time

Parallel master creation: drop the MasterNodeN -> WaitConditionN-1
DependsOn chain. Each master gets a fixed MASTER_NODE_NUM via UserData
(replaces subnet-matching detection) and publishes its private IP to its
own SSM parameter (atomic per-key writes; the 4 parameters are stack
resources), removing the IP exchange through the shared Secrets Manager
JSON and its lost-update race. after_install.sh is now leader-only, so
the shared JSON keeps a single writer with parallel masters.

Ungate media nodes and the NLB from WaitCondition4: media wait for
secrets + the 4 SSM IPs (bounded polls) before the heavy install, and
for a healthy master (:7880/health/caddy) before starting the service.
Bounded timeout on the previously infinite IP wait, robust installer
fetch (curl --retry to file instead of sh <(curl)), SSM read/write IAM
statements, MEET_INITIAL_API_KEY dead-code fix, SubnetProcessor Lambda
cleanup (unused boto3 client and ec2:DescribeSubnets permission).

Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS.
deploy 5m24s (baseline ~12m), ready 21m14s (wait-ready dominated by
external DNS propagation), destroy 7m39s.

openvidu-deployment/pro/ha/aws/DOC-CHANGES-ha-optimizations.md

@@ -0,0 +1,195 @@
+# Documentation changes for `ha-optimizations` (AWS HA deployment)
+
+Instructions for the Claude working on the docs repo
+`/home/sergio/Escritorio/openvidu/openvidu.io` (branch `next`, already up to date).
+
+These instructions derive from the changes applied in
+`openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml` on the `ha-optimizations` branch.
+Functional summary of what changed in the deployment:
+
+- The 4 master EC2 instances are now created **in parallel** (the `DependsOn` chain
+  MasterNode2→WaitCondition1, 3→2, 4→3 was removed; coordination stays data-driven).
+- Each master now publishes its private IP to its **own** SSM Parameter Store parameter
+  (`/openvidu/<stack-name>/master-node-N-private-ip`), an atomic per-key write, instead
+  of a racy read-modify-write into the single shared Secrets Manager JSON. The 4
+  `MASTER_NODE_*_PRIVATE_IP` keys were removed from that JSON.
+- Media Nodes now install (Docker + image pulls) **in parallel** with the masters. They
+  are no longer gated behind `MasterNodesWaitCondition4`; instead they poll for the
+  shared secrets and the master IPs, and wait for at least one healthy master before
+  starting. The Network Load Balancer is likewise no longer gated behind
+  `MasterNodesWaitCondition4`.
+- Hardening: the previously unbounded "wait for the 4 master IPs" loop is now bounded
+  (30 min), and the installer download is retried and validated
+  (`curl --retry 8 ... -o file` + non-empty check) instead of the fragile
+  `sh <(curl ...)`.
+
+SUMMARY: **no doc content change is strictly required.** The AWS HA docs publish no
+deployment-time figure and describe no internals that changed. The only *optional*
+edits are (a) adding a deployment-time figure and (c.1) regenerating one screenshot.
+Everything else (public parameters, the single Output, screenshots, other clouds, other
+deployment types, on-node config files) stays the same. Read this file in full before
+touching anything.
+
+---
+
+## (a) Deployment-time figure — OPTIONAL (none exists today)
+
+Unlike the GCP and Azure HA `install.md` files, the **AWS HA `install.md` currently
+publishes NO deployment-time figure at all** (no "wait about N minutes" sentence). So
+there is nothing to update here. This is only a suggestion in case you want to add one
+now that the deployment is faster (masters in parallel + media install overlapping the
+masters' install + no IP handshake through Secrets Manager).
+
+File: `docs/docs/self-hosting/ha/aws/install.md`
+
+Two natural, OPTIONAL insertion points (locate by literal text; line numbers approximate):
+
+### Option 1 — end of "## Deploying the stack" (~line 155)
+
+Current literal text (the paragraph that ends the section):
+
+```
+When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_.
+```
+
+Proposed (append one sentence; fill in `X`/`Y` from `ov-cloud-tester`):
+
+```
+When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_. The stack will take about X to Y minutes to create all resources.
+```
+
+### Option 2 — "## Configuration and administration" (~line 190)
+
+Current literal text:
+
+```
+When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status, your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment.
+```
+
+Proposed (fill in `X`/`Y` from `ov-cloud-tester`):
+
+```
+When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status (about X to Y minutes), your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment.
+```
+
+> Placeholder to fill in: `X to Y minutes — fill with the ov-cloud-tester measurement`.
+> If you add a figure in BOTH options, keep them consistent. Do NOT invent a number —
+> leave the `X to Y` placeholder until the real measurement is available.
+> This is entirely optional; skipping section (a) leaves the docs correct.
+
+---
+
+## (b) Public template parameters / outputs: NO CHANGES — confirmation
+
+Do not touch parameter tables or screenshots in the parameters section.
+
+Reason: **no CloudFormation `Parameters` were added, removed or renamed, and the single
+`Outputs` entry (`ServicesAndCredentials`) is unchanged.** The new per-master IPs live in
+SSM Parameter Store parameters that are created internally by the template; they are not
+template parameters and require no user input. Therefore:
+
+- The parameter sections of `docs/docs/self-hosting/ha/aws/install.md`
+  ("## CloudFormation Parameters" and its subsections, ~lines 52–137) **stay the same**.
+- There are no new fields the user must fill in the CloudFormation form.
+- The parameter form screenshots **do not change**.
+- The `ServicesAndCredentials` output and the "Configure your application to use the
+  deployment" section (~lines 163–180) are unaffected.
+
+---
+
+## (c) Other statements in the AWS HA docs that may become outdated
+
+The AWS HA documentation (`docs/docs/self-hosting/ha/aws/*.md`) and the shared AWS
+includes (`shared/self-hosting/aws/*.md`) were reviewed against the internal changes.
+Conclusions:
+
+### c.1 — Secrets Manager contents: `MASTER_NODE_{1..4}_PRIVATE_IP` removed from the JSON
+
+The 4 keys `MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` were removed from the
+`OpenViduSharedInfo` Secrets Manager JSON (the masters' IPs now live in SSM Parameter
+Store). The AWS HA docs **do not mention these keys by name in any text**, so there is no
+prose to fix.
+
+- `install.md` (~line 165–177) and `admin.md` (~line 226–238) describe the
+  `ServicesAndCredentials` secret only generically ("contains all URLs and credentials",
+  "get the JSON with all the information"). The "most relevant" values are listed via the
+  shared includes `shared/self-hosting/aws/credentials-general.md` and
+  `credentials-v2compatibility.md`, neither of which lists the IP keys. No change.
+- Only possible impact: the secret screenshot
+  `assets/images/platform/self-hosting/ha/aws/2-secrets.png` (shown in `install.md`)
+  COULD still display the 4 removed IP keys. This is purely cosmetic. OPTIONAL action:
+  if/when the screenshot is regenerated, do it against an `ha-optimizations` deployment.
+  This is NOT blocking.
+
+### c.2 — `upgrade.md`: DO NOT TOUCH
+
+`docs/docs/self-hosting/ha/aws/upgrade.md` (~lines 82–92) documents
+`/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "<VERSION>"` and the
+`OPENVIDU_VERSION` Secrets Manager secret. **Neither `store_secret.sh` nor the
+`OPENVIDU_VERSION` secret was changed** by this work. No changes required.
+
+### c.3 — `backup-and-restore.md`: DO NOT TOUCH
+
+`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (~lines 676–692) documents
+`MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` inside the **on-node**
+configuration file `/opt/openvidu/config/node/master-node.env`.
+
+IMPORTANT: **these are NOT the removed Secrets Manager keys.** They are config-file
+variables that the installer still writes from the `--master-node-private-ip-list` flag
+(which is still passed — now built from the SSM parameters instead of the shared JSON).
+That on-node flow is unchanged. **Do not modify `backup-and-restore.md`.**
+
+### c.4 — Master boot order / parallelization: not documented
+
+The AWS HA docs never state that master nodes are created sequentially / "one by one",
+nor do they describe the `DependsOn` chain, `MasterNodesWaitCondition*`, or the fact that
+the Load Balancer / Media Nodes used to wait for `MasterNodesWaitCondition4`. Removing
+that chain and un-gating the NLB and Media Nodes changes nothing user-facing. No text to
+update.
+
+### c.5 — Internal coordination (`ALL_SECRETS_GENERATED`, SSM IP handshake, health gates): not documented
+
+The master-1-as-leader secret generation, the `ALL_SECRETS_GENERATED` flag, the new SSM
+per-master IP handshake, the bounded wait loops, and the media "wait for a healthy
+master" gate are all internal and not publicly documented. Nothing user-facing changes.
+No text to update. `admin.md` and `index.md` were also reviewed: no references to the
+changed internals. No changes required.
+
+---
+
+## (d) New resources created by the stack: 4 SSM parameters
+
+The stack now creates **4 SSM Parameter Store parameters** as CloudFormation resources:
+
+```
+/openvidu/<stack-name>/master-node-1-private-ip
+/openvidu/<stack-name>/master-node-2-private-ip
+/openvidu/<stack-name>/master-node-3-private-ip
+/openvidu/<stack-name>/master-node-4-private-ip
+```
+
+They are visible in the AWS Systems Manager → Parameter Store console and are deleted
+with the stack. They are harmless and require no user interaction.
+
+The AWS HA docs do **not** enumerate the resources the stack creates anywhere, so there
+is nothing to update. This note exists only so that, if such an enumeration is ever added
+(or if a reader asks about unfamiliar parameters in the console), the 4 parameters are
+accounted for and expected.
+
+---
+
+## Implementation checklist
+
+- [ ] (OPTIONAL, section a) In `docs/docs/self-hosting/ha/aws/install.md`, add a
+      deployment-time sentence with the `X to Y minutes` placeholder at one of the two
+      insertion points; measure with `ov-cloud-tester` on `ha-optimizations` and fill in
+      `X`/`Y`. Skipping this leaves the docs correct.
+- [ ] Confirm NO parameter tables or screenshots were touched (section b).
+- [ ] Confirm `upgrade.md` (`store_secret.sh` / `OPENVIDU_VERSION`) was NOT touched (c.2).
+- [ ] Confirm `backup-and-restore.md` (on-node `master-node.env` IP vars) was NOT touched
+      (c.3).
+- [ ] (OPTIONAL, section c.1) Regenerate `2-secrets.png` against an `ha-optimizations`
+      deployment if you want it to stop showing the 4 removed `MASTER_NODE_*_PRIVATE_IP`
+      keys. Not blocking.
+- [ ] No action needed for the 4 new SSM parameters unless a resource enumeration is
+      added to the docs (section d).

openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml

@@ -1242,14 +1242,39 @@ Resources:
           "GRAFANA_ADMIN_USERNAME": "none",
           "GRAFANA_ADMIN_PASSWORD": "none",
           "ENABLED_MODULES": "none",
-          "MASTER_NODE_1_PRIVATE_IP": "none",
-          "MASTER_NODE_2_PRIVATE_IP": "none",
-          "MASTER_NODE_3_PRIVATE_IP": "none",
-          "MASTER_NODE_4_PRIVATE_IP": "none",
           "OPENVIDU_VERSION": "none",
           "ALL_SECRETS_GENERATED": "false"
         }
 
+  # One SSM parameter per master: atomic per-key writes
+  MasterNode1PrivateIpParameter:
+    Type: AWS::SSM::Parameter
+    Properties:
+      Name: !Sub /openvidu/${AWS::StackName}/master-node-1-private-ip
+      Type: String
+      Value: "none"
+
+  MasterNode2PrivateIpParameter:
+    Type: AWS::SSM::Parameter
+    Properties:
+      Name: !Sub /openvidu/${AWS::StackName}/master-node-2-private-ip
+      Type: String
+      Value: "none"
+
+  MasterNode3PrivateIpParameter:
+    Type: AWS::SSM::Parameter
+    Properties:
+      Name: !Sub /openvidu/${AWS::StackName}/master-node-3-private-ip
+      Type: String
+      Value: "none"
+
+  MasterNode4PrivateIpParameter:
+    Type: AWS::SSM::Parameter
+    Properties:
+      Name: !Sub /openvidu/${AWS::StackName}/master-node-4-private-ip
+      Type: String
+      Value: "none"
+
   S3AppDataBucketResource:
     Type: 'AWS::S3::Bucket'
     Properties:
@@ -1281,8 +1306,7 @@ Resources:
     Condition: CreateClusterDataBucket
 
   # -------------------------
-  # Preprocess subnets to allocate Volumes and ENIs across Availability Zones
+  # Normalize the master node subnet list to exactly 4 entries, one per OpenVidu Master Node
-  # For OpenVidu Master Nodes
   # -------------------------
   SubnetProcessorFunction:
     Type: AWS::Lambda::Function
@@ -1293,13 +1317,11 @@ Resources:
       Code:
         ZipFile: |
           import cfnresponse
-          import boto3
 
           def lambda_handler(event, context):
               try:
                   # Process event data
                   subnets = event['ResourceProperties']['Subnets']
-                  ec2 = boto3.client('ec2')
 
                   # Ensure we have at least four subnets by cycling through the available subnets
                   subnets = (subnets * 4)[:4]  # Repeat the list to have at least 4 elements and then take the first 4
@@ -1355,10 +1377,6 @@ Resources:
                   - logs:CreateLogStream
                   - logs:PutLogEvents
                 Resource: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/SubnetProcessor-${AWS::Region}-${AWS::StackName}:*'
-              - Effect: Allow
-                Action:
-                  - ec2:DescribeSubnets
-                Resource: '*'
 
   OpenViduMasterNodeRole:
     Type: AWS::IAM::Role
@@ -1390,6 +1408,11 @@ Resources:
                   - secretsmanager:GetSecretValue
                   - secretsmanager:UpdateSecret
                 Resource: !Ref OpenViduSharedInfo
+              - Effect: Allow
+                Action:
+                  - ssm:GetParameter
+                  - ssm:PutParameter
+                Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/*
               - Fn::If:
                 - CreateRecordingsBucket
                 - Effect: Allow
@@ -1474,6 +1497,10 @@ Resources:
                 Action:
                   - secretsmanager:GetSecretValue
                 Resource: !Ref OpenViduSharedInfo
+              - Effect: Allow
+                Action:
+                  - ssm:GetParameter
+                Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/*
               - Effect: Allow
                 Action:
                   - autoscaling:SetInstanceHealth
@@ -1526,8 +1553,6 @@ Resources:
 
   OpenViduMediaInstanceProfile:
     Type: AWS::IAM::InstanceProfile
-    DependsOn:
-         - MasterNodesWaitCondition4
     Properties:
       InstanceProfileName: !Sub OpenViduMediaInstanceProfile-${AWS::Region}-${AWS::StackName}
       Roles:
@@ -1566,42 +1591,15 @@ Resources:
                   rm -rf awscliv2.zip aws
                 fi
 
-                # Singal to notify instance is waiting
+                # Signal to notify instance is waiting
                 SIGNAL_NAME="$1"
 
+                # Exported earlier in this instance's UserData
+                MASTER_NODE_NUM="${!MASTER_NODE_NUM}"
+
                 # Token for IMDSv2
                 TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")"
 
-                # Subnets IDs
-                SUBNETS=(
-                  "${SubnetProcessor.Subnet1}"
-                  "${SubnetProcessor.Subnet2}"
-                  "${SubnetProcessor.Subnet3}"
-                  "${SubnetProcessor.Subnet4}"
-                )
-
-                MAC_ADDRESS="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254//latest/meta-data/mac)"
-                SUBNET_ID="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDRESS/subnet-id")"
-
-                # Check master node number
-                MASTER_NODE_NUM=1
-                for subnet in "${!SUBNETS[@]}"; do
-                  if [[ "$subnet" == "$SUBNET_ID" ]]; then
-
-                    SHARED_SECRET=$(aws secretsmanager get-secret-value \
-                      --region ${AWS::Region} \
-                      --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
-                      --query SecretString --output text || echo 'none')
-
-                      # Check if current master node is reacheable with ping
-                      ACUTAL_MASTER_NODE_IP=$(echo "$SHARED_SECRET" | jq -r ".MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP")
-                      if [[ "$ACUTAL_MASTER_NODE_IP" == "none" ]]; then
-                        break
-                      fi
-                  fi
-                  MASTER_NODE_NUM=$((MASTER_NODE_NUM + 1))
-                done
-
                 # Get own private IP
                 PRIVATE_IP="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/local-ipv4)"
 
@@ -1610,8 +1608,12 @@ Resources:
                   exit 1
                 fi
 
-                # Store current private IP
+                aws ssm put-parameter \
-                /usr/local/bin/store_secret.sh save MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP "${!PRIVATE_IP}"
+                  --region ${AWS::Region} \
+                  --name "/openvidu/${AWS::StackName}/master-node-${!MASTER_NODE_NUM}-private-ip" \
+                  --value "${!PRIVATE_IP}" \
+                  --type String \
+                  --overwrite > /dev/null
 
                 SHARED_SECRET=$(aws secretsmanager get-secret-value \
                   --region ${AWS::Region} \
@@ -1690,39 +1692,39 @@ Resources:
                 # sending the signal call
                 cfn-signal -e $? --stack ${AWS::StackId} --resource "$SIGNAL_NAME" --region ${AWS::Region}
 
-                # Wait for all master nodes to store their private IPs
+                IP_WAIT_ATTEMPTS=0
+                IP_WAIT_MAX=360
                 while true; do
-                  SHARED_SECRET=$(aws secretsmanager get-secret-value \
+                  MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                    --region ${AWS::Region} \
+                  MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                    --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
+                  MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                    --query SecretString --output text || echo 'none')
+                  MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
 
-                  MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
+                  # Check if all master nodes have published their private IPs
-                  MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
+                  if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] &&
-                  MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
+                      [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] &&
-                  MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
+                      [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] &&
-
+                      [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then
-                  # Check if all master nodes have stored their private IPs
-                  if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] &&
-                      [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] &&
-                      [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] &&
-                      [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]]; then
                     break
                   fi
+
+                  IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1))
+                  if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then
+                    echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM"
+                    exit 1
+                  fi
                   sleep 5
                 done
 
+                # Comma-separated 1-4: must match installer's --master-node-private-ip-list format
+                MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
+
+                # Re-read: other (non-IP) values still come from the shared secret
                 SHARED_SECRET=$(aws secretsmanager get-secret-value \
                   --region ${AWS::Region} \
                   --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
                   --query SecretString --output text)
 
-                MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
-                MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
-                MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
-                MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
-                MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
-
                 DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
                 OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
                 OPENVIDU_RTC_ENGINE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_RTC_ENGINE')
@@ -1738,18 +1740,22 @@ Resources:
                 GRAFANA_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_PASSWORD')
                 MEET_INITIAL_ADMIN_USER=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_USER')
                 MEET_INITIAL_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_PASSWORD')
+                MEET_INITIAL_API_KEY=""
                 if [[ "${InitialMeetApiKey}" != '' ]]; then
                   MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
-                else
-                  MEET_INITIAL_API_KEY=""
                 fi
-                MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
                 LIVEKIT_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_KEY')
                 LIVEKIT_API_SECRET=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_SECRET')
                 ENABLED_MODULES=$(echo "$SHARED_SECRET" | jq -r '.ENABLED_MODULES')
 
-                # Base command
+                # Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
-                INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)"
+                INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
+                curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh"
+                if [ ! -s "$INSTALLER_SCRIPT" ]; then
+                  echo "Downloaded OpenVidu master node installer is empty or missing" >&2
+                  exit 1
+                fi
+                INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
 
                 # Common arguments
                 COMMON_ARGS=(
@@ -1837,6 +1843,10 @@ Resources:
               content: !Sub |
                 #!/bin/bash
                 set -e
+                # Only master 1 writes the shared secret, to avoid a lost-update race
+                if [[ "${!MASTER_NODE_NUM:-}" != "1" ]]; then
+                  exit 0
+                fi
                 # Get current shared secret
                 SHARED_SECRET=$(aws secretsmanager get-secret-value \
                   --region ${AWS::Region} \
@@ -2103,6 +2113,8 @@ Resources:
 
   OpenViduMasterNode1:
     Type: AWS::EC2::Instance
+    # Depends on its own SSM parameter so its put-parameter can't race the parameter's creation
+    DependsOn: MasterNode1PrivateIpParameter
     Properties:
       LaunchTemplate:
         LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
@@ -2133,6 +2145,8 @@ Resources:
 
           cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
 
+          export MASTER_NODE_NUM=1
+
           # Install OpenVidu
           /usr/local/bin/install.sh "MasterNodesWaitCondition1" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
 
@@ -2179,7 +2193,7 @@ Resources:
 
   OpenViduMasterNode2:
     Type: AWS::EC2::Instance
-    DependsOn: MasterNodesWaitCondition1
+    DependsOn: MasterNode2PrivateIpParameter
     Properties:
       LaunchTemplate:
         LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
@@ -2210,6 +2224,8 @@ Resources:
 
           cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
 
+          export MASTER_NODE_NUM=2
+
           # Install OpenVidu
           /usr/local/bin/install.sh "MasterNodesWaitCondition2" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
 
@@ -2256,7 +2272,7 @@ Resources:
 
   OpenViduMasterNode3:
     Type: AWS::EC2::Instance
-    DependsOn: MasterNodesWaitCondition2
+    DependsOn: MasterNode3PrivateIpParameter
     Properties:
       LaunchTemplate:
         LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
@@ -2287,6 +2303,8 @@ Resources:
 
           cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
 
+          export MASTER_NODE_NUM=3
+
           # Install OpenVidu
           /usr/local/bin/install.sh "MasterNodesWaitCondition3" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
 
@@ -2333,7 +2351,7 @@ Resources:
 
   OpenViduMasterNode4:
     Type: AWS::EC2::Instance
-    DependsOn: MasterNodesWaitCondition3
+    DependsOn: MasterNode4PrivateIpParameter
     Properties:
       LaunchTemplate:
         LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
@@ -2364,6 +2382,8 @@ Resources:
 
           cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
 
+          export MASTER_NODE_NUM=4
+
           # Install OpenVidu
           /usr/local/bin/install.sh "MasterNodesWaitCondition4" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
 
@@ -2453,6 +2473,25 @@ Resources:
                   exit 1
                 fi
 
+                SECRETS_WAIT_ATTEMPTS=0
+                SECRETS_WAIT_MAX=360
+                while true; do
+                  ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
+                  if [[ "$ALL_SECRETS_GENERATED" == "true" ]]; then
+                    break
+                  fi
+                  SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1))
+                  if [[ $SECRETS_WAIT_ATTEMPTS -ge $SECRETS_WAIT_MAX ]]; then
+                    echo "Error: timed out after 30 minutes waiting for shared secrets to be generated"
+                    exit 1
+                  fi
+                  sleep 5
+                  SHARED_SECRET="$(aws secretsmanager get-secret-value \
+                    --region ${AWS::Region} \
+                    --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
+                    --query SecretString --output text || echo 'none')"
+                done
+
                 # Get OpenVidu Media Nodes version to deploy
                 OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
                 if [[ "$OPENVIDU_VERSION" == "none" ]]; then
@@ -2460,20 +2499,39 @@ Resources:
                   exit 1
                 fi
 
-                ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
+                IP_WAIT_ATTEMPTS=0
-                if [[ "$ALL_SECRETS_GENERATED" == "none" ]]; then
+                IP_WAIT_MAX=360
-                  echo "Error: Secrets not generated"
+                while true; do
-                  exit 1
+                  MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                fi
+                  MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
+                  MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
+                  MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
-                MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
+
-                MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
+                  if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] &&
+                      [[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] &&
+                      [[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] &&
+                      [[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then
+                    break
+                  fi
+
+                  IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1))
+                  if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then
+                    echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM"
+                    exit 1
+                  fi
+                  sleep 5
+                done
                 MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
                 REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
 
-                # Base command
+                # Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
-                INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)"
+                INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
+                curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh"
+                if [ ! -s "$INSTALLER_SCRIPT" ]; then
+                  echo "Downloaded OpenVidu media node installer is empty or missing" >&2
+                  exit 1
+                fi
+                INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
 
                 # Common arguments
                 COMMON_ARGS=(
@@ -2609,13 +2667,36 @@ Resources:
               python3 -m pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-${!CFN_BOOTSTRAP_VERSION}.tar.gz
             fi
 
-            cfn-init --region ${AWS::Region} --stack ${AWS::StackId} --resource OpenViduMediaNodeLaunchTemplate
+            cfn-init --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMediaNodeLaunchTemplate
 
             export HOME="/root"
 
             # Install OpenVidu
             /usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
 
+            MASTER_IP_1=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
+            MASTER_IP_2=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
+            MASTER_IP_3=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
+            MASTER_IP_4=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
+            MASTER_HEALTHY=false
+            for i in $(seq 1 360); do
+              for MASTER_IP in "$MASTER_IP_1" "$MASTER_IP_2" "$MASTER_IP_3" "$MASTER_IP_4"; do
+                if [ "$MASTER_IP" != "none" ] && [ -n "$MASTER_IP" ] && curl -sf "http://$MASTER_IP:7880/health/caddy" >/dev/null 2>&1; then
+                  MASTER_HEALTHY=true
+                  break
+                fi
+              done
+              if [ "$MASTER_HEALTHY" = "true" ]; then
+                break
+              fi
+              sleep 5
+            done
+            if [ "$MASTER_HEALTHY" != "true" ]; then
+              echo "[OpenVidu] no master node became healthy after 30 minutes"
+              /usr/local/bin/set_as_unhealthy.sh
+              exit 1
+            fi
+
             # Start OpenVidu
             systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
 
@@ -3191,8 +3272,6 @@ Resources:
 
   LoadBalancer:
     Type: AWS::ElasticLoadBalancingV2::LoadBalancer
-    DependsOn:
-      - MasterNodesWaitCondition4
     Properties:
       Name:
         Fn::Join: