mirror of https://github.com/OpenVidu/openvidu.git
openvidu-deployment: optimize AWS HA deployment time
Parallel master creation: drop the MasterNodeN -> WaitConditionN-1 DependsOn chain. Each master gets a fixed MASTER_NODE_NUM via UserData (replaces subnet-matching detection) and publishes its private IP to its own SSM parameter (atomic per-key writes; the 4 parameters are stack resources), removing the IP exchange through the shared Secrets Manager JSON and its lost-update race. after_install.sh is now leader-only, so the shared JSON keeps a single writer with parallel masters. Ungate media nodes and the NLB from WaitCondition4: media wait for secrets + the 4 SSM IPs (bounded polls) before the heavy install, and for a healthy master (:7880/health/caddy) before starting the service. Bounded timeout on the previously infinite IP wait, robust installer fetch (curl --retry to file instead of sh <(curl)), SSM read/write IAM statements, MEET_INITIAL_API_KEY dead-code fix, SubnetProcessor Lambda cleanup (unused boto3 client and ec2:DescribeSubnets permission). Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS. deploy 5m24s (baseline ~12m), ready 21m14s (wait-ready dominated by external DNS propagation), destroy 7m39s.master
Piwccle
parent
b8fdde9751
commit
4d6b7acd7c
|
|
@ -0,0 +1,195 @@
|
|||
# Documentation changes for `ha-optimizations` (AWS HA deployment)
|
||||
|
||||
Instructions for the Claude working on the docs repo
|
||||
`/home/sergio/Escritorio/openvidu/openvidu.io` (branch `next`, already up to date).
|
||||
|
||||
These instructions derive from the changes applied in
|
||||
`openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml` on the `ha-optimizations` branch.
|
||||
Functional summary of what changed in the deployment:
|
||||
|
||||
- The 4 master EC2 instances are now created **in parallel** (the `DependsOn` chain
|
||||
MasterNode2→WaitCondition1, 3→2, 4→3 was removed; coordination stays data-driven).
|
||||
- Each master now publishes its private IP to its **own** SSM Parameter Store parameter
|
||||
(`/openvidu/<stack-name>/master-node-N-private-ip`), an atomic per-key write, instead
|
||||
of a racy read-modify-write into the single shared Secrets Manager JSON. The 4
|
||||
`MASTER_NODE_*_PRIVATE_IP` keys were removed from that JSON.
|
||||
- Media Nodes now install (Docker + image pulls) **in parallel** with the masters. They
|
||||
are no longer gated behind `MasterNodesWaitCondition4`; instead they poll for the
|
||||
shared secrets and the master IPs, and wait for at least one healthy master before
|
||||
starting. The Network Load Balancer is likewise no longer gated behind
|
||||
`MasterNodesWaitCondition4`.
|
||||
- Hardening: the previously unbounded "wait for the 4 master IPs" loop is now bounded
|
||||
(30 min), and the installer download is retried and validated
|
||||
(`curl --retry 8 ... -o file` + non-empty check) instead of the fragile
|
||||
`sh <(curl ...)`.
|
||||
|
||||
SUMMARY: **no doc content change is strictly required.** The AWS HA docs publish no
|
||||
deployment-time figure and describe no internals that changed. The only *optional*
|
||||
edits are (a) adding a deployment-time figure and (c.1) regenerating one screenshot.
|
||||
Everything else (public parameters, the single Output, screenshots, other clouds, other
|
||||
deployment types, on-node config files) stays the same. Read this file in full before
|
||||
touching anything.
|
||||
|
||||
---
|
||||
|
||||
## (a) Deployment-time figure — OPTIONAL (none exists today)
|
||||
|
||||
Unlike the GCP and Azure HA `install.md` files, the **AWS HA `install.md` currently
|
||||
publishes NO deployment-time figure at all** (no "wait about N minutes" sentence). So
|
||||
there is nothing to update here. This is only a suggestion in case you want to add one
|
||||
now that the deployment is faster (masters in parallel + media install overlapping the
|
||||
masters' install + no IP handshake through Secrets Manager).
|
||||
|
||||
File: `docs/docs/self-hosting/ha/aws/install.md`
|
||||
|
||||
Two natural, OPTIONAL insertion points (locate by literal text; line numbers approximate):
|
||||
|
||||
### Option 1 — end of "## Deploying the stack" (~line 155)
|
||||
|
||||
Current literal text (the paragraph that ends the section):
|
||||
|
||||
```
|
||||
When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_.
|
||||
```
|
||||
|
||||
Proposed (append one sentence; fill in `X`/`Y` from `ov-cloud-tester`):
|
||||
|
||||
```
|
||||
When you are ready with your CloudFormation parameters, just click on _"Next"_, specify in _"Stack failure options"_ the option _"Preserve successfully provisioned resources"_ to be able to troubleshoot the deployment in case of error, click on _"Next"_ again, and finally _"Submit"_. The stack will take about X to Y minutes to create all resources.
|
||||
```
|
||||
|
||||
### Option 2 — "## Configuration and administration" (~line 190)
|
||||
|
||||
Current literal text:
|
||||
|
||||
```
|
||||
When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status, your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment.
|
||||
```
|
||||
|
||||
Proposed (fill in `X`/`Y` from `ov-cloud-tester`):
|
||||
|
||||
```
|
||||
When your CloudFormation stack reaches the **`CREATE_COMPLETE`** status (about X to Y minutes), your OpenVidu High Availability deployment is ready to use. You can check the [Administration](./admin.md) section to learn how to manage your deployment.
|
||||
```
|
||||
|
||||
> Placeholder to fill in: `X to Y minutes — fill with the ov-cloud-tester measurement`.
|
||||
> If you add a figure in BOTH options, keep them consistent. Do NOT invent a number —
|
||||
> leave the `X to Y` placeholder until the real measurement is available.
|
||||
> This is entirely optional; skipping section (a) leaves the docs correct.
|
||||
|
||||
---
|
||||
|
||||
## (b) Public template parameters / outputs: NO CHANGES — confirmation
|
||||
|
||||
Do not touch parameter tables or screenshots in the parameters section.
|
||||
|
||||
Reason: **no CloudFormation `Parameters` were added, removed or renamed, and the single
|
||||
`Outputs` entry (`ServicesAndCredentials`) is unchanged.** The new per-master IPs live in
|
||||
SSM Parameter Store parameters that are created internally by the template; they are not
|
||||
template parameters and require no user input. Therefore:
|
||||
|
||||
- The parameter sections of `docs/docs/self-hosting/ha/aws/install.md`
|
||||
("## CloudFormation Parameters" and its subsections, ~lines 52–137) **stay the same**.
|
||||
- There are no new fields the user must fill in the CloudFormation form.
|
||||
- The parameter form screenshots **do not change**.
|
||||
- The `ServicesAndCredentials` output and the "Configure your application to use the
|
||||
deployment" section (~lines 163–180) are unaffected.
|
||||
|
||||
---
|
||||
|
||||
## (c) Other statements in the AWS HA docs that may become outdated
|
||||
|
||||
The AWS HA documentation (`docs/docs/self-hosting/ha/aws/*.md`) and the shared AWS
|
||||
includes (`shared/self-hosting/aws/*.md`) were reviewed against the internal changes.
|
||||
Conclusions:
|
||||
|
||||
### c.1 — Secrets Manager contents: `MASTER_NODE_{1..4}_PRIVATE_IP` removed from the JSON
|
||||
|
||||
The 4 keys `MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` were removed from the
|
||||
`OpenViduSharedInfo` Secrets Manager JSON (the masters' IPs now live in SSM Parameter
|
||||
Store). The AWS HA docs **do not mention these keys by name in any text**, so there is no
|
||||
prose to fix.
|
||||
|
||||
- `install.md` (~line 165–177) and `admin.md` (~line 226–238) describe the
|
||||
`ServicesAndCredentials` secret only generically ("contains all URLs and credentials",
|
||||
"get the JSON with all the information"). The "most relevant" values are listed via the
|
||||
shared includes `shared/self-hosting/aws/credentials-general.md` and
|
||||
`credentials-v2compatibility.md`, neither of which lists the IP keys. No change.
|
||||
- Only possible impact: the secret screenshot
|
||||
`assets/images/platform/self-hosting/ha/aws/2-secrets.png` (shown in `install.md`)
|
||||
COULD still display the 4 removed IP keys. This is purely cosmetic. OPTIONAL action:
|
||||
if/when the screenshot is regenerated, do it against an `ha-optimizations` deployment.
|
||||
This is NOT blocking.
|
||||
|
||||
### c.2 — `upgrade.md`: DO NOT TOUCH
|
||||
|
||||
`docs/docs/self-hosting/ha/aws/upgrade.md` (~lines 82–92) documents
|
||||
`/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "<VERSION>"` and the
|
||||
`OPENVIDU_VERSION` Secrets Manager secret. **Neither `store_secret.sh` nor the
|
||||
`OPENVIDU_VERSION` secret was changed** by this work. No changes required.
|
||||
|
||||
### c.3 — `backup-and-restore.md`: DO NOT TOUCH
|
||||
|
||||
`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (~lines 676–692) documents
|
||||
`MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` inside the **on-node**
|
||||
configuration file `/opt/openvidu/config/node/master-node.env`.
|
||||
|
||||
IMPORTANT: **these are NOT the removed Secrets Manager keys.** They are config-file
|
||||
variables that the installer still writes from the `--master-node-private-ip-list` flag
|
||||
(which is still passed — now built from the SSM parameters instead of the shared JSON).
|
||||
That on-node flow is unchanged. **Do not modify `backup-and-restore.md`.**
|
||||
|
||||
### c.4 — Master boot order / parallelization: not documented
|
||||
|
||||
The AWS HA docs never state that master nodes are created sequentially / "one by one",
|
||||
nor do they describe the `DependsOn` chain, `MasterNodesWaitCondition*`, or the fact that
|
||||
the Load Balancer / Media Nodes used to wait for `MasterNodesWaitCondition4`. Removing
|
||||
that chain and un-gating the NLB and Media Nodes changes nothing user-facing. No text to
|
||||
update.
|
||||
|
||||
### c.5 — Internal coordination (`ALL_SECRETS_GENERATED`, SSM IP handshake, health gates): not documented
|
||||
|
||||
The master-1-as-leader secret generation, the `ALL_SECRETS_GENERATED` flag, the new SSM
|
||||
per-master IP handshake, the bounded wait loops, and the media "wait for a healthy
|
||||
master" gate are all internal and not publicly documented. Nothing user-facing changes.
|
||||
No text to update. `admin.md` and `index.md` were also reviewed: no references to the
|
||||
changed internals. No changes required.
|
||||
|
||||
---
|
||||
|
||||
## (d) New resources created by the stack: 4 SSM parameters
|
||||
|
||||
The stack now creates **4 SSM Parameter Store parameters** as CloudFormation resources:
|
||||
|
||||
```
|
||||
/openvidu/<stack-name>/master-node-1-private-ip
|
||||
/openvidu/<stack-name>/master-node-2-private-ip
|
||||
/openvidu/<stack-name>/master-node-3-private-ip
|
||||
/openvidu/<stack-name>/master-node-4-private-ip
|
||||
```
|
||||
|
||||
They are visible in the AWS Systems Manager → Parameter Store console and are deleted
|
||||
with the stack. They are harmless and require no user interaction.
|
||||
|
||||
The AWS HA docs do **not** enumerate the resources the stack creates anywhere, so there
|
||||
is nothing to update. This note exists only so that, if such an enumeration is ever added
|
||||
(or if a reader asks about unfamiliar parameters in the console), the 4 parameters are
|
||||
accounted for and expected.
|
||||
|
||||
---
|
||||
|
||||
## Implementation checklist
|
||||
|
||||
- [ ] (OPTIONAL, section a) In `docs/docs/self-hosting/ha/aws/install.md`, add a
|
||||
deployment-time sentence with the `X to Y minutes` placeholder at one of the two
|
||||
insertion points; measure with `ov-cloud-tester` on `ha-optimizations` and fill in
|
||||
`X`/`Y`. Skipping this leaves the docs correct.
|
||||
- [ ] Confirm NO parameter tables or screenshots were touched (section b).
|
||||
- [ ] Confirm `upgrade.md` (`store_secret.sh` / `OPENVIDU_VERSION`) was NOT touched (c.2).
|
||||
- [ ] Confirm `backup-and-restore.md` (on-node `master-node.env` IP vars) was NOT touched
|
||||
(c.3).
|
||||
- [ ] (OPTIONAL, section c.1) Regenerate `2-secrets.png` against an `ha-optimizations`
|
||||
deployment if you want it to stop showing the 4 removed `MASTER_NODE_*_PRIVATE_IP`
|
||||
keys. Not blocking.
|
||||
- [ ] No action needed for the 4 new SSM parameters unless a resource enumeration is
|
||||
added to the docs (section d).
|
||||
|
|
@ -1242,14 +1242,39 @@ Resources:
|
|||
"GRAFANA_ADMIN_USERNAME": "none",
|
||||
"GRAFANA_ADMIN_PASSWORD": "none",
|
||||
"ENABLED_MODULES": "none",
|
||||
"MASTER_NODE_1_PRIVATE_IP": "none",
|
||||
"MASTER_NODE_2_PRIVATE_IP": "none",
|
||||
"MASTER_NODE_3_PRIVATE_IP": "none",
|
||||
"MASTER_NODE_4_PRIVATE_IP": "none",
|
||||
"OPENVIDU_VERSION": "none",
|
||||
"ALL_SECRETS_GENERATED": "false"
|
||||
}
|
||||
|
||||
# One SSM parameter per master: atomic per-key writes
|
||||
MasterNode1PrivateIpParameter:
|
||||
Type: AWS::SSM::Parameter
|
||||
Properties:
|
||||
Name: !Sub /openvidu/${AWS::StackName}/master-node-1-private-ip
|
||||
Type: String
|
||||
Value: "none"
|
||||
|
||||
MasterNode2PrivateIpParameter:
|
||||
Type: AWS::SSM::Parameter
|
||||
Properties:
|
||||
Name: !Sub /openvidu/${AWS::StackName}/master-node-2-private-ip
|
||||
Type: String
|
||||
Value: "none"
|
||||
|
||||
MasterNode3PrivateIpParameter:
|
||||
Type: AWS::SSM::Parameter
|
||||
Properties:
|
||||
Name: !Sub /openvidu/${AWS::StackName}/master-node-3-private-ip
|
||||
Type: String
|
||||
Value: "none"
|
||||
|
||||
MasterNode4PrivateIpParameter:
|
||||
Type: AWS::SSM::Parameter
|
||||
Properties:
|
||||
Name: !Sub /openvidu/${AWS::StackName}/master-node-4-private-ip
|
||||
Type: String
|
||||
Value: "none"
|
||||
|
||||
S3AppDataBucketResource:
|
||||
Type: 'AWS::S3::Bucket'
|
||||
Properties:
|
||||
|
|
@ -1281,8 +1306,7 @@ Resources:
|
|||
Condition: CreateClusterDataBucket
|
||||
|
||||
# -------------------------
|
||||
# Preprocess subnets to allocate Volumes and ENIs across Availability Zones
|
||||
# For OpenVidu Master Nodes
|
||||
# Normalize the master node subnet list to exactly 4 entries, one per OpenVidu Master Node
|
||||
# -------------------------
|
||||
SubnetProcessorFunction:
|
||||
Type: AWS::Lambda::Function
|
||||
|
|
@ -1293,13 +1317,11 @@ Resources:
|
|||
Code:
|
||||
ZipFile: |
|
||||
import cfnresponse
|
||||
import boto3
|
||||
|
||||
def lambda_handler(event, context):
|
||||
try:
|
||||
# Process event data
|
||||
subnets = event['ResourceProperties']['Subnets']
|
||||
ec2 = boto3.client('ec2')
|
||||
|
||||
# Ensure we have at least four subnets by cycling through the available subnets
|
||||
subnets = (subnets * 4)[:4] # Repeat the list to have at least 4 elements and then take the first 4
|
||||
|
|
@ -1355,10 +1377,6 @@ Resources:
|
|||
- logs:CreateLogStream
|
||||
- logs:PutLogEvents
|
||||
Resource: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/SubnetProcessor-${AWS::Region}-${AWS::StackName}:*'
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- ec2:DescribeSubnets
|
||||
Resource: '*'
|
||||
|
||||
OpenViduMasterNodeRole:
|
||||
Type: AWS::IAM::Role
|
||||
|
|
@ -1390,6 +1408,11 @@ Resources:
|
|||
- secretsmanager:GetSecretValue
|
||||
- secretsmanager:UpdateSecret
|
||||
Resource: !Ref OpenViduSharedInfo
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- ssm:GetParameter
|
||||
- ssm:PutParameter
|
||||
Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/*
|
||||
- Fn::If:
|
||||
- CreateRecordingsBucket
|
||||
- Effect: Allow
|
||||
|
|
@ -1474,6 +1497,10 @@ Resources:
|
|||
Action:
|
||||
- secretsmanager:GetSecretValue
|
||||
Resource: !Ref OpenViduSharedInfo
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- ssm:GetParameter
|
||||
Resource: !Sub arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/openvidu/${AWS::StackName}/*
|
||||
- Effect: Allow
|
||||
Action:
|
||||
- autoscaling:SetInstanceHealth
|
||||
|
|
@ -1526,8 +1553,6 @@ Resources:
|
|||
|
||||
OpenViduMediaInstanceProfile:
|
||||
Type: AWS::IAM::InstanceProfile
|
||||
DependsOn:
|
||||
- MasterNodesWaitCondition4
|
||||
Properties:
|
||||
InstanceProfileName: !Sub OpenViduMediaInstanceProfile-${AWS::Region}-${AWS::StackName}
|
||||
Roles:
|
||||
|
|
@ -1566,42 +1591,15 @@ Resources:
|
|||
rm -rf awscliv2.zip aws
|
||||
fi
|
||||
|
||||
# Singal to notify instance is waiting
|
||||
# Signal to notify instance is waiting
|
||||
SIGNAL_NAME="$1"
|
||||
|
||||
# Exported earlier in this instance's UserData
|
||||
MASTER_NODE_NUM="${!MASTER_NODE_NUM}"
|
||||
|
||||
# Token for IMDSv2
|
||||
TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")"
|
||||
|
||||
# Subnets IDs
|
||||
SUBNETS=(
|
||||
"${SubnetProcessor.Subnet1}"
|
||||
"${SubnetProcessor.Subnet2}"
|
||||
"${SubnetProcessor.Subnet3}"
|
||||
"${SubnetProcessor.Subnet4}"
|
||||
)
|
||||
|
||||
MAC_ADDRESS="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254//latest/meta-data/mac)"
|
||||
SUBNET_ID="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s "http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDRESS/subnet-id")"
|
||||
|
||||
# Check master node number
|
||||
MASTER_NODE_NUM=1
|
||||
for subnet in "${!SUBNETS[@]}"; do
|
||||
if [[ "$subnet" == "$SUBNET_ID" ]]; then
|
||||
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text || echo 'none')
|
||||
|
||||
# Check if current master node is reacheable with ping
|
||||
ACUTAL_MASTER_NODE_IP=$(echo "$SHARED_SECRET" | jq -r ".MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP")
|
||||
if [[ "$ACUTAL_MASTER_NODE_IP" == "none" ]]; then
|
||||
break
|
||||
fi
|
||||
fi
|
||||
MASTER_NODE_NUM=$((MASTER_NODE_NUM + 1))
|
||||
done
|
||||
|
||||
# Get own private IP
|
||||
PRIVATE_IP="$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/local-ipv4)"
|
||||
|
||||
|
|
@ -1610,8 +1608,12 @@ Resources:
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# Store current private IP
|
||||
/usr/local/bin/store_secret.sh save MASTER_NODE_${!MASTER_NODE_NUM}_PRIVATE_IP "${!PRIVATE_IP}"
|
||||
aws ssm put-parameter \
|
||||
--region ${AWS::Region} \
|
||||
--name "/openvidu/${AWS::StackName}/master-node-${!MASTER_NODE_NUM}-private-ip" \
|
||||
--value "${!PRIVATE_IP}" \
|
||||
--type String \
|
||||
--overwrite > /dev/null
|
||||
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
|
|
@ -1690,39 +1692,39 @@ Resources:
|
|||
# sending the signal call
|
||||
cfn-signal -e $? --stack ${AWS::StackId} --resource "$SIGNAL_NAME" --region ${AWS::Region}
|
||||
|
||||
# Wait for all master nodes to store their private IPs
|
||||
IP_WAIT_ATTEMPTS=0
|
||||
IP_WAIT_MAX=360
|
||||
while true; do
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text || echo 'none')
|
||||
MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
|
||||
MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
|
||||
MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
|
||||
MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
|
||||
MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
|
||||
|
||||
# Check if all master nodes have stored their private IPs
|
||||
if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] &&
|
||||
[[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] &&
|
||||
[[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] &&
|
||||
[[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]]; then
|
||||
# Check if all master nodes have published their private IPs
|
||||
if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then
|
||||
break
|
||||
fi
|
||||
|
||||
IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1))
|
||||
if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then
|
||||
echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM"
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Comma-separated 1-4: must match installer's --master-node-private-ip-list format
|
||||
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
|
||||
|
||||
# Re-read: other (non-IP) values still come from the shared secret
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text)
|
||||
|
||||
MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
|
||||
MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
|
||||
MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
|
||||
MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
|
||||
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
|
||||
|
||||
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
|
||||
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
|
||||
OPENVIDU_RTC_ENGINE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_RTC_ENGINE')
|
||||
|
|
@ -1738,18 +1740,22 @@ Resources:
|
|||
GRAFANA_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_PASSWORD')
|
||||
MEET_INITIAL_ADMIN_USER=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_USER')
|
||||
MEET_INITIAL_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_PASSWORD')
|
||||
MEET_INITIAL_API_KEY=""
|
||||
if [[ "${InitialMeetApiKey}" != '' ]]; then
|
||||
MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
|
||||
else
|
||||
MEET_INITIAL_API_KEY=""
|
||||
fi
|
||||
MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
|
||||
LIVEKIT_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_KEY')
|
||||
LIVEKIT_API_SECRET=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_SECRET')
|
||||
ENABLED_MODULES=$(echo "$SHARED_SECRET" | jq -r '.ENABLED_MODULES')
|
||||
|
||||
# Base command
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)"
|
||||
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu master node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Common arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -1837,6 +1843,10 @@ Resources:
|
|||
content: !Sub |
|
||||
#!/bin/bash
|
||||
set -e
|
||||
# Only master 1 writes the shared secret, to avoid a lost-update race
|
||||
if [[ "${!MASTER_NODE_NUM:-}" != "1" ]]; then
|
||||
exit 0
|
||||
fi
|
||||
# Get current shared secret
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
|
|
@ -2103,6 +2113,8 @@ Resources:
|
|||
|
||||
OpenViduMasterNode1:
|
||||
Type: AWS::EC2::Instance
|
||||
# Depends on its own SSM parameter so its put-parameter can't race the parameter's creation
|
||||
DependsOn: MasterNode1PrivateIpParameter
|
||||
Properties:
|
||||
LaunchTemplate:
|
||||
LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
|
||||
|
|
@ -2133,6 +2145,8 @@ Resources:
|
|||
|
||||
cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
|
||||
|
||||
export MASTER_NODE_NUM=1
|
||||
|
||||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh "MasterNodesWaitCondition1" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
|
||||
|
||||
|
|
@ -2179,7 +2193,7 @@ Resources:
|
|||
|
||||
OpenViduMasterNode2:
|
||||
Type: AWS::EC2::Instance
|
||||
DependsOn: MasterNodesWaitCondition1
|
||||
DependsOn: MasterNode2PrivateIpParameter
|
||||
Properties:
|
||||
LaunchTemplate:
|
||||
LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
|
||||
|
|
@ -2210,6 +2224,8 @@ Resources:
|
|||
|
||||
cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
|
||||
|
||||
export MASTER_NODE_NUM=2
|
||||
|
||||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh "MasterNodesWaitCondition2" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
|
||||
|
||||
|
|
@ -2256,7 +2272,7 @@ Resources:
|
|||
|
||||
OpenViduMasterNode3:
|
||||
Type: AWS::EC2::Instance
|
||||
DependsOn: MasterNodesWaitCondition2
|
||||
DependsOn: MasterNode3PrivateIpParameter
|
||||
Properties:
|
||||
LaunchTemplate:
|
||||
LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
|
||||
|
|
@ -2287,6 +2303,8 @@ Resources:
|
|||
|
||||
cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
|
||||
|
||||
export MASTER_NODE_NUM=3
|
||||
|
||||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh "MasterNodesWaitCondition3" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
|
||||
|
||||
|
|
@ -2333,7 +2351,7 @@ Resources:
|
|||
|
||||
OpenViduMasterNode4:
|
||||
Type: AWS::EC2::Instance
|
||||
DependsOn: MasterNodesWaitCondition3
|
||||
DependsOn: MasterNode4PrivateIpParameter
|
||||
Properties:
|
||||
LaunchTemplate:
|
||||
LaunchTemplateId: !Ref OpenViduMasterLaunchTemplate
|
||||
|
|
@ -2364,6 +2382,8 @@ Resources:
|
|||
|
||||
cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMasterLaunchTemplate
|
||||
|
||||
export MASTER_NODE_NUM=4
|
||||
|
||||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh "MasterNodesWaitCondition4" || { echo "[OpenVidu] error installing OpenVidu"; exit 1; }
|
||||
|
||||
|
|
@ -2453,6 +2473,25 @@ Resources:
|
|||
exit 1
|
||||
fi
|
||||
|
||||
SECRETS_WAIT_ATTEMPTS=0
|
||||
SECRETS_WAIT_MAX=360
|
||||
while true; do
|
||||
ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
|
||||
if [[ "$ALL_SECRETS_GENERATED" == "true" ]]; then
|
||||
break
|
||||
fi
|
||||
SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1))
|
||||
if [[ $SECRETS_WAIT_ATTEMPTS -ge $SECRETS_WAIT_MAX ]]; then
|
||||
echo "Error: timed out after 30 minutes waiting for shared secrets to be generated"
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
SHARED_SECRET="$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text || echo 'none')"
|
||||
done
|
||||
|
||||
# Get OpenVidu Media Nodes version to deploy
|
||||
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||
if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
||||
|
|
@ -2460,20 +2499,39 @@ Resources:
|
|||
exit 1
|
||||
fi
|
||||
|
||||
ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
|
||||
if [[ "$ALL_SECRETS_GENERATED" == "none" ]]; then
|
||||
echo "Error: Secrets not generated"
|
||||
exit 1
|
||||
fi
|
||||
MASTER_NODE_1_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_1_PRIVATE_IP')
|
||||
MASTER_NODE_2_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_2_PRIVATE_IP')
|
||||
MASTER_NODE_3_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_3_PRIVATE_IP')
|
||||
MASTER_NODE_4_PRIVATE_IP=$(echo "$SHARED_SECRET" | jq -r '.MASTER_NODE_4_PRIVATE_IP')
|
||||
IP_WAIT_ATTEMPTS=0
|
||||
IP_WAIT_MAX=360
|
||||
while true; do
|
||||
MASTER_NODE_1_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_2_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_3_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_NODE_4_PRIVATE_IP=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
|
||||
if [[ "$MASTER_NODE_1_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_1_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_2_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_2_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_3_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_3_PRIVATE_IP" ]] &&
|
||||
[[ "$MASTER_NODE_4_PRIVATE_IP" != "none" ]] && [[ -n "$MASTER_NODE_4_PRIVATE_IP" ]]; then
|
||||
break
|
||||
fi
|
||||
|
||||
IP_WAIT_ATTEMPTS=$((IP_WAIT_ATTEMPTS + 1))
|
||||
if [[ $IP_WAIT_ATTEMPTS -ge $IP_WAIT_MAX ]]; then
|
||||
echo "Error: timed out after 30 minutes waiting for all master nodes to publish their private IPs to SSM"
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
|
||||
REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
|
||||
|
||||
# Base command
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)"
|
||||
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu media node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Common arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -2609,13 +2667,36 @@ Resources:
|
|||
python3 -m pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-py3-${!CFN_BOOTSTRAP_VERSION}.tar.gz
|
||||
fi
|
||||
|
||||
cfn-init --region ${AWS::Region} --stack ${AWS::StackId} --resource OpenViduMediaNodeLaunchTemplate
|
||||
cfn-init --region ${AWS::Region} --stack ${AWS::StackName} --resource OpenViduMediaNodeLaunchTemplate
|
||||
|
||||
export HOME="/root"
|
||||
|
||||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||
|
||||
MASTER_IP_1=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-1-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_IP_2=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-2-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_IP_3=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-3-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_IP_4=$(aws ssm get-parameter --region ${AWS::Region} --name "/openvidu/${AWS::StackName}/master-node-4-private-ip" --query Parameter.Value --output text 2>/dev/null || echo 'none')
|
||||
MASTER_HEALTHY=false
|
||||
for i in $(seq 1 360); do
|
||||
for MASTER_IP in "$MASTER_IP_1" "$MASTER_IP_2" "$MASTER_IP_3" "$MASTER_IP_4"; do
|
||||
if [ "$MASTER_IP" != "none" ] && [ -n "$MASTER_IP" ] && curl -sf "http://$MASTER_IP:7880/health/caddy" >/dev/null 2>&1; then
|
||||
MASTER_HEALTHY=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
if [ "$MASTER_HEALTHY" = "true" ]; then
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
if [ "$MASTER_HEALTHY" != "true" ]; then
|
||||
echo "[OpenVidu] no master node became healthy after 30 minutes"
|
||||
/usr/local/bin/set_as_unhealthy.sh
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Start OpenVidu
|
||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||
|
||||
|
|
@ -3191,8 +3272,6 @@ Resources:
|
|||
|
||||
LoadBalancer:
|
||||
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
|
||||
DependsOn:
|
||||
- MasterNodesWaitCondition4
|
||||
Properties:
|
||||
Name:
|
||||
Fn::Join:
|
||||
|
|
|
|||
Loading…
Reference in New Issue