mirror of https://github.com/OpenVidu/openvidu.git
openvidu-deployment: harden AWS elastic deployment
WaitCondition PT10M -> PT20M (signal fires after the full install), robust installer fetch (curl --retry to file), bounded master readiness gate with one restart retry feeding cfn-signal, media waits for secrets with content validation (read-until-valid, eventually-consistent reads) and for a healthy master before starting instead of self-destructing, idempotent secret generation with post-guard read-back, vestigial IAM retry loop and duplicated saves removed, ASG DependsOn completed, DOMAIN_NAME persisted explicitly. Validated with ov-cloud-tester (sc-deploy-destroy, elastic, dev): PASS. deploy 6m49s, wait-ready 32s, destroy 3m37s.master^2
Piwccle
parent
73f8b57498
commit
20eef140c5
|
|
@ -0,0 +1,50 @@
|
||||||
|
# Doc changes for `elastic-hardening` (OpenVidu Elastic, AWS)
|
||||||
|
|
||||||
|
Target docs repo: `openvidu.io`, branch `next`
|
||||||
|
Docs affected: `docs/docs/self-hosting/elastic/aws/*.md`
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Hardening of `pro/elastic/aws/cf-openvidu-elastic.yaml` (deployment repo, branch
|
||||||
|
`elastic-hardening`). All changes are internal robustness improvements to the
|
||||||
|
CloudFormation template's bootstrap scripts: robust installer download, bounded
|
||||||
|
readiness gates on master and media nodes, an idempotency guard around master
|
||||||
|
secret generation, and a wider `WaitCondition` timeout (`PT10M` -> `PT20M`).
|
||||||
|
|
||||||
|
## Impact on the docs: essentially none
|
||||||
|
|
||||||
|
No user-facing surface changes. Specifically:
|
||||||
|
|
||||||
|
- **Parameters**: unchanged. No parameter added, removed, or renamed
|
||||||
|
(`install.md#cloudformation-parameters` needs no edit).
|
||||||
|
- **Outputs**: unchanged. `ServicesAndCredentials` and the rest of the
|
||||||
|
Outputs section are identical (`install.md`).
|
||||||
|
- **Times / durations**: no documented time changes. The `WaitCondition`
|
||||||
|
timeout is an internal upper bound for failure detection, not a documented
|
||||||
|
deployment duration; success still signals as soon as the node is healthy.
|
||||||
|
`install.md` does not state a fixed deployment time, so nothing to update.
|
||||||
|
- **Install / upgrade flow**: unchanged (`install.md`, `upgrade.md`).
|
||||||
|
|
||||||
|
No screenshots need to be retaken.
|
||||||
|
|
||||||
|
## Optional precision for `admin.md`
|
||||||
|
|
||||||
|
Section "Administration and configuration" -> "Changing Configuration through
|
||||||
|
AWS Secrets". After editing the secret and rebooting, the doc currently states
|
||||||
|
(around line 237):
|
||||||
|
|
||||||
|
> Changes will be applied automatically in all the nodes of your OpenVidu Elastic deployment.
|
||||||
|
|
||||||
|
This is imprecise for Elastic. Only the **master node** re-reads the secret on
|
||||||
|
reboot (its `@reboot` cron runs `update_config_from_secret.sh`). **Media nodes**
|
||||||
|
read the shared secret only at launch, so they do not pick up secret changes on
|
||||||
|
reboot. Suggested replacement:
|
||||||
|
|
||||||
|
> Rebooting the Master Node re-applies the configuration on the master. Media
|
||||||
|
> Nodes read the shared configuration only when they start, so to propagate
|
||||||
|
> changes that affect Media Nodes you must relaunch them (terminate the running
|
||||||
|
> Media Nodes; the Auto Scaling Group launches replacements that read the
|
||||||
|
> updated configuration).
|
||||||
|
|
||||||
|
This is a pre-existing behavior clarification, not a consequence of the
|
||||||
|
hardening changes; include it only if a docs pass is being made anyway.
|
||||||
|
|
@ -1427,7 +1427,8 @@ Resources:
|
||||||
Type: 'AWS::CloudFormation::WaitCondition'
|
Type: 'AWS::CloudFormation::WaitCondition'
|
||||||
CreationPolicy:
|
CreationPolicy:
|
||||||
ResourceSignal:
|
ResourceSignal:
|
||||||
Timeout: PT10M
|
# Elastic signals AFTER the full install completes, so it needs a wider window than HA
|
||||||
|
Timeout: PT20M
|
||||||
Count: '1'
|
Count: '1'
|
||||||
|
|
||||||
OpenViduMasterNode:
|
OpenViduMasterNode:
|
||||||
|
|
@ -1473,69 +1474,110 @@ Resources:
|
||||||
else
|
else
|
||||||
DOMAIN=${DomainName}
|
DOMAIN=${DomainName}
|
||||||
fi
|
fi
|
||||||
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
|
||||||
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
|
||||||
# Store version so media nodes can use it to install the same version
|
|
||||||
/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "${!OPENVIDU_VERSION}"
|
|
||||||
|
|
||||||
# Get own private IP
|
# Get own private IP
|
||||||
PRIVATE_IP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4)
|
PRIVATE_IP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||||
|
|
||||||
# Unfortunately, EC2 instance assigned role is not immediately available after the instance is launched.
|
# Instance role already proven by cfn-init above, so a single read needs no IAM-availability retry
|
||||||
# Therefore, we need to retry this operation until the aws-cli command is successful.
|
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||||
MAX_RETRIES=10
|
--region ${AWS::Region} \
|
||||||
RETRY_COUNT=0
|
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||||
while : ; do
|
--query SecretString --output text || echo 'none')
|
||||||
# Get current shared secret and random seed
|
if [[ "$SHARED_SECRET" == "none" ]]; then
|
||||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
echo "Error: Shared secret not found"
|
||||||
--region ${AWS::Region} \
|
exit 1
|
||||||
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
fi
|
||||||
--query SecretString --output text || echo 'none')
|
ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
|
||||||
|
|
||||||
if [[ "$SHARED_SECRET" != "none" ]]; then
|
# Generate secrets only once: a re-run must never regenerate passwords already in use by the cluster
|
||||||
break
|
if [[ "$ALL_SECRETS_GENERATED" == "false" ]]; then
|
||||||
fi
|
# Persist the domain so after_install.sh reads it from the secret instead of the installer
|
||||||
|
/usr/local/bin/store_secret.sh save DOMAIN_NAME "$DOMAIN"
|
||||||
|
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
||||||
|
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
||||||
|
# Store version so media nodes can use it to install the same version
|
||||||
|
/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "${!OPENVIDU_VERSION}"
|
||||||
|
|
||||||
RETRY_COUNT=$((RETRY_COUNT+1))
|
# Meet initial admin user and password
|
||||||
if [[ $RETRY_COUNT -ge $MAX_RETRIES ]]; then
|
MEET_INITIAL_ADMIN_USER="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_ADMIN_USER "admin")"
|
||||||
exit 1
|
if [[ "${InitialMeetAdminPassword}" != '' ]]; then
|
||||||
fi
|
MEET_INITIAL_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_ADMIN_PASSWORD "${InitialMeetAdminPassword}")"
|
||||||
sleep 6
|
else
|
||||||
|
MEET_INITIAL_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate MEET_INITIAL_ADMIN_PASSWORD)"
|
||||||
|
fi
|
||||||
|
if [[ "${InitialMeetApiKey}" != '' ]]; then
|
||||||
|
MEET_INITIAL_API_KEY="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_API_KEY "${InitialMeetApiKey}")"
|
||||||
|
else
|
||||||
|
MEET_INITIAL_API_KEY="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_API_KEY "")"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Store usernames and generate random passwords
|
||||||
|
REDIS_PASSWORD="$(/usr/local/bin/store_secret.sh generate REDIS_PASSWORD)"
|
||||||
|
MONGO_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save MONGO_ADMIN_USERNAME "mongoadmin")"
|
||||||
|
MONGO_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate MONGO_ADMIN_PASSWORD)"
|
||||||
|
MONGO_REPLICA_SET_KEY="$(/usr/local/bin/store_secret.sh generate MONGO_REPLICA_SET_KEY)"
|
||||||
|
MINIO_ACCESS_KEY="$(/usr/local/bin/store_secret.sh save MINIO_ACCESS_KEY "minioadmin")"
|
||||||
|
MINIO_SECRET_KEY="$(/usr/local/bin/store_secret.sh generate MINIO_SECRET_KEY)"
|
||||||
|
DASHBOARD_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save DASHBOARD_ADMIN_USERNAME "dashboardadmin")"
|
||||||
|
DASHBOARD_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate DASHBOARD_ADMIN_PASSWORD)"
|
||||||
|
GRAFANA_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save GRAFANA_ADMIN_USERNAME "grafanaadmin")"
|
||||||
|
GRAFANA_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate GRAFANA_ADMIN_PASSWORD)"
|
||||||
|
LIVEKIT_API_KEY="$(/usr/local/bin/store_secret.sh generate LIVEKIT_API_KEY "API" 12)"
|
||||||
|
LIVEKIT_API_SECRET="$(/usr/local/bin/store_secret.sh generate LIVEKIT_API_SECRET)"
|
||||||
|
ENABLED_MODULES="$(/usr/local/bin/store_secret.sh save ENABLED_MODULES "observability,v2compatibility,openviduMeet")"
|
||||||
|
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Source every installer value from the secret so a re-run (guard skipped) still has them
|
||||||
|
# GetSecretValue is eventually consistent: retry (up to 300s at 5s) until the read returns the generated values
|
||||||
|
SECRET_READ_ATTEMPTS=0
|
||||||
|
while true; do
|
||||||
|
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||||
|
--region ${AWS::Region} \
|
||||||
|
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||||
|
--query SecretString --output text)
|
||||||
|
if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and ([.DOMAIN_NAME, .OPENVIDU_VERSION, .OPENVIDU_PRO_LICENSE, .REDIS_PASSWORD, .MONGO_ADMIN_PASSWORD, .MONGO_REPLICA_SET_KEY, .MINIO_SECRET_KEY, .DASHBOARD_ADMIN_PASSWORD, .GRAFANA_ADMIN_PASSWORD, .LIVEKIT_API_KEY, .LIVEKIT_API_SECRET] | all(. != "none"))' > /dev/null; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
SECRET_READ_ATTEMPTS=$((SECRET_READ_ATTEMPTS + 1))
|
||||||
|
if [[ $SECRET_READ_ATTEMPTS -ge 60 ]]; then
|
||||||
|
echo "Error: shared secret still incomplete after 5 minutes of stale reads"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
done
|
done
|
||||||
|
|
||||||
# Meet initial admin user and password
|
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
|
||||||
MEET_INITIAL_ADMIN_USER="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_ADMIN_USER "admin")"
|
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||||
if [[ "${InitialMeetAdminPassword}" != '' ]]; then
|
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
|
||||||
MEET_INITIAL_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_ADMIN_PASSWORD "${InitialMeetAdminPassword}")"
|
OPENVIDU_RTC_ENGINE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_RTC_ENGINE')
|
||||||
else
|
MEET_INITIAL_ADMIN_USER=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_USER')
|
||||||
MEET_INITIAL_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate MEET_INITIAL_ADMIN_PASSWORD)"
|
MEET_INITIAL_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_PASSWORD')
|
||||||
fi
|
MEET_INITIAL_API_KEY=""
|
||||||
if [[ "${InitialMeetApiKey}" != '' ]]; then
|
if [[ "${InitialMeetApiKey}" != '' ]]; then
|
||||||
MEET_INITIAL_API_KEY="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_API_KEY "${InitialMeetApiKey}")"
|
MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
|
||||||
else
|
|
||||||
MEET_INITIAL_API_KEY="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_API_KEY "")"
|
|
||||||
fi
|
fi
|
||||||
|
REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
|
||||||
|
MONGO_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.MONGO_ADMIN_USERNAME')
|
||||||
|
MONGO_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MONGO_ADMIN_PASSWORD')
|
||||||
|
MONGO_REPLICA_SET_KEY=$(echo "$SHARED_SECRET" | jq -r '.MONGO_REPLICA_SET_KEY')
|
||||||
|
MINIO_ACCESS_KEY=$(echo "$SHARED_SECRET" | jq -r '.MINIO_ACCESS_KEY')
|
||||||
|
MINIO_SECRET_KEY=$(echo "$SHARED_SECRET" | jq -r '.MINIO_SECRET_KEY')
|
||||||
|
DASHBOARD_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.DASHBOARD_ADMIN_USERNAME')
|
||||||
|
DASHBOARD_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.DASHBOARD_ADMIN_PASSWORD')
|
||||||
|
GRAFANA_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_USERNAME')
|
||||||
|
GRAFANA_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_PASSWORD')
|
||||||
|
LIVEKIT_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_KEY')
|
||||||
|
LIVEKIT_API_SECRET=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_SECRET')
|
||||||
|
ENABLED_MODULES=$(echo "$SHARED_SECRET" | jq -r '.ENABLED_MODULES')
|
||||||
|
|
||||||
# Store usernames and generate random passwords
|
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||||
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
|
||||||
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_master_node.sh"
|
||||||
REDIS_PASSWORD="$(/usr/local/bin/store_secret.sh generate REDIS_PASSWORD)"
|
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||||
MONGO_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save MONGO_ADMIN_USERNAME "mongoadmin")"
|
echo "Downloaded OpenVidu master node installer is empty or missing" >&2
|
||||||
MONGO_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate MONGO_ADMIN_PASSWORD)"
|
exit 1
|
||||||
MONGO_REPLICA_SET_KEY="$(/usr/local/bin/store_secret.sh generate MONGO_REPLICA_SET_KEY)"
|
fi
|
||||||
MINIO_ACCESS_KEY="$(/usr/local/bin/store_secret.sh save MINIO_ACCESS_KEY "minioadmin")"
|
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||||
MINIO_SECRET_KEY="$(/usr/local/bin/store_secret.sh generate MINIO_SECRET_KEY)"
|
|
||||||
DASHBOARD_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save DASHBOARD_ADMIN_USERNAME "dashboardadmin")"
|
|
||||||
DASHBOARD_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate DASHBOARD_ADMIN_PASSWORD)"
|
|
||||||
GRAFANA_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save GRAFANA_ADMIN_USERNAME "grafanaadmin")"
|
|
||||||
GRAFANA_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate GRAFANA_ADMIN_PASSWORD)"
|
|
||||||
LIVEKIT_API_KEY="$(/usr/local/bin/store_secret.sh generate LIVEKIT_API_KEY "API" 12)"
|
|
||||||
LIVEKIT_API_SECRET="$(/usr/local/bin/store_secret.sh generate LIVEKIT_API_SECRET)"
|
|
||||||
ENABLED_MODULES="$(/usr/local/bin/store_secret.sh save ENABLED_MODULES "observability,v2compatibility,openviduMeet")"
|
|
||||||
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")"
|
|
||||||
|
|
||||||
# Base command
|
|
||||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_master_node.sh)"
|
|
||||||
|
|
||||||
# Common arguments
|
# Common arguments
|
||||||
COMMON_ARGS=(
|
COMMON_ARGS=(
|
||||||
|
|
@ -1858,20 +1900,6 @@ Resources:
|
||||||
mode: "000755"
|
mode: "000755"
|
||||||
owner: "root"
|
owner: "root"
|
||||||
group: "root"
|
group: "root"
|
||||||
'/usr/local/bin/check_app_ready.sh':
|
|
||||||
content: |
|
|
||||||
#!/bin/bash
|
|
||||||
set -e
|
|
||||||
while true; do
|
|
||||||
HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy | head -n1 | awk '{print $2}')
|
|
||||||
if [ $HTTP_STATUS == 200 ]; then
|
|
||||||
break
|
|
||||||
fi
|
|
||||||
sleep 5
|
|
||||||
done
|
|
||||||
mode: "000755"
|
|
||||||
owner: "root"
|
|
||||||
group: "root"
|
|
||||||
'/usr/local/bin/restart.sh':
|
'/usr/local/bin/restart.sh':
|
||||||
content: |
|
content: |
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|
@ -1950,9 +1978,30 @@ Resources:
|
||||||
# Launch on reboot
|
# Launch on reboot
|
||||||
echo "@reboot /usr/local/bin/restart.sh &> /var/log/openvidu-restart.log" | crontab
|
echo "@reboot /usr/local/bin/restart.sh &> /var/log/openvidu-restart.log" | crontab
|
||||||
|
|
||||||
# Wait for the app
|
# Local readiness gate: wait up to 300s for Caddy health, restart once if it does not converge
|
||||||
sleep 20
|
OPENVIDU_READY=false
|
||||||
/usr/local/bin/check_app_ready.sh
|
for i in $(seq 1 60); do
|
||||||
|
if curl -fsS http://127.0.0.1:7880/health/caddy >/dev/null 2>&1; then
|
||||||
|
OPENVIDU_READY=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
if [ "$OPENVIDU_READY" != "true" ]; then
|
||||||
|
echo "[OpenVidu] not healthy after 300s, restarting once"
|
||||||
|
systemctl restart openvidu || true
|
||||||
|
for i in $(seq 1 60); do
|
||||||
|
if curl -fsS http://127.0.0.1:7880/health/caddy >/dev/null 2>&1; then
|
||||||
|
OPENVIDU_READY=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Disable errexit so cfn-signal always runs and reports readiness via $? (fast-fail instead of a PT20M timeout)
|
||||||
|
set +e
|
||||||
|
[ "$OPENVIDU_READY" = "true" ]
|
||||||
|
|
||||||
# sending the finish call
|
# sending the finish call
|
||||||
cfn-signal -e $? --stack ${AWS::StackId} --resource WaitCondition --region ${AWS::Region}
|
cfn-signal -e $? --stack ${AWS::StackId} --resource WaitCondition --region ${AWS::Region}
|
||||||
|
|
@ -2022,31 +2071,48 @@ Resources:
|
||||||
sleep 6
|
sleep 6
|
||||||
done
|
done
|
||||||
|
|
||||||
# Get current shared secret
|
# Wait until the master node has generated all shared secrets.
|
||||||
DOMAIN=$(echo $SHARED_SECRET | jq -r .DOMAIN_NAME)
|
# Bounded (up to 1800s at 5s), validating content on each snapshot: stale eventually-consistent reads can return pre-generation values
|
||||||
OPENVIDU_PRO_LICENSE=$(echo $SHARED_SECRET | jq -r .OPENVIDU_PRO_LICENSE)
|
SECRETS_WAIT_ATTEMPTS=0
|
||||||
REDIS_PASSWORD=$(echo $SHARED_SECRET | jq -r .REDIS_PASSWORD)
|
SECRETS_WAIT_MAX=360
|
||||||
|
while true; do
|
||||||
|
if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and (.OPENVIDU_VERSION != "none") and (.REDIS_PASSWORD != "none")' > /dev/null; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1))
|
||||||
|
if [[ $SECRETS_WAIT_ATTEMPTS -ge $SECRETS_WAIT_MAX ]]; then
|
||||||
|
echo "Error: timed out after 30 minutes waiting for the master node to generate the shared secrets"
|
||||||
|
/usr/local/bin/set_as_unhealthy.sh
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||||
|
--region ${AWS::Region} \
|
||||||
|
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||||
|
--query SecretString --output text || echo 'none')
|
||||||
|
done
|
||||||
|
|
||||||
# Get OpenVidu Media Nodes version to deploy
|
# Read values only after the gate confirms the master node has published them
|
||||||
|
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
|
||||||
|
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
|
||||||
|
REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
|
||||||
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||||
|
|
||||||
if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
||||||
echo "OpenVidu version not found"
|
echo "OpenVidu version not found"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
ALL_SECRETS_GENERATED=$(echo $SHARED_SECRET | jq -r .ALL_SECRETS_GENERATED)
|
|
||||||
if [[ "$ALL_SECRETS_GENERATED" == "false" ]]; then
|
|
||||||
echo "Master node not ready"
|
|
||||||
/usr/local/bin/set_as_unhealthy.sh
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Get Master Node private IP
|
# Get Master Node private IP
|
||||||
MASTER_NODE_IP=${OpenViduMasterNode.PrivateIp}
|
MASTER_NODE_IP=${OpenViduMasterNode.PrivateIp}
|
||||||
|
|
||||||
# Base command
|
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_media_node.sh)"
|
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
|
||||||
|
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_media_node.sh"
|
||||||
|
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||||
|
echo "Downloaded OpenVidu media node installer is empty or missing" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||||
|
|
||||||
# Common arguments
|
# Common arguments
|
||||||
COMMON_ARGS=(
|
COMMON_ARGS=(
|
||||||
|
|
@ -2190,6 +2256,22 @@ Resources:
|
||||||
# Install OpenVidu
|
# Install OpenVidu
|
||||||
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||||
|
|
||||||
|
# Wait for the master node to be healthy before starting (bounded: up to 1800s at 5s)
|
||||||
|
MASTER_NODE_IP=${OpenViduMasterNode.PrivateIp}
|
||||||
|
MASTER_HEALTHY=false
|
||||||
|
for i in $(seq 1 360); do
|
||||||
|
if curl -sf "http://$MASTER_NODE_IP:7880/health/caddy" >/dev/null 2>&1; then
|
||||||
|
MASTER_HEALTHY=true
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
if [ "$MASTER_HEALTHY" != "true" ]; then
|
||||||
|
echo "[OpenVidu] master node did not become healthy after 30 minutes"
|
||||||
|
/usr/local/bin/set_as_unhealthy.sh
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
# Start OpenVidu
|
# Start OpenVidu
|
||||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||||
|
|
||||||
|
|
@ -2202,6 +2284,8 @@ Resources:
|
||||||
|
|
||||||
OpenViduMediaNodeASG:
|
OpenViduMediaNodeASG:
|
||||||
DependsOn:
|
DependsOn:
|
||||||
|
- OpenViduMediaNodeInstanceProfile
|
||||||
|
- OpenViduMasterNodeInstanceProfile
|
||||||
- StopMediaNodeCloudWatchEventRule
|
- StopMediaNodeCloudWatchEventRule
|
||||||
Type: AWS::AutoScaling::AutoScalingGroup
|
Type: AWS::AutoScaling::AutoScalingGroup
|
||||||
Properties:
|
Properties:
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue