mirror of https://github.com/OpenVidu/openvidu.git
openvidu-deployment: harden AWS elastic deployment
WaitCondition PT10M -> PT20M (signal fires after the full install), robust installer fetch (curl --retry to file), bounded master readiness gate with one restart retry feeding cfn-signal, media waits for secrets with content validation (read-until-valid, eventually-consistent reads) and for a healthy master before starting instead of self-destructing, idempotent secret generation with post-guard read-back, vestigial IAM retry loop and duplicated saves removed, ASG DependsOn completed, DOMAIN_NAME persisted explicitly. Validated with ov-cloud-tester (sc-deploy-destroy, elastic, dev): PASS. deploy 6m49s, wait-ready 32s, destroy 3m37s.master^2
Piwccle
parent
73f8b57498
commit
20eef140c5
|
|
@ -0,0 +1,50 @@
|
|||
# Doc changes for `elastic-hardening` (OpenVidu Elastic, AWS)
|
||||
|
||||
Target docs repo: `openvidu.io`, branch `next`
|
||||
Docs affected: `docs/docs/self-hosting/elastic/aws/*.md`
|
||||
|
||||
## Context
|
||||
|
||||
Hardening of `pro/elastic/aws/cf-openvidu-elastic.yaml` (deployment repo, branch
|
||||
`elastic-hardening`). All changes are internal robustness improvements to the
|
||||
CloudFormation template's bootstrap scripts: robust installer download, bounded
|
||||
readiness gates on master and media nodes, an idempotency guard around master
|
||||
secret generation, and a wider `WaitCondition` timeout (`PT10M` -> `PT20M`).
|
||||
|
||||
## Impact on the docs: essentially none
|
||||
|
||||
No user-facing surface changes. Specifically:
|
||||
|
||||
- **Parameters**: unchanged. No parameter added, removed, or renamed
|
||||
(`install.md#cloudformation-parameters` needs no edit).
|
||||
- **Outputs**: unchanged. `ServicesAndCredentials` and the rest of the
|
||||
Outputs section are identical (`install.md`).
|
||||
- **Times / durations**: no documented time changes. The `WaitCondition`
|
||||
timeout is an internal upper bound for failure detection, not a documented
|
||||
deployment duration; success still signals as soon as the node is healthy.
|
||||
`install.md` does not state a fixed deployment time, so nothing to update.
|
||||
- **Install / upgrade flow**: unchanged (`install.md`, `upgrade.md`).
|
||||
|
||||
No screenshots need to be retaken.
|
||||
|
||||
## Optional precision for `admin.md`
|
||||
|
||||
Section "Administration and configuration" -> "Changing Configuration through
|
||||
AWS Secrets". After editing the secret and rebooting, the doc currently states
|
||||
(around line 237):
|
||||
|
||||
> Changes will be applied automatically in all the nodes of your OpenVidu Elastic deployment.
|
||||
|
||||
This is imprecise for Elastic. Only the **master node** re-reads the secret on
|
||||
reboot (its `@reboot` cron runs `update_config_from_secret.sh`). **Media nodes**
|
||||
read the shared secret only at launch, so they do not pick up secret changes on
|
||||
reboot. Suggested replacement:
|
||||
|
||||
> Rebooting the Master Node re-applies the configuration on the master. Media
|
||||
> Nodes read the shared configuration only when they start, so to propagate
|
||||
> changes that affect Media Nodes you must relaunch them (terminate the running
|
||||
> Media Nodes; the Auto Scaling Group launches replacements that read the
|
||||
> updated configuration).
|
||||
|
||||
This is a pre-existing behavior clarification, not a consequence of the
|
||||
hardening changes; include it only if a docs pass is being made anyway.
|
||||
|
|
@ -1427,7 +1427,8 @@ Resources:
|
|||
Type: 'AWS::CloudFormation::WaitCondition'
|
||||
CreationPolicy:
|
||||
ResourceSignal:
|
||||
Timeout: PT10M
|
||||
# Elastic signals AFTER the full install completes, so it needs a wider window than HA
|
||||
Timeout: PT20M
|
||||
Count: '1'
|
||||
|
||||
OpenViduMasterNode:
|
||||
|
|
@ -1473,35 +1474,28 @@ Resources:
|
|||
else
|
||||
DOMAIN=${DomainName}
|
||||
fi
|
||||
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
||||
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
||||
# Store version so media nodes can use it to install the same version
|
||||
/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "${!OPENVIDU_VERSION}"
|
||||
|
||||
# Get own private IP
|
||||
PRIVATE_IP=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/local-ipv4)
|
||||
|
||||
# Unfortunately, EC2 instance assigned role is not immediately available after the instance is launched.
|
||||
# Therefore, we need to retry this operation until the aws-cli command is successful.
|
||||
MAX_RETRIES=10
|
||||
RETRY_COUNT=0
|
||||
while : ; do
|
||||
# Get current shared secret and random seed
|
||||
# Instance role already proven by cfn-init above, so a single read needs no IAM-availability retry
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text || echo 'none')
|
||||
|
||||
if [[ "$SHARED_SECRET" != "none" ]]; then
|
||||
break
|
||||
fi
|
||||
|
||||
RETRY_COUNT=$((RETRY_COUNT+1))
|
||||
if [[ $RETRY_COUNT -ge $MAX_RETRIES ]]; then
|
||||
if [[ "$SHARED_SECRET" == "none" ]]; then
|
||||
echo "Error: Shared secret not found"
|
||||
exit 1
|
||||
fi
|
||||
sleep 6
|
||||
done
|
||||
ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED')
|
||||
|
||||
# Generate secrets only once: a re-run must never regenerate passwords already in use by the cluster
|
||||
if [[ "$ALL_SECRETS_GENERATED" == "false" ]]; then
|
||||
# Persist the domain so after_install.sh reads it from the secret instead of the installer
|
||||
/usr/local/bin/store_secret.sh save DOMAIN_NAME "$DOMAIN"
|
||||
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
||||
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
||||
# Store version so media nodes can use it to install the same version
|
||||
/usr/local/bin/store_secret.sh save OPENVIDU_VERSION "${!OPENVIDU_VERSION}"
|
||||
|
||||
# Meet initial admin user and password
|
||||
MEET_INITIAL_ADMIN_USER="$(/usr/local/bin/store_secret.sh save MEET_INITIAL_ADMIN_USER "admin")"
|
||||
|
|
@ -1517,8 +1511,6 @@ Resources:
|
|||
fi
|
||||
|
||||
# Store usernames and generate random passwords
|
||||
OPENVIDU_PRO_LICENSE="$(/usr/local/bin/store_secret.sh save OPENVIDU_PRO_LICENSE "${OpenViduLicense}")"
|
||||
OPENVIDU_RTC_ENGINE="$(/usr/local/bin/store_secret.sh save OPENVIDU_RTC_ENGINE "${RTCEngine}")"
|
||||
REDIS_PASSWORD="$(/usr/local/bin/store_secret.sh generate REDIS_PASSWORD)"
|
||||
MONGO_ADMIN_USERNAME="$(/usr/local/bin/store_secret.sh save MONGO_ADMIN_USERNAME "mongoadmin")"
|
||||
MONGO_ADMIN_PASSWORD="$(/usr/local/bin/store_secret.sh generate MONGO_ADMIN_PASSWORD)"
|
||||
|
|
@ -1533,9 +1525,59 @@ Resources:
|
|||
LIVEKIT_API_SECRET="$(/usr/local/bin/store_secret.sh generate LIVEKIT_API_SECRET)"
|
||||
ENABLED_MODULES="$(/usr/local/bin/store_secret.sh save ENABLED_MODULES "observability,v2compatibility,openviduMeet")"
|
||||
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")"
|
||||
fi
|
||||
|
||||
# Base command
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_master_node.sh)"
|
||||
# Source every installer value from the secret so a re-run (guard skipped) still has them
|
||||
# GetSecretValue is eventually consistent: retry (up to 300s at 5s) until the read returns the generated values
|
||||
SECRET_READ_ATTEMPTS=0
|
||||
while true; do
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text)
|
||||
if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and ([.DOMAIN_NAME, .OPENVIDU_VERSION, .OPENVIDU_PRO_LICENSE, .REDIS_PASSWORD, .MONGO_ADMIN_PASSWORD, .MONGO_REPLICA_SET_KEY, .MINIO_SECRET_KEY, .DASHBOARD_ADMIN_PASSWORD, .GRAFANA_ADMIN_PASSWORD, .LIVEKIT_API_KEY, .LIVEKIT_API_SECRET] | all(. != "none"))' > /dev/null; then
|
||||
break
|
||||
fi
|
||||
SECRET_READ_ATTEMPTS=$((SECRET_READ_ATTEMPTS + 1))
|
||||
if [[ $SECRET_READ_ATTEMPTS -ge 60 ]]; then
|
||||
echo "Error: shared secret still incomplete after 5 minutes of stale reads"
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
|
||||
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
|
||||
OPENVIDU_RTC_ENGINE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_RTC_ENGINE')
|
||||
MEET_INITIAL_ADMIN_USER=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_USER')
|
||||
MEET_INITIAL_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_ADMIN_PASSWORD')
|
||||
MEET_INITIAL_API_KEY=""
|
||||
if [[ "${InitialMeetApiKey}" != '' ]]; then
|
||||
MEET_INITIAL_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.MEET_INITIAL_API_KEY')
|
||||
fi
|
||||
REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
|
||||
MONGO_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.MONGO_ADMIN_USERNAME')
|
||||
MONGO_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.MONGO_ADMIN_PASSWORD')
|
||||
MONGO_REPLICA_SET_KEY=$(echo "$SHARED_SECRET" | jq -r '.MONGO_REPLICA_SET_KEY')
|
||||
MINIO_ACCESS_KEY=$(echo "$SHARED_SECRET" | jq -r '.MINIO_ACCESS_KEY')
|
||||
MINIO_SECRET_KEY=$(echo "$SHARED_SECRET" | jq -r '.MINIO_SECRET_KEY')
|
||||
DASHBOARD_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.DASHBOARD_ADMIN_USERNAME')
|
||||
DASHBOARD_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.DASHBOARD_ADMIN_PASSWORD')
|
||||
GRAFANA_ADMIN_USERNAME=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_USERNAME')
|
||||
GRAFANA_ADMIN_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.GRAFANA_ADMIN_PASSWORD')
|
||||
LIVEKIT_API_KEY=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_KEY')
|
||||
LIVEKIT_API_SECRET=$(echo "$SHARED_SECRET" | jq -r '.LIVEKIT_API_SECRET')
|
||||
ENABLED_MODULES=$(echo "$SHARED_SECRET" | jq -r '.ENABLED_MODULES')
|
||||
|
||||
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_master_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu master node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Common arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -1858,20 +1900,6 @@ Resources:
|
|||
mode: "000755"
|
||||
owner: "root"
|
||||
group: "root"
|
||||
'/usr/local/bin/check_app_ready.sh':
|
||||
content: |
|
||||
#!/bin/bash
|
||||
set -e
|
||||
while true; do
|
||||
HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy | head -n1 | awk '{print $2}')
|
||||
if [ $HTTP_STATUS == 200 ]; then
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
mode: "000755"
|
||||
owner: "root"
|
||||
group: "root"
|
||||
'/usr/local/bin/restart.sh':
|
||||
content: |
|
||||
#!/bin/bash
|
||||
|
|
@ -1950,9 +1978,30 @@ Resources:
|
|||
# Launch on reboot
|
||||
echo "@reboot /usr/local/bin/restart.sh &> /var/log/openvidu-restart.log" | crontab
|
||||
|
||||
# Wait for the app
|
||||
sleep 20
|
||||
/usr/local/bin/check_app_ready.sh
|
||||
# Local readiness gate: wait up to 300s for Caddy health, restart once if it does not converge
|
||||
OPENVIDU_READY=false
|
||||
for i in $(seq 1 60); do
|
||||
if curl -fsS http://127.0.0.1:7880/health/caddy >/dev/null 2>&1; then
|
||||
OPENVIDU_READY=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
if [ "$OPENVIDU_READY" != "true" ]; then
|
||||
echo "[OpenVidu] not healthy after 300s, restarting once"
|
||||
systemctl restart openvidu || true
|
||||
for i in $(seq 1 60); do
|
||||
if curl -fsS http://127.0.0.1:7880/health/caddy >/dev/null 2>&1; then
|
||||
OPENVIDU_READY=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
fi
|
||||
|
||||
# Disable errexit so cfn-signal always runs and reports readiness via $? (fast-fail instead of a PT20M timeout)
|
||||
set +e
|
||||
[ "$OPENVIDU_READY" = "true" ]
|
||||
|
||||
# sending the finish call
|
||||
cfn-signal -e $? --stack ${AWS::StackId} --resource WaitCondition --region ${AWS::Region}
|
||||
|
|
@ -2022,31 +2071,48 @@ Resources:
|
|||
sleep 6
|
||||
done
|
||||
|
||||
# Get current shared secret
|
||||
DOMAIN=$(echo $SHARED_SECRET | jq -r .DOMAIN_NAME)
|
||||
OPENVIDU_PRO_LICENSE=$(echo $SHARED_SECRET | jq -r .OPENVIDU_PRO_LICENSE)
|
||||
REDIS_PASSWORD=$(echo $SHARED_SECRET | jq -r .REDIS_PASSWORD)
|
||||
|
||||
# Get OpenVidu Media Nodes version to deploy
|
||||
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||
|
||||
if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
||||
echo "OpenVidu version not found"
|
||||
# Wait until the master node has generated all shared secrets.
|
||||
# Bounded (up to 1800s at 5s), validating content on each snapshot: stale eventually-consistent reads can return pre-generation values
|
||||
SECRETS_WAIT_ATTEMPTS=0
|
||||
SECRETS_WAIT_MAX=360
|
||||
while true; do
|
||||
if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and (.OPENVIDU_VERSION != "none") and (.REDIS_PASSWORD != "none")' > /dev/null; then
|
||||
break
|
||||
fi
|
||||
SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1))
|
||||
if [[ $SECRETS_WAIT_ATTEMPTS -ge $SECRETS_WAIT_MAX ]]; then
|
||||
echo "Error: timed out after 30 minutes waiting for the master node to generate the shared secrets"
|
||||
/usr/local/bin/set_as_unhealthy.sh
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
SHARED_SECRET=$(aws secretsmanager get-secret-value \
|
||||
--region ${AWS::Region} \
|
||||
--secret-id openvidu-elastic-${AWS::Region}-${AWS::StackName} \
|
||||
--query SecretString --output text || echo 'none')
|
||||
done
|
||||
|
||||
ALL_SECRETS_GENERATED=$(echo $SHARED_SECRET | jq -r .ALL_SECRETS_GENERATED)
|
||||
if [[ "$ALL_SECRETS_GENERATED" == "false" ]]; then
|
||||
echo "Master node not ready"
|
||||
/usr/local/bin/set_as_unhealthy.sh
|
||||
# Read values only after the gate confirms the master node has published them
|
||||
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
|
||||
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
|
||||
REDIS_PASSWORD=$(echo "$SHARED_SECRET" | jq -r '.REDIS_PASSWORD')
|
||||
OPENVIDU_VERSION=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_VERSION')
|
||||
if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
||||
echo "OpenVidu version not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Get Master Node private IP
|
||||
MASTER_NODE_IP=${OpenViduMasterNode.PrivateIp}
|
||||
|
||||
# Base command
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_media_node.sh)"
|
||||
# Download first: sh <(curl ...) would silently run an empty script on a transient curl failure
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/elastic/$OPENVIDU_VERSION/install_ov_media_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu media node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Common arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -2190,6 +2256,22 @@ Resources:
|
|||
# Install OpenVidu
|
||||
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||
|
||||
# Wait for the master node to be healthy before starting (bounded: up to 1800s at 5s)
|
||||
MASTER_NODE_IP=${OpenViduMasterNode.PrivateIp}
|
||||
MASTER_HEALTHY=false
|
||||
for i in $(seq 1 360); do
|
||||
if curl -sf "http://$MASTER_NODE_IP:7880/health/caddy" >/dev/null 2>&1; then
|
||||
MASTER_HEALTHY=true
|
||||
break
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
if [ "$MASTER_HEALTHY" != "true" ]; then
|
||||
echo "[OpenVidu] master node did not become healthy after 30 minutes"
|
||||
/usr/local/bin/set_as_unhealthy.sh
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Start OpenVidu
|
||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; /usr/local/bin/set_as_unhealthy.sh; exit 1; }
|
||||
|
||||
|
|
@ -2202,6 +2284,8 @@ Resources:
|
|||
|
||||
OpenViduMediaNodeASG:
|
||||
DependsOn:
|
||||
- OpenViduMediaNodeInstanceProfile
|
||||
- OpenViduMasterNodeInstanceProfile
|
||||
- StopMediaNodeCloudWatchEventRule
|
||||
Type: AWS::AutoScaling::AutoScalingGroup
|
||||
Properties:
|
||||
|
|
|
|||
Loading…
Reference in New Issue