ambiser
/
openvidu
mirror of https://github.com/OpenVidu/openvidu.git
9
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

openvidu-deployment: fix eventually-consistent secret reads in AWS HA 

GetSecretValue is eventually consistent: with masters booting in
parallel, the final re-read of the shared JSON could return a stale
pre-generation version (ALL_SECRETS_GENERATED already observed true but
values still 'none'), so the installer aborted with e.g.
'mongo-admin-password: must have at least 10 characters'. The old
serialized boot masked this window (minutes between generation and
re-reads); parallel boot exposed it intermittently (1 of 3 tries).

Fix: read-until-valid — retry the re-read (5s, 5 min cap) until the same
snapshot has ALL_SECRETS_GENERATED=true AND every always-generated value
!= 'none'; the media gate now validates OPENVIDU_VERSION and
REDIS_PASSWORD content too, not just the flag.

Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev, --tries 3):
3/3 PASS, deploy mean 5m19s (spread 76ms), ready mean 21m24s.
Control run without the fix reproduced the race (1/3 failed).
master
Piwccle 2026-07-25 00:46:02 +02:00
parent 4d6b7acd7c
commit 31a8488f23
1 changed files with 20 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
openvidu-deployment/pro/ha/aws/cf-openvidu-ha.yaml
View File

@ -1719,11 +1719,24 @@ Resources:
# Comma-separated 1-4: must match installer's --master-node-private-ip-list format # Comma-separated 1-4: must match installer's --master-node-private-ip-list format
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP" MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
# Re-read: other (non-IP) values still come from the shared secret # Re-read: other (non-IP) values still come from the shared secret.
# GetSecretValue is eventually consistent: retry until a read returns the generated values
SECRET_READ_ATTEMPTS=0
while true; do
SHARED_SECRET=$(aws secretsmanager get-secret-value \ SHARED_SECRET=$(aws secretsmanager get-secret-value \
--region ${AWS::Region} \ --region ${AWS::Region} \
--secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \ --secret-id openvidu-ha-${AWS::Region}-${AWS::StackName} \
--query SecretString --output text) --query SecretString --output text)
if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and ([.DOMAIN_NAME, .OPENVIDU_VERSION, .REDIS_PASSWORD, .MONGO_ADMIN_PASSWORD, .MONGO_REPLICA_SET_KEY, .MINIO_SECRET_KEY, .DASHBOARD_ADMIN_PASSWORD, .GRAFANA_ADMIN_PASSWORD, .LIVEKIT_API_KEY, .LIVEKIT_API_SECRET] | all(. != "none"))' > /dev/null; then
break
fi
SECRET_READ_ATTEMPTS=$((SECRET_READ_ATTEMPTS + 1))
if [[ $SECRET_READ_ATTEMPTS -ge 60 ]]; then
echo "Error: shared secret still incomplete after 5 minutes of stale reads"
exit 1
fi
sleep 5
done
DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME') DOMAIN=$(echo "$SHARED_SECRET" | jq -r '.DOMAIN_NAME')
OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE') OPENVIDU_PRO_LICENSE=$(echo "$SHARED_SECRET" | jq -r '.OPENVIDU_PRO_LICENSE')
@ -2476,8 +2489,8 @@ Resources:
SECRETS_WAIT_ATTEMPTS=0 SECRETS_WAIT_ATTEMPTS=0
SECRETS_WAIT_MAX=360 SECRETS_WAIT_MAX=360
while true; do while true; do
ALL_SECRETS_GENERATED=$(echo "$SHARED_SECRET" | jq -r '.ALL_SECRETS_GENERATED') # Validate content, not just the flag: stale eventually-consistent reads can return pre-generation values
if [[ "$ALL_SECRETS_GENERATED" == "true" ]]; then if echo "$SHARED_SECRET" | jq -e '(.ALL_SECRETS_GENERATED == "true") and (.OPENVIDU_VERSION != "none") and (.REDIS_PASSWORD != "none")' > /dev/null; then
break break
fi fi
SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1)) SECRETS_WAIT_ATTEMPTS=$((SECRETS_WAIT_ATTEMPTS + 1))