diff --git a/openvidu-deployment/pro/ha/gcp/DOC-CHANGES-ha-optimizations.md b/openvidu-deployment/pro/ha/gcp/DOC-CHANGES-ha-optimizations.md new file mode 100644 index 000000000..e73e2c6f8 --- /dev/null +++ b/openvidu-deployment/pro/ha/gcp/DOC-CHANGES-ha-optimizations.md @@ -0,0 +1,144 @@ +# Cambios de documentación para `ha-optimizations` (despliegue HA GCP) + +Instrucciones para el Claude que trabaje en el repo de docs +`/home/sergio/Escritorio/openvidu/openvidu.io` (rama `next`, ya actualizada). + +Estas instrucciones derivan de los cambios aplicados en +`openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf` en la rama `ha-optimizations` +(paralelización de los 4 master nodes, IPs internas estáticas para los masters, +timeouts en las esperas, fetch robusto del instalador y `depends_on` hacia las APIs). + +RESUMEN: el ÚNICO cambio de contenido necesario son las dos cifras de tiempo de +despliegue en el `install.md` del HA de GCP. Todo lo demás (parámetros públicos, +capturas, otros clouds, otros deployment types, ficheros de config on-node) se +mantiene igual. Léelo entero antes de tocar nada. + +--- + +## (a) Cifras de tiempo a actualizar — OBLIGATORIO + +Fichero: `docs/docs/self-hosting/ha/gcp/install.md` + +Hay DOS ocurrencias de "7 to 12 minutes" en este fichero (aprox. líneas 249 y 308). +Localízalas por el texto literal, no confíes solo en el número de línea (puede haberse +desplazado). Sustituye la cifra por el placeholder y deja una nota para rellenarla con +la medición real de `ov-cloud-tester` tras estos cambios (el despliegue ahora es más +rápido: masters en paralelo y sin el handshake de IPs por Secret Manager). + +### Ocurrencia 1 (sección "## Deploying the stack", ~línea 249) + +Texto actual literal: + +``` +When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around 7 to 12 minutes for the nodes to install OpenVidu. +``` + +Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`): + +``` +When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around X to Y minutes for the nodes to install OpenVidu. +``` + +### Ocurrencia 2 (sección "## Checking...", ~línea 308) + +Texto actual literal: + +``` +When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about 7 to 12 minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment. +``` + +Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`): + +``` +When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about X to Y minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment. +``` + +> Placeholder a rellenar: `X to Y minutes — rellenar con la medición de ov-cloud-tester`. +> Mantén las dos ocurrencias con la MISMA cifra (la segunda dice "as mentioned before"). + +### NO tocar las cifras de otros ficheros + +Estos cambios afectan SOLO al HA de GCP. NO modifiques las cifras "7 to 12 minutes" +que aparecen en otros documentos (son otros clouds / otros deployment types y están +fuera del alcance de este trabajo): + +- `docs/docs/self-hosting/elastic/gcp/install.md` (líneas ~236 y ~295) +- `docs/docs/self-hosting/elastic/azure/install.md` (líneas ~95 y ~149) +- Cualquier otro `install.md` de la doc. + +--- + +## (b) Parámetros públicos de la plantilla GCP: SIN CAMBIOS — confirmación + +NO hay que tocar tablas de parámetros ni capturas de la sección de parámetros. + +Motivo: las nuevas IPs estáticas de los master nodes son **internas** +(`google_compute_address` con `address_type = "INTERNAL"`, reservadas dentro de la +subred `default` de la región) y se asignan vía `network_interface.network_ip`. No se +ha añadido, eliminado ni renombrado ninguna variable de entrada ni salida de Terraform +(`variables.tf` y `output.tf` intactos). Por tanto: + +- La tabla "Mandatory Parameters" y "Optional Parameters" de + `docs/docs/self-hosting/ha/gcp/install.md` (aprox. líneas 95–230) **se queda igual**. +- No hay parámetros nuevos que el usuario deba introducir en Infrastructure Manager. +- Las capturas del formulario de parámetros **no cambian**. + +--- + +## (c) Otras afirmaciones de la doc GCP HA que puedan quedar obsoletas + +Se ha revisado toda la doc del HA de GCP (`docs/docs/self-hosting/ha/gcp/*.md`) y los +includes compartidos. Conclusiones: + +### c.1 — Secretos `MASTER_NODE_{1..4}_PRIVATE_IP` de Secret Manager (ELIMINADOS) + +En `ha-optimizations` se han eliminado del despliegue los 4 secretos de Secret Manager +`MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` (ya no se generan: las IPs de +los masters son ahora estáticas y se pasan por metadata). La doc del HA de GCP **no los +menciona por nombre en el texto**, así que no hay texto que corregir. + +- Único impacto posible: la captura del Secret Manager en + `install.md` (sección "Check deployment outputs in GCP Secret Manager"), + imagen `assets/images/platform/self-hosting/shared/gcp/secrets-manager.png`. + Esa captura PODRÍA mostrar los 4 secretos de IP que ya no existirán. Es puramente + cosmético (el texto solo dice "you will see all the secrets by their name"). Acción + OPCIONAL: si se regenera la captura en algún momento, hacerlo tras un despliegue con + `ha-optimizations`. NO es bloqueante para este cambio. + +### c.2 — `backup-and-restore.md`: NO TOCAR + +`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (aprox. líneas 676–692) +documenta variables `MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` dentro del +fichero de configuración on-node `/opt/openvidu/config/node/master-node.env`. + +IMPORTANTE: **esas NO son los secretos eliminados**. Son variables del fichero de +configuración que el instalador sigue escribiendo a partir del flag +`--master-node-private-ip-list` (que se sigue pasando, ahora derivado de las IPs +estáticas). Ese flujo no cambia. **No modifiques `backup-and-restore.md`.** + +### c.3 — `upgrade.md`: NO TOCAR + +`docs/docs/self-hosting/ha/gcp/upgrade.md` (línea ~91) menciona `store_secret.sh` y el +secreto `OPENVIDU_VERSION`. Ni el script `store_secret.sh` ni el secreto +`OPENVIDU_VERSION` se han tocado. **No requiere cambios.** + +### c.4 — Flujo interno (ALL_SECRETS_GENERATED, handshake de IPs): no documentado + +El mecanismo interno de coordinación (líder master-1 genera secretos y marca +`ALL_SECRETS_GENERATED`, polling, etc.) no está documentado públicamente, y su cambio +(eliminación del intercambio de IPs por Secret Manager) no altera nada de cara al +usuario. No hay texto que actualizar. + +--- + +## Checklist de aplicación + +- [ ] En `docs/docs/self-hosting/ha/gcp/install.md`, sustituir "7 to 12 minutes" por + "X to Y minutes" en las DOS ocurrencias (~249 y ~308), con la misma cifra en ambas. +- [ ] Medir el tiempo real de despliegue con `ov-cloud-tester` sobre `ha-optimizations` + y rellenar `X`/`Y`. +- [ ] Verificar que NO se han tocado tablas de parámetros ni capturas. +- [ ] Verificar que NO se han tocado `backup-and-restore.md`, `upgrade.md`, ni las + cifras de tiempo de elastic/azure/otros clouds. +- [ ] (Opcional) Regenerar `secrets-manager.png` si se desea reflejar la ausencia de los + 4 secretos de IP. diff --git a/openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf b/openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf index 781fc7472..a01c72442 100644 --- a/openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf +++ b/openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf @@ -20,14 +20,15 @@ resource "google_secret_manager_secret" "openvidu_shared_info" { "OPENVIDU_PRO_LICENSE", "OPENVIDU_RTC_ENGINE", "REDIS_PASSWORD", "MONGO_ADMIN_USERNAME", "MONGO_ADMIN_PASSWORD", "MONGO_REPLICA_SET_KEY", "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY", "DASHBOARD_ADMIN_USERNAME", "DASHBOARD_ADMIN_PASSWORD", "GRAFANA_ADMIN_USERNAME", - "GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED", - "MASTER_NODE_1_PRIVATE_IP", "MASTER_NODE_2_PRIVATE_IP", "MASTER_NODE_3_PRIVATE_IP", "MASTER_NODE_4_PRIVATE_IP" + "GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED" ]) secret_id = each.key replication { auto {} } + + depends_on = [google_project_service.secretmanager_api] } # GCS buckets for HA deployment @@ -37,6 +38,8 @@ resource "google_storage_bucket" "appdata_bucket" { location = var.region force_destroy = true uniform_bucket_level_access = true + + depends_on = [google_project_service.storage_api] } resource "google_storage_bucket" "clusterdata_bucket" { @@ -45,12 +48,16 @@ resource "google_storage_bucket" "clusterdata_bucket" { location = var.region force_destroy = true uniform_bucket_level_access = true + + depends_on = [google_project_service.storage_api] } # Service account for the instances resource "google_service_account" "service_account" { account_id = lower("${substr(var.stackName, 0, 12)}-sa") display_name = "OpenVidu instance service account" + + depends_on = [google_project_service.iam_api] } # IAM bindings for the service account @@ -58,6 +65,8 @@ resource "google_project_iam_member" "iam_project_role" { project = var.projectId role = "roles/owner" member = "serviceAccount:${google_service_account.service_account.email}" + + depends_on = [google_project_service.cloudresourcemanager_api] } # External SSH access to Master Nodes @@ -72,6 +81,8 @@ resource "google_compute_firewall" "external_master_ssh" { source_ranges = ["0.0.0.0/0"] target_tags = [lower("${var.stackName}-master-node")] + + depends_on = [google_project_service.compute_api] } # External access to Media Nodes (SSH and media traffic) @@ -90,6 +101,8 @@ resource "google_compute_firewall" "external_media_access" { source_ranges = ["0.0.0.0/0"] target_tags = [lower("${var.stackName}-media-node")] + + depends_on = [google_project_service.compute_api] } # Load Balancer health checks and HTTP traffic to Master Nodes @@ -104,6 +117,8 @@ resource "google_compute_firewall" "lb_to_master_http" { source_ranges = ["0.0.0.0/0"] target_tags = [lower("${var.stackName}-master-node")] + + depends_on = [google_project_service.compute_api] } # Master node internal services communication @@ -129,6 +144,8 @@ resource "google_compute_firewall" "master_to_master_internal" { source_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-master-node")] + + depends_on = [google_project_service.compute_api] } # Media Nodes to Master Nodes communication @@ -153,6 +170,8 @@ resource "google_compute_firewall" "media_to_master_services" { source_tags = [lower("${var.stackName}-media-node")] target_tags = [lower("${var.stackName}-master-node")] + + depends_on = [google_project_service.compute_api] } # Master Nodes to Media Nodes communication @@ -171,6 +190,8 @@ resource "google_compute_firewall" "master_to_media_services" { source_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-media-node")] + + depends_on = [google_project_service.compute_api] } @@ -179,6 +200,8 @@ resource "google_compute_address" "nlb_ip" { count = var.publicIpAddress == "" ? 1 : 0 name = lower("${var.stackName}-nlb-ip") region = var.region + + depends_on = [google_project_service.compute_api] } # Data source for existing IP address when publicIpAddress is provided @@ -206,6 +229,8 @@ resource "google_compute_region_health_check" "tcp_health_check" { timeout_sec = 5 healthy_threshold = 3 unhealthy_threshold = 4 + + depends_on = [google_project_service.compute_api] } # Regional backend service for the TCP NLB @@ -289,6 +314,34 @@ locals { is_c4a_instance = startswith(var.masterNodesInstanceType, "c4a-") } +# Default subnetwork the master instances live in (they use network = "default" with no +# explicit subnetwork, so GCP places them in the auto-mode default subnetwork of the region) +data "google_compute_subnetwork" "default" { + name = "default" + region = var.region + + depends_on = [google_project_service.compute_api] +} + +# Static internal IPs for the master nodes. Reserving them up front lets the media instance +# template and every master receive the full list without waiting for the master instances to +# boot, which removes the runtime IP-exchange handshake through Secret Manager. +resource "google_compute_address" "master_internal_ip" { + count = 4 + name = lower("${var.stackName}-master-node-${count.index + 1}-internal-ip") + address_type = "INTERNAL" + region = var.region + subnetwork = data.google_compute_subnetwork.default.id + + depends_on = [google_project_service.compute_api] +} + +locals { + # Master node private IPs come from the reserved static internal addresses (index 0 -> master + # node 1, ...). Comma-separated to match the installer's --master-node-private-ip-list format. + master_node_private_ip_list = join(",", google_compute_address.master_internal_ip[*].address) +} + # Master Node 1 resource "google_compute_instance" "openvidu_master_node_1" { name = lower("${var.stackName}-master-node-1") @@ -305,13 +358,15 @@ resource "google_compute_instance" "openvidu_master_node_1" { } network_interface { - network = "default" + network = "default" + network_ip = google_compute_address.master_internal_ip[0].address access_config {} } metadata = { stackName = var.stackName masterNodeNum = "1" + masterNodePrivateIPList = local.master_node_private_ip_list domainName = var.domainName certificateType = var.certificateType ownPublicCertificate = var.ownPublicCertificate @@ -355,13 +410,15 @@ resource "google_compute_instance" "openvidu_master_node_2" { } network_interface { - network = "default" + network = "default" + network_ip = google_compute_address.master_internal_ip[1].address access_config {} } metadata = { stackName = var.stackName masterNodeNum = "2" + masterNodePrivateIPList = local.master_node_private_ip_list domainName = var.domainName certificateType = var.certificateType ownPublicCertificate = var.ownPublicCertificate @@ -387,8 +444,6 @@ resource "google_compute_instance" "openvidu_master_node_2" { node-type = "master" node-num = "2" } - - depends_on = [google_compute_instance.openvidu_master_node_1] } # Master Node 3 @@ -407,13 +462,15 @@ resource "google_compute_instance" "openvidu_master_node_3" { } network_interface { - network = "default" + network = "default" + network_ip = google_compute_address.master_internal_ip[2].address access_config {} } metadata = { stackName = var.stackName masterNodeNum = "3" + masterNodePrivateIPList = local.master_node_private_ip_list domainName = var.domainName certificateType = var.certificateType ownPublicCertificate = var.ownPublicCertificate @@ -439,8 +496,6 @@ resource "google_compute_instance" "openvidu_master_node_3" { node-type = "master" node-num = "3" } - - depends_on = [google_compute_instance.openvidu_master_node_2] } # Master Node 4 @@ -459,13 +514,15 @@ resource "google_compute_instance" "openvidu_master_node_4" { } network_interface { - network = "default" + network = "default" + network_ip = google_compute_address.master_internal_ip[3].address access_config {} } metadata = { stackName = var.stackName masterNodeNum = "4" + masterNodePrivateIPList = local.master_node_private_ip_list domainName = var.domainName certificateType = var.certificateType ownPublicCertificate = var.ownPublicCertificate @@ -491,8 +548,6 @@ resource "google_compute_instance" "openvidu_master_node_4" { node-type = "master" node-num = "4" } - - depends_on = [google_compute_instance.openvidu_master_node_3] } # ------------------------- scale in resources ------------------------- @@ -800,6 +855,12 @@ resource "google_cloudfunctions2_function" "scalein_function" { } service_account_email = google_service_account.service_account.email } + + depends_on = [ + google_project_service.cloudfunctions_api, + google_project_service.cloudbuild_api, + google_project_service.run_api + ] } # Cloud Scheduler to trigger the function every 5 minutes @@ -829,6 +890,8 @@ resource "google_cloud_scheduler_job" "scale_scheduler" { service_account_email = google_service_account.service_account.email } } + + depends_on = [google_project_service.cloudscheduler_api] } locals { @@ -858,7 +921,7 @@ resource "google_compute_instance_template" "media_node_template" { metadata = { stackName = var.stackName - masterNodePrivateIPList = "${google_compute_instance.openvidu_master_node_1.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_2.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_3.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_4.network_interface[0].network_ip}" + masterNodePrivateIPList = local.master_node_private_ip_list bucketAppDataName = local.isEmptyAppData ? google_storage_bucket.appdata_bucket[0].name : var.GCSAppDataBucketName bucketClusterDataName = local.isEmptyClusterData ? google_storage_bucket.clusterdata_bucket[0].name : var.GCSClusterDataBucketName region = var.region @@ -880,13 +943,6 @@ resource "google_compute_instance_template" "media_node_template" { lifecycle { create_before_destroy = true } - - depends_on = [ - google_compute_instance.openvidu_master_node_1, - google_compute_instance.openvidu_master_node_2, - google_compute_instance.openvidu_master_node_3, - google_compute_instance.openvidu_master_node_4 - ] } # Managed Instance Group for Media Nodes @@ -904,13 +960,6 @@ resource "google_compute_region_instance_group_manager" "media_node_group" { name = "http" port = 7880 } - - depends_on = [ - google_compute_instance.openvidu_master_node_1, - google_compute_instance.openvidu_master_node_2, - google_compute_instance.openvidu_master_node_3, - google_compute_instance.openvidu_master_node_4 - ] } # Autoscaler for Media Nodes @@ -969,11 +1018,8 @@ get_meta() { curl -s -H "Metadata-Flavor: Google" "$${METADATA_URL}/$1"; } # Get master node number from metadata MASTER_NODE_NUM=$(get_meta "instance/attributes/masterNodeNum") -# Get own private IP -PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip") - -# Store current private IP -PRIVATE_IP="$(/usr/local/bin/store_secret.sh save MASTER_NODE_$${MASTER_NODE_NUM}_PRIVATE_IP $PRIVATE_IP)" +# Get the list of master node private IPs (static internal IPs passed via metadata) +MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPList") # Check if secrets have been generated ALL_SECRETS_GENERATED=$(gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null || echo "false") @@ -1033,33 +1079,20 @@ if [[ $MASTER_NODE_NUM -eq 1 ]] && [[ "$ALL_SECRETS_GENERATED" == "false" ]]; th ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")" fi -# Wait for all master nodes to store their private IPs -while true; do - MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP 2>/dev/null || echo "") - MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP 2>/dev/null || echo "") - MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP 2>/dev/null || echo "") - MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP 2>/dev/null || echo "") - - # Check if all master nodes have stored their private IPs - if [[ "$MASTER_NODE_1_PRIVATE_IP" != "" ]] && - [[ "$MASTER_NODE_2_PRIVATE_IP" != "" ]] && - [[ "$MASTER_NODE_3_PRIVATE_IP" != "" ]] && - [[ "$MASTER_NODE_4_PRIVATE_IP" != "" ]]; then - break +# Wait for master-1 to finish generating all shared secrets before reading them. +# Bounded to 360 iterations x 5s = 30 minutes. +i=0 +while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do + i=$((i + 1)) + if [ "$i" -ge 360 ]; then + echo "Timed out after 30 minutes waiting for master-1 to finish generating secrets" >&2 + exit 1 fi + echo "Waiting for master-1 to finish generating secrets..." sleep 5 done -# Wait for master-1 to finish generating all shared secrets before reading them -while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do echo "Waiting for master-1 to finish generating secrets..."; sleep 5; done - # Fetch all values from Secret Manager -MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP) -MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP) -MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP) -MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP) -MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP" - DOMAIN=$(gcloud secrets versions access latest --secret=DOMAIN_NAME) OPENVIDU_PRO_LICENSE=$(gcloud secrets versions access latest --secret=OPENVIDU_PRO_LICENSE) OPENVIDU_RTC_ENGINE=$(gcloud secrets versions access latest --secret=OPENVIDU_RTC_ENGINE) @@ -1082,8 +1115,15 @@ LIVEKIT_API_KEY=$(gcloud secrets versions access latest --secret=LIVEKIT_API_KEY LIVEKIT_API_SECRET=$(gcloud secrets versions access latest --secret=LIVEKIT_API_SECRET) ENABLED_MODULES=$(gcloud secrets versions access latest --secret=ENABLED_MODULES) -# Build install command -INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)" +# Build install command. Download the installer to a file first: process substitution +# (sh <(curl ...)) would silently run an empty script and exit 0 on a transient curl failure. +INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh" +curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh" +if [ ! -s "$INSTALLER_SCRIPT" ]; then + echo "Downloaded OpenVidu master node installer is empty or missing" >&2 + exit 1 +fi +INSTALL_COMMAND="sh $INSTALLER_SCRIPT" # Common arguments COMMON_ARGS=( @@ -1434,11 +1474,18 @@ EOF check_app_ready_script = <<-EOF #!/bin/bash +# Bounded to 240 iterations x 5s = 20 minutes. +i=0 while true; do HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy 2>/dev/null | head -n1 | awk '{print $2}') if [ "$HTTP_STATUS" == "200" ]; then break fi + i=$((i + 1)) + if [ "$i" -ge 240 ]; then + echo "Timed out after 20 minutes waiting for OpenVidu to become ready" >&2 + exit 1 + fi sleep 5 done EOF @@ -1513,8 +1560,6 @@ CONFIG_S3_EOF echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout - apt-get update && apt-get install -y - GCLOUD_VERSION=573.0.0 # Install google cli if ! command -v gcloud >/dev/null 2>&1; then @@ -1578,8 +1623,15 @@ MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPL STACK_NAME=$(get_meta "instance/attributes/stackName") PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip") -# Wait for master nodes to be ready by checking secrets +# Wait for master nodes to be ready by checking secrets. +# Bounded to 180 iterations x 10s = 30 minutes. +i=0 while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do + i=$((i + 1)) + if [ "$i" -ge 180 ]; then + echo "Timed out after 30 minutes waiting for master nodes to initialize secrets" >&2 + exit 1 + fi echo "Waiting for master nodes to initialize secrets..." sleep 10 done @@ -1597,8 +1649,16 @@ if [[ "$OPENVIDU_VERSION" == "none" ]]; then exit 1 fi -# Build install command for media node -INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)" +# Build install command for media node. Download the installer to a file first: process +# substitution (sh <(curl ...)) would silently run an empty script and exit 0 on a transient +# curl failure. +INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh" +curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh" +if [ ! -s "$INSTALLER_SCRIPT" ]; then + echo "Downloaded OpenVidu media node installer is empty or missing" >&2 + exit 1 +fi +INSTALL_COMMAND="sh $INSTALLER_SCRIPT" # Media node arguments COMMON_ARGS=( @@ -1698,51 +1758,55 @@ EOF #!/bin/bash -x set -eu -o pipefail -# Create scripts -cat > /usr/local/bin/install.sh << 'INSTALL_EOF' +# Check if installation already completed +if [ -f /usr/local/bin/openvidu_install_counter.txt ]; then + # Launch on reboot + systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; } +else + # Create scripts + cat > /usr/local/bin/install.sh << 'INSTALL_EOF' ${local.install_script_media} INSTALL_EOF -chmod +x /usr/local/bin/install.sh + chmod +x /usr/local/bin/install.sh -cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF' + cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF' ${local.graceful_shutdown_script} GRACEFUL_SHUTDOWN_EOF -chmod +x /usr/local/bin/graceful_shutdown.sh + chmod +x /usr/local/bin/graceful_shutdown.sh -echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout + echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout -apt-get update && apt-get install -y + GCLOUD_VERSION=573.0.0 + # Install google cli + if ! command -v gcloud >/dev/null 2>&1; then + curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg + echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list + apt-get update && apt-get install -y google-cloud-cli=$${GCLOUD_VERSION}-0 + fi -GCLOUD_VERSION=573.0.0 -# Install google cli -if ! command -v gcloud >/dev/null 2>&1; then - curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg - echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list - apt-get update && apt-get install -y google-cloud-cli=$${GCLOUD_VERSION}-0 -fi + # Authenticate with gcloud using instance service account + gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true + gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google") + gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google") -# Authenticate with gcloud using instance service account -gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true -gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google") -gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google") + export HOME="/root" -export HOME="/root" + # Install OpenVidu Media Node + /usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; } -# Install OpenVidu Media Node -/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; } + # Mark installation as complete + echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt -# Mark installation as complete -echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt + # Start OpenVidu + systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; } -# Start OpenVidu -systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; } - -# Add cron job to check if instance is abandoned every minute -cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF' + # Add cron job to check if instance is abandoned every minute + cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF' ${local.crontab_job_media} CHECK_ABANDONED_EOF -chmod +x /usr/local/bin/check_abandoned.sh + chmod +x /usr/local/bin/check_abandoned.sh -echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab - + echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab - +fi EOF }