openvidu-deployment: optimize GCP HA deployment time

Parallel master node creation (drop chained depends_on), static internal
IPs for masters passed via metadata (removes the runtime IP exchange
through Secret Manager and its unbounded wait), bounded timeouts on
ALL_SECRETS_GENERATED/check_app_ready polls, robust installer fetch
(curl --retry to file instead of sh <(curl)), media node reboot guard,
apt cleanup and depends_on on enabled APIs.

Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS.
deploy 1m54s (baseline 3m30s), ready 6m26s, destroy 3m13s.
master
Piwccle 2026-07-24 13:42:09 +02:00
parent 5a3347c12a
commit 1794e7e16b
2 changed files with 299 additions and 91 deletions

View File

@ -0,0 +1,144 @@
# Cambios de documentación para `ha-optimizations` (despliegue HA GCP)
Instrucciones para el Claude que trabaje en el repo de docs
`/home/sergio/Escritorio/openvidu/openvidu.io` (rama `next`, ya actualizada).
Estas instrucciones derivan de los cambios aplicados en
`openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf` en la rama `ha-optimizations`
(paralelización de los 4 master nodes, IPs internas estáticas para los masters,
timeouts en las esperas, fetch robusto del instalador y `depends_on` hacia las APIs).
RESUMEN: el ÚNICO cambio de contenido necesario son las dos cifras de tiempo de
despliegue en el `install.md` del HA de GCP. Todo lo demás (parámetros públicos,
capturas, otros clouds, otros deployment types, ficheros de config on-node) se
mantiene igual. Léelo entero antes de tocar nada.
---
## (a) Cifras de tiempo a actualizar — OBLIGATORIO
Fichero: `docs/docs/self-hosting/ha/gcp/install.md`
Hay DOS ocurrencias de "7 to 12 minutes" en este fichero (aprox. líneas 249 y 308).
Localízalas por el texto literal, no confíes solo en el número de línea (puede haberse
desplazado). Sustituye la cifra por el placeholder y deja una nota para rellenarla con
la medición real de `ov-cloud-tester` tras estos cambios (el despliegue ahora es más
rápido: masters en paralelo y sin el handshake de IPs por Secret Manager).
### Ocurrencia 1 (sección "## Deploying the stack", ~línea 249)
Texto actual literal:
```
When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around 7 to 12 minutes for the nodes to install OpenVidu.
```
Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`):
```
When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around X to Y minutes for the nodes to install OpenVidu.
```
### Ocurrencia 2 (sección "## Checking...", ~línea 308)
Texto actual literal:
```
When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about 7 to 12 minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
```
Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`):
```
When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about X to Y minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
```
> Placeholder a rellenar: `X to Y minutes — rellenar con la medición de ov-cloud-tester`.
> Mantén las dos ocurrencias con la MISMA cifra (la segunda dice "as mentioned before").
### NO tocar las cifras de otros ficheros
Estos cambios afectan SOLO al HA de GCP. NO modifiques las cifras "7 to 12 minutes"
que aparecen en otros documentos (son otros clouds / otros deployment types y están
fuera del alcance de este trabajo):
- `docs/docs/self-hosting/elastic/gcp/install.md` (líneas ~236 y ~295)
- `docs/docs/self-hosting/elastic/azure/install.md` (líneas ~95 y ~149)
- Cualquier otro `install.md` de la doc.
---
## (b) Parámetros públicos de la plantilla GCP: SIN CAMBIOS — confirmación
NO hay que tocar tablas de parámetros ni capturas de la sección de parámetros.
Motivo: las nuevas IPs estáticas de los master nodes son **internas**
(`google_compute_address` con `address_type = "INTERNAL"`, reservadas dentro de la
subred `default` de la región) y se asignan vía `network_interface.network_ip`. No se
ha añadido, eliminado ni renombrado ninguna variable de entrada ni salida de Terraform
(`variables.tf` y `output.tf` intactos). Por tanto:
- La tabla "Mandatory Parameters" y "Optional Parameters" de
`docs/docs/self-hosting/ha/gcp/install.md` (aprox. líneas 95230) **se queda igual**.
- No hay parámetros nuevos que el usuario deba introducir en Infrastructure Manager.
- Las capturas del formulario de parámetros **no cambian**.
---
## (c) Otras afirmaciones de la doc GCP HA que puedan quedar obsoletas
Se ha revisado toda la doc del HA de GCP (`docs/docs/self-hosting/ha/gcp/*.md`) y los
includes compartidos. Conclusiones:
### c.1 — Secretos `MASTER_NODE_{1..4}_PRIVATE_IP` de Secret Manager (ELIMINADOS)
En `ha-optimizations` se han eliminado del despliegue los 4 secretos de Secret Manager
`MASTER_NODE_1_PRIVATE_IP``MASTER_NODE_4_PRIVATE_IP` (ya no se generan: las IPs de
los masters son ahora estáticas y se pasan por metadata). La doc del HA de GCP **no los
menciona por nombre en el texto**, así que no hay texto que corregir.
- Único impacto posible: la captura del Secret Manager en
`install.md` (sección "Check deployment outputs in GCP Secret Manager"),
imagen `assets/images/platform/self-hosting/shared/gcp/secrets-manager.png`.
Esa captura PODRÍA mostrar los 4 secretos de IP que ya no existirán. Es puramente
cosmético (el texto solo dice "you will see all the secrets by their name"). Acción
OPCIONAL: si se regenera la captura en algún momento, hacerlo tras un despliegue con
`ha-optimizations`. NO es bloqueante para este cambio.
### c.2 — `backup-and-restore.md`: NO TOCAR
`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (aprox. líneas 676692)
documenta variables `MASTER_NODE_1_PRIVATE_IP``MASTER_NODE_4_PRIVATE_IP` dentro del
fichero de configuración on-node `/opt/openvidu/config/node/master-node.env`.
IMPORTANTE: **esas NO son los secretos eliminados**. Son variables del fichero de
configuración que el instalador sigue escribiendo a partir del flag
`--master-node-private-ip-list` (que se sigue pasando, ahora derivado de las IPs
estáticas). Ese flujo no cambia. **No modifiques `backup-and-restore.md`.**
### c.3 — `upgrade.md`: NO TOCAR
`docs/docs/self-hosting/ha/gcp/upgrade.md` (línea ~91) menciona `store_secret.sh` y el
secreto `OPENVIDU_VERSION`. Ni el script `store_secret.sh` ni el secreto
`OPENVIDU_VERSION` se han tocado. **No requiere cambios.**
### c.4 — Flujo interno (ALL_SECRETS_GENERATED, handshake de IPs): no documentado
El mecanismo interno de coordinación (líder master-1 genera secretos y marca
`ALL_SECRETS_GENERATED`, polling, etc.) no está documentado públicamente, y su cambio
(eliminación del intercambio de IPs por Secret Manager) no altera nada de cara al
usuario. No hay texto que actualizar.
---
## Checklist de aplicación
- [ ] En `docs/docs/self-hosting/ha/gcp/install.md`, sustituir "7 to 12 minutes" por
"X to Y minutes" en las DOS ocurrencias (~249 y ~308), con la misma cifra en ambas.
- [ ] Medir el tiempo real de despliegue con `ov-cloud-tester` sobre `ha-optimizations`
y rellenar `X`/`Y`.
- [ ] Verificar que NO se han tocado tablas de parámetros ni capturas.
- [ ] Verificar que NO se han tocado `backup-and-restore.md`, `upgrade.md`, ni las
cifras de tiempo de elastic/azure/otros clouds.
- [ ] (Opcional) Regenerar `secrets-manager.png` si se desea reflejar la ausencia de los
4 secretos de IP.

View File

@ -20,14 +20,15 @@ resource "google_secret_manager_secret" "openvidu_shared_info" {
"OPENVIDU_PRO_LICENSE", "OPENVIDU_RTC_ENGINE", "REDIS_PASSWORD", "MONGO_ADMIN_USERNAME", "OPENVIDU_PRO_LICENSE", "OPENVIDU_RTC_ENGINE", "REDIS_PASSWORD", "MONGO_ADMIN_USERNAME",
"MONGO_ADMIN_PASSWORD", "MONGO_REPLICA_SET_KEY", "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY", "MONGO_ADMIN_PASSWORD", "MONGO_REPLICA_SET_KEY", "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY",
"DASHBOARD_ADMIN_USERNAME", "DASHBOARD_ADMIN_PASSWORD", "GRAFANA_ADMIN_USERNAME", "DASHBOARD_ADMIN_USERNAME", "DASHBOARD_ADMIN_PASSWORD", "GRAFANA_ADMIN_USERNAME",
"GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED", "GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED"
"MASTER_NODE_1_PRIVATE_IP", "MASTER_NODE_2_PRIVATE_IP", "MASTER_NODE_3_PRIVATE_IP", "MASTER_NODE_4_PRIVATE_IP"
]) ])
secret_id = each.key secret_id = each.key
replication { replication {
auto {} auto {}
} }
depends_on = [google_project_service.secretmanager_api]
} }
# GCS buckets for HA deployment # GCS buckets for HA deployment
@ -37,6 +38,8 @@ resource "google_storage_bucket" "appdata_bucket" {
location = var.region location = var.region
force_destroy = true force_destroy = true
uniform_bucket_level_access = true uniform_bucket_level_access = true
depends_on = [google_project_service.storage_api]
} }
resource "google_storage_bucket" "clusterdata_bucket" { resource "google_storage_bucket" "clusterdata_bucket" {
@ -45,12 +48,16 @@ resource "google_storage_bucket" "clusterdata_bucket" {
location = var.region location = var.region
force_destroy = true force_destroy = true
uniform_bucket_level_access = true uniform_bucket_level_access = true
depends_on = [google_project_service.storage_api]
} }
# Service account for the instances # Service account for the instances
resource "google_service_account" "service_account" { resource "google_service_account" "service_account" {
account_id = lower("${substr(var.stackName, 0, 12)}-sa") account_id = lower("${substr(var.stackName, 0, 12)}-sa")
display_name = "OpenVidu instance service account" display_name = "OpenVidu instance service account"
depends_on = [google_project_service.iam_api]
} }
# IAM bindings for the service account # IAM bindings for the service account
@ -58,6 +65,8 @@ resource "google_project_iam_member" "iam_project_role" {
project = var.projectId project = var.projectId
role = "roles/owner" role = "roles/owner"
member = "serviceAccount:${google_service_account.service_account.email}" member = "serviceAccount:${google_service_account.service_account.email}"
depends_on = [google_project_service.cloudresourcemanager_api]
} }
# External SSH access to Master Nodes # External SSH access to Master Nodes
@ -72,6 +81,8 @@ resource "google_compute_firewall" "external_master_ssh" {
source_ranges = ["0.0.0.0/0"] source_ranges = ["0.0.0.0/0"]
target_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-master-node")]
depends_on = [google_project_service.compute_api]
} }
# External access to Media Nodes (SSH and media traffic) # External access to Media Nodes (SSH and media traffic)
@ -90,6 +101,8 @@ resource "google_compute_firewall" "external_media_access" {
source_ranges = ["0.0.0.0/0"] source_ranges = ["0.0.0.0/0"]
target_tags = [lower("${var.stackName}-media-node")] target_tags = [lower("${var.stackName}-media-node")]
depends_on = [google_project_service.compute_api]
} }
# Load Balancer health checks and HTTP traffic to Master Nodes # Load Balancer health checks and HTTP traffic to Master Nodes
@ -104,6 +117,8 @@ resource "google_compute_firewall" "lb_to_master_http" {
source_ranges = ["0.0.0.0/0"] source_ranges = ["0.0.0.0/0"]
target_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-master-node")]
depends_on = [google_project_service.compute_api]
} }
# Master node internal services communication # Master node internal services communication
@ -129,6 +144,8 @@ resource "google_compute_firewall" "master_to_master_internal" {
source_tags = [lower("${var.stackName}-master-node")] source_tags = [lower("${var.stackName}-master-node")]
target_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-master-node")]
depends_on = [google_project_service.compute_api]
} }
# Media Nodes to Master Nodes communication # Media Nodes to Master Nodes communication
@ -153,6 +170,8 @@ resource "google_compute_firewall" "media_to_master_services" {
source_tags = [lower("${var.stackName}-media-node")] source_tags = [lower("${var.stackName}-media-node")]
target_tags = [lower("${var.stackName}-master-node")] target_tags = [lower("${var.stackName}-master-node")]
depends_on = [google_project_service.compute_api]
} }
# Master Nodes to Media Nodes communication # Master Nodes to Media Nodes communication
@ -171,6 +190,8 @@ resource "google_compute_firewall" "master_to_media_services" {
source_tags = [lower("${var.stackName}-master-node")] source_tags = [lower("${var.stackName}-master-node")]
target_tags = [lower("${var.stackName}-media-node")] target_tags = [lower("${var.stackName}-media-node")]
depends_on = [google_project_service.compute_api]
} }
@ -179,6 +200,8 @@ resource "google_compute_address" "nlb_ip" {
count = var.publicIpAddress == "" ? 1 : 0 count = var.publicIpAddress == "" ? 1 : 0
name = lower("${var.stackName}-nlb-ip") name = lower("${var.stackName}-nlb-ip")
region = var.region region = var.region
depends_on = [google_project_service.compute_api]
} }
# Data source for existing IP address when publicIpAddress is provided # Data source for existing IP address when publicIpAddress is provided
@ -206,6 +229,8 @@ resource "google_compute_region_health_check" "tcp_health_check" {
timeout_sec = 5 timeout_sec = 5
healthy_threshold = 3 healthy_threshold = 3
unhealthy_threshold = 4 unhealthy_threshold = 4
depends_on = [google_project_service.compute_api]
} }
# Regional backend service for the TCP NLB # Regional backend service for the TCP NLB
@ -289,6 +314,34 @@ locals {
is_c4a_instance = startswith(var.masterNodesInstanceType, "c4a-") is_c4a_instance = startswith(var.masterNodesInstanceType, "c4a-")
} }
# Default subnetwork the master instances live in (they use network = "default" with no
# explicit subnetwork, so GCP places them in the auto-mode default subnetwork of the region)
data "google_compute_subnetwork" "default" {
name = "default"
region = var.region
depends_on = [google_project_service.compute_api]
}
# Static internal IPs for the master nodes. Reserving them up front lets the media instance
# template and every master receive the full list without waiting for the master instances to
# boot, which removes the runtime IP-exchange handshake through Secret Manager.
resource "google_compute_address" "master_internal_ip" {
count = 4
name = lower("${var.stackName}-master-node-${count.index + 1}-internal-ip")
address_type = "INTERNAL"
region = var.region
subnetwork = data.google_compute_subnetwork.default.id
depends_on = [google_project_service.compute_api]
}
locals {
# Master node private IPs come from the reserved static internal addresses (index 0 -> master
# node 1, ...). Comma-separated to match the installer's --master-node-private-ip-list format.
master_node_private_ip_list = join(",", google_compute_address.master_internal_ip[*].address)
}
# Master Node 1 # Master Node 1
resource "google_compute_instance" "openvidu_master_node_1" { resource "google_compute_instance" "openvidu_master_node_1" {
name = lower("${var.stackName}-master-node-1") name = lower("${var.stackName}-master-node-1")
@ -305,13 +358,15 @@ resource "google_compute_instance" "openvidu_master_node_1" {
} }
network_interface { network_interface {
network = "default" network = "default"
network_ip = google_compute_address.master_internal_ip[0].address
access_config {} access_config {}
} }
metadata = { metadata = {
stackName = var.stackName stackName = var.stackName
masterNodeNum = "1" masterNodeNum = "1"
masterNodePrivateIPList = local.master_node_private_ip_list
domainName = var.domainName domainName = var.domainName
certificateType = var.certificateType certificateType = var.certificateType
ownPublicCertificate = var.ownPublicCertificate ownPublicCertificate = var.ownPublicCertificate
@ -355,13 +410,15 @@ resource "google_compute_instance" "openvidu_master_node_2" {
} }
network_interface { network_interface {
network = "default" network = "default"
network_ip = google_compute_address.master_internal_ip[1].address
access_config {} access_config {}
} }
metadata = { metadata = {
stackName = var.stackName stackName = var.stackName
masterNodeNum = "2" masterNodeNum = "2"
masterNodePrivateIPList = local.master_node_private_ip_list
domainName = var.domainName domainName = var.domainName
certificateType = var.certificateType certificateType = var.certificateType
ownPublicCertificate = var.ownPublicCertificate ownPublicCertificate = var.ownPublicCertificate
@ -387,8 +444,6 @@ resource "google_compute_instance" "openvidu_master_node_2" {
node-type = "master" node-type = "master"
node-num = "2" node-num = "2"
} }
depends_on = [google_compute_instance.openvidu_master_node_1]
} }
# Master Node 3 # Master Node 3
@ -407,13 +462,15 @@ resource "google_compute_instance" "openvidu_master_node_3" {
} }
network_interface { network_interface {
network = "default" network = "default"
network_ip = google_compute_address.master_internal_ip[2].address
access_config {} access_config {}
} }
metadata = { metadata = {
stackName = var.stackName stackName = var.stackName
masterNodeNum = "3" masterNodeNum = "3"
masterNodePrivateIPList = local.master_node_private_ip_list
domainName = var.domainName domainName = var.domainName
certificateType = var.certificateType certificateType = var.certificateType
ownPublicCertificate = var.ownPublicCertificate ownPublicCertificate = var.ownPublicCertificate
@ -439,8 +496,6 @@ resource "google_compute_instance" "openvidu_master_node_3" {
node-type = "master" node-type = "master"
node-num = "3" node-num = "3"
} }
depends_on = [google_compute_instance.openvidu_master_node_2]
} }
# Master Node 4 # Master Node 4
@ -459,13 +514,15 @@ resource "google_compute_instance" "openvidu_master_node_4" {
} }
network_interface { network_interface {
network = "default" network = "default"
network_ip = google_compute_address.master_internal_ip[3].address
access_config {} access_config {}
} }
metadata = { metadata = {
stackName = var.stackName stackName = var.stackName
masterNodeNum = "4" masterNodeNum = "4"
masterNodePrivateIPList = local.master_node_private_ip_list
domainName = var.domainName domainName = var.domainName
certificateType = var.certificateType certificateType = var.certificateType
ownPublicCertificate = var.ownPublicCertificate ownPublicCertificate = var.ownPublicCertificate
@ -491,8 +548,6 @@ resource "google_compute_instance" "openvidu_master_node_4" {
node-type = "master" node-type = "master"
node-num = "4" node-num = "4"
} }
depends_on = [google_compute_instance.openvidu_master_node_3]
} }
# ------------------------- scale in resources ------------------------- # ------------------------- scale in resources -------------------------
@ -800,6 +855,12 @@ resource "google_cloudfunctions2_function" "scalein_function" {
} }
service_account_email = google_service_account.service_account.email service_account_email = google_service_account.service_account.email
} }
depends_on = [
google_project_service.cloudfunctions_api,
google_project_service.cloudbuild_api,
google_project_service.run_api
]
} }
# Cloud Scheduler to trigger the function every 5 minutes # Cloud Scheduler to trigger the function every 5 minutes
@ -829,6 +890,8 @@ resource "google_cloud_scheduler_job" "scale_scheduler" {
service_account_email = google_service_account.service_account.email service_account_email = google_service_account.service_account.email
} }
} }
depends_on = [google_project_service.cloudscheduler_api]
} }
locals { locals {
@ -858,7 +921,7 @@ resource "google_compute_instance_template" "media_node_template" {
metadata = { metadata = {
stackName = var.stackName stackName = var.stackName
masterNodePrivateIPList = "${google_compute_instance.openvidu_master_node_1.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_2.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_3.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_4.network_interface[0].network_ip}" masterNodePrivateIPList = local.master_node_private_ip_list
bucketAppDataName = local.isEmptyAppData ? google_storage_bucket.appdata_bucket[0].name : var.GCSAppDataBucketName bucketAppDataName = local.isEmptyAppData ? google_storage_bucket.appdata_bucket[0].name : var.GCSAppDataBucketName
bucketClusterDataName = local.isEmptyClusterData ? google_storage_bucket.clusterdata_bucket[0].name : var.GCSClusterDataBucketName bucketClusterDataName = local.isEmptyClusterData ? google_storage_bucket.clusterdata_bucket[0].name : var.GCSClusterDataBucketName
region = var.region region = var.region
@ -880,13 +943,6 @@ resource "google_compute_instance_template" "media_node_template" {
lifecycle { lifecycle {
create_before_destroy = true create_before_destroy = true
} }
depends_on = [
google_compute_instance.openvidu_master_node_1,
google_compute_instance.openvidu_master_node_2,
google_compute_instance.openvidu_master_node_3,
google_compute_instance.openvidu_master_node_4
]
} }
# Managed Instance Group for Media Nodes # Managed Instance Group for Media Nodes
@ -904,13 +960,6 @@ resource "google_compute_region_instance_group_manager" "media_node_group" {
name = "http" name = "http"
port = 7880 port = 7880
} }
depends_on = [
google_compute_instance.openvidu_master_node_1,
google_compute_instance.openvidu_master_node_2,
google_compute_instance.openvidu_master_node_3,
google_compute_instance.openvidu_master_node_4
]
} }
# Autoscaler for Media Nodes # Autoscaler for Media Nodes
@ -969,11 +1018,8 @@ get_meta() { curl -s -H "Metadata-Flavor: Google" "$${METADATA_URL}/$1"; }
# Get master node number from metadata # Get master node number from metadata
MASTER_NODE_NUM=$(get_meta "instance/attributes/masterNodeNum") MASTER_NODE_NUM=$(get_meta "instance/attributes/masterNodeNum")
# Get own private IP # Get the list of master node private IPs (static internal IPs passed via metadata)
PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip") MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPList")
# Store current private IP
PRIVATE_IP="$(/usr/local/bin/store_secret.sh save MASTER_NODE_$${MASTER_NODE_NUM}_PRIVATE_IP $PRIVATE_IP)"
# Check if secrets have been generated # Check if secrets have been generated
ALL_SECRETS_GENERATED=$(gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null || echo "false") ALL_SECRETS_GENERATED=$(gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null || echo "false")
@ -1033,33 +1079,20 @@ if [[ $MASTER_NODE_NUM -eq 1 ]] && [[ "$ALL_SECRETS_GENERATED" == "false" ]]; th
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")" ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")"
fi fi
# Wait for all master nodes to store their private IPs # Wait for master-1 to finish generating all shared secrets before reading them.
while true; do # Bounded to 360 iterations x 5s = 30 minutes.
MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP 2>/dev/null || echo "") i=0
MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP 2>/dev/null || echo "") while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do
MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP 2>/dev/null || echo "") i=$((i + 1))
MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP 2>/dev/null || echo "") if [ "$i" -ge 360 ]; then
echo "Timed out after 30 minutes waiting for master-1 to finish generating secrets" >&2
# Check if all master nodes have stored their private IPs exit 1
if [[ "$MASTER_NODE_1_PRIVATE_IP" != "" ]] &&
[[ "$MASTER_NODE_2_PRIVATE_IP" != "" ]] &&
[[ "$MASTER_NODE_3_PRIVATE_IP" != "" ]] &&
[[ "$MASTER_NODE_4_PRIVATE_IP" != "" ]]; then
break
fi fi
echo "Waiting for master-1 to finish generating secrets..."
sleep 5 sleep 5
done done
# Wait for master-1 to finish generating all shared secrets before reading them
while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do echo "Waiting for master-1 to finish generating secrets..."; sleep 5; done
# Fetch all values from Secret Manager # Fetch all values from Secret Manager
MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP)
MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP)
MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP)
MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP)
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
DOMAIN=$(gcloud secrets versions access latest --secret=DOMAIN_NAME) DOMAIN=$(gcloud secrets versions access latest --secret=DOMAIN_NAME)
OPENVIDU_PRO_LICENSE=$(gcloud secrets versions access latest --secret=OPENVIDU_PRO_LICENSE) OPENVIDU_PRO_LICENSE=$(gcloud secrets versions access latest --secret=OPENVIDU_PRO_LICENSE)
OPENVIDU_RTC_ENGINE=$(gcloud secrets versions access latest --secret=OPENVIDU_RTC_ENGINE) OPENVIDU_RTC_ENGINE=$(gcloud secrets versions access latest --secret=OPENVIDU_RTC_ENGINE)
@ -1082,8 +1115,15 @@ LIVEKIT_API_KEY=$(gcloud secrets versions access latest --secret=LIVEKIT_API_KEY
LIVEKIT_API_SECRET=$(gcloud secrets versions access latest --secret=LIVEKIT_API_SECRET) LIVEKIT_API_SECRET=$(gcloud secrets versions access latest --secret=LIVEKIT_API_SECRET)
ENABLED_MODULES=$(gcloud secrets versions access latest --secret=ENABLED_MODULES) ENABLED_MODULES=$(gcloud secrets versions access latest --secret=ENABLED_MODULES)
# Build install command # Build install command. Download the installer to a file first: process substitution
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)" # (sh <(curl ...)) would silently run an empty script and exit 0 on a transient curl failure.
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh"
if [ ! -s "$INSTALLER_SCRIPT" ]; then
echo "Downloaded OpenVidu master node installer is empty or missing" >&2
exit 1
fi
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
# Common arguments # Common arguments
COMMON_ARGS=( COMMON_ARGS=(
@ -1434,11 +1474,18 @@ EOF
check_app_ready_script = <<-EOF check_app_ready_script = <<-EOF
#!/bin/bash #!/bin/bash
# Bounded to 240 iterations x 5s = 20 minutes.
i=0
while true; do while true; do
HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy 2>/dev/null | head -n1 | awk '{print $2}') HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy 2>/dev/null | head -n1 | awk '{print $2}')
if [ "$HTTP_STATUS" == "200" ]; then if [ "$HTTP_STATUS" == "200" ]; then
break break
fi fi
i=$((i + 1))
if [ "$i" -ge 240 ]; then
echo "Timed out after 20 minutes waiting for OpenVidu to become ready" >&2
exit 1
fi
sleep 5 sleep 5
done done
EOF EOF
@ -1513,8 +1560,6 @@ CONFIG_S3_EOF
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
apt-get update && apt-get install -y
GCLOUD_VERSION=573.0.0 GCLOUD_VERSION=573.0.0
# Install google cli # Install google cli
if ! command -v gcloud >/dev/null 2>&1; then if ! command -v gcloud >/dev/null 2>&1; then
@ -1578,8 +1623,15 @@ MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPL
STACK_NAME=$(get_meta "instance/attributes/stackName") STACK_NAME=$(get_meta "instance/attributes/stackName")
PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip") PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip")
# Wait for master nodes to be ready by checking secrets # Wait for master nodes to be ready by checking secrets.
# Bounded to 180 iterations x 10s = 30 minutes.
i=0
while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do
i=$((i + 1))
if [ "$i" -ge 180 ]; then
echo "Timed out after 30 minutes waiting for master nodes to initialize secrets" >&2
exit 1
fi
echo "Waiting for master nodes to initialize secrets..." echo "Waiting for master nodes to initialize secrets..."
sleep 10 sleep 10
done done
@ -1597,8 +1649,16 @@ if [[ "$OPENVIDU_VERSION" == "none" ]]; then
exit 1 exit 1
fi fi
# Build install command for media node # Build install command for media node. Download the installer to a file first: process
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)" # substitution (sh <(curl ...)) would silently run an empty script and exit 0 on a transient
# curl failure.
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh"
if [ ! -s "$INSTALLER_SCRIPT" ]; then
echo "Downloaded OpenVidu media node installer is empty or missing" >&2
exit 1
fi
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
# Media node arguments # Media node arguments
COMMON_ARGS=( COMMON_ARGS=(
@ -1698,51 +1758,55 @@ EOF
#!/bin/bash -x #!/bin/bash -x
set -eu -o pipefail set -eu -o pipefail
# Create scripts # Check if installation already completed
cat > /usr/local/bin/install.sh << 'INSTALL_EOF' if [ -f /usr/local/bin/openvidu_install_counter.txt ]; then
# Launch on reboot
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; }
else
# Create scripts
cat > /usr/local/bin/install.sh << 'INSTALL_EOF'
${local.install_script_media} ${local.install_script_media}
INSTALL_EOF INSTALL_EOF
chmod +x /usr/local/bin/install.sh chmod +x /usr/local/bin/install.sh
cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF' cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF'
${local.graceful_shutdown_script} ${local.graceful_shutdown_script}
GRACEFUL_SHUTDOWN_EOF GRACEFUL_SHUTDOWN_EOF
chmod +x /usr/local/bin/graceful_shutdown.sh chmod +x /usr/local/bin/graceful_shutdown.sh
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
apt-get update && apt-get install -y GCLOUD_VERSION=573.0.0
# Install google cli
if ! command -v gcloud >/dev/null 2>&1; then
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update && apt-get install -y google-cloud-cli=$${GCLOUD_VERSION}-0
fi
GCLOUD_VERSION=573.0.0 # Authenticate with gcloud using instance service account
# Install google cli gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true
if ! command -v gcloud >/dev/null 2>&1; then gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google")
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
apt-get update && apt-get install -y google-cloud-cli=$${GCLOUD_VERSION}-0
fi
# Authenticate with gcloud using instance service account export HOME="/root"
gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true
gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google")
gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
export HOME="/root" # Install OpenVidu Media Node
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; }
# Install OpenVidu Media Node # Mark installation as complete
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; } echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt
# Mark installation as complete # Start OpenVidu
echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; }
# Start OpenVidu # Add cron job to check if instance is abandoned every minute
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; } cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF'
# Add cron job to check if instance is abandoned every minute
cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF'
${local.crontab_job_media} ${local.crontab_job_media}
CHECK_ABANDONED_EOF CHECK_ABANDONED_EOF
chmod +x /usr/local/bin/check_abandoned.sh chmod +x /usr/local/bin/check_abandoned.sh
echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab - echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab -
fi
EOF EOF
} }