mirror of https://github.com/OpenVidu/openvidu.git
openvidu-deployment: optimize GCP HA deployment time
Parallel master node creation (drop chained depends_on), static internal IPs for masters passed via metadata (removes the runtime IP exchange through Secret Manager and its unbounded wait), bounded timeouts on ALL_SECRETS_GENERATED/check_app_ready polls, robust installer fetch (curl --retry to file instead of sh <(curl)), media node reboot guard, apt cleanup and depends_on on enabled APIs. Validated with ov-cloud-tester (sc-deploy-destroy, ha, dev): PASS. deploy 1m54s (baseline 3m30s), ready 6m26s, destroy 3m13s.master
Piwccle
parent
5a3347c12a
commit
1794e7e16b
|
|
@ -0,0 +1,144 @@
|
|||
# Cambios de documentación para `ha-optimizations` (despliegue HA GCP)
|
||||
|
||||
Instrucciones para el Claude que trabaje en el repo de docs
|
||||
`/home/sergio/Escritorio/openvidu/openvidu.io` (rama `next`, ya actualizada).
|
||||
|
||||
Estas instrucciones derivan de los cambios aplicados en
|
||||
`openvidu-deployment/pro/ha/gcp/tf-gpc-openvidu-ha.tf` en la rama `ha-optimizations`
|
||||
(paralelización de los 4 master nodes, IPs internas estáticas para los masters,
|
||||
timeouts en las esperas, fetch robusto del instalador y `depends_on` hacia las APIs).
|
||||
|
||||
RESUMEN: el ÚNICO cambio de contenido necesario son las dos cifras de tiempo de
|
||||
despliegue en el `install.md` del HA de GCP. Todo lo demás (parámetros públicos,
|
||||
capturas, otros clouds, otros deployment types, ficheros de config on-node) se
|
||||
mantiene igual. Léelo entero antes de tocar nada.
|
||||
|
||||
---
|
||||
|
||||
## (a) Cifras de tiempo a actualizar — OBLIGATORIO
|
||||
|
||||
Fichero: `docs/docs/self-hosting/ha/gcp/install.md`
|
||||
|
||||
Hay DOS ocurrencias de "7 to 12 minutes" en este fichero (aprox. líneas 249 y 308).
|
||||
Localízalas por el texto literal, no confíes solo en el número de línea (puede haberse
|
||||
desplazado). Sustituye la cifra por el placeholder y deja una nota para rellenarla con
|
||||
la medición real de `ov-cloud-tester` tras estos cambios (el despliegue ahora es más
|
||||
rápido: masters en paralelo y sin el handshake de IPs por Secret Manager).
|
||||
|
||||
### Ocurrencia 1 (sección "## Deploying the stack", ~línea 249)
|
||||
|
||||
Texto actual literal:
|
||||
|
||||
```
|
||||
When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around 7 to 12 minutes for the nodes to install OpenVidu.
|
||||
```
|
||||
|
||||
Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`):
|
||||
|
||||
```
|
||||
When you are satisfied with your input values, click _"Continue"_ and then _"Create deployment"_. The deployment will be validated and all resources will be created. Wait around X to Y minutes for the nodes to install OpenVidu.
|
||||
```
|
||||
|
||||
### Ocurrencia 2 (sección "## Checking...", ~línea 308)
|
||||
|
||||
Texto actual literal:
|
||||
|
||||
```
|
||||
When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about 7 to 12 minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
|
||||
```
|
||||
|
||||
Texto propuesto (rellenar `X`/`Y` con la medición de `ov-cloud-tester`):
|
||||
|
||||
```
|
||||
When your Google Cloud Platform deployment reaches the **`Active`** state, it means that all resources have been created. You will need to wait about X to Y minutes for the instances to install OpenVidu, as mentioned before. After this time, try connecting to the deployment URL. If it doesn't work, we recommend checking the previous section. Once everything is ready, you can check the [Administration](./admin.md) section to learn how to manage your deployment.
|
||||
```
|
||||
|
||||
> Placeholder a rellenar: `X to Y minutes — rellenar con la medición de ov-cloud-tester`.
|
||||
> Mantén las dos ocurrencias con la MISMA cifra (la segunda dice "as mentioned before").
|
||||
|
||||
### NO tocar las cifras de otros ficheros
|
||||
|
||||
Estos cambios afectan SOLO al HA de GCP. NO modifiques las cifras "7 to 12 minutes"
|
||||
que aparecen en otros documentos (son otros clouds / otros deployment types y están
|
||||
fuera del alcance de este trabajo):
|
||||
|
||||
- `docs/docs/self-hosting/elastic/gcp/install.md` (líneas ~236 y ~295)
|
||||
- `docs/docs/self-hosting/elastic/azure/install.md` (líneas ~95 y ~149)
|
||||
- Cualquier otro `install.md` de la doc.
|
||||
|
||||
---
|
||||
|
||||
## (b) Parámetros públicos de la plantilla GCP: SIN CAMBIOS — confirmación
|
||||
|
||||
NO hay que tocar tablas de parámetros ni capturas de la sección de parámetros.
|
||||
|
||||
Motivo: las nuevas IPs estáticas de los master nodes son **internas**
|
||||
(`google_compute_address` con `address_type = "INTERNAL"`, reservadas dentro de la
|
||||
subred `default` de la región) y se asignan vía `network_interface.network_ip`. No se
|
||||
ha añadido, eliminado ni renombrado ninguna variable de entrada ni salida de Terraform
|
||||
(`variables.tf` y `output.tf` intactos). Por tanto:
|
||||
|
||||
- La tabla "Mandatory Parameters" y "Optional Parameters" de
|
||||
`docs/docs/self-hosting/ha/gcp/install.md` (aprox. líneas 95–230) **se queda igual**.
|
||||
- No hay parámetros nuevos que el usuario deba introducir en Infrastructure Manager.
|
||||
- Las capturas del formulario de parámetros **no cambian**.
|
||||
|
||||
---
|
||||
|
||||
## (c) Otras afirmaciones de la doc GCP HA que puedan quedar obsoletas
|
||||
|
||||
Se ha revisado toda la doc del HA de GCP (`docs/docs/self-hosting/ha/gcp/*.md`) y los
|
||||
includes compartidos. Conclusiones:
|
||||
|
||||
### c.1 — Secretos `MASTER_NODE_{1..4}_PRIVATE_IP` de Secret Manager (ELIMINADOS)
|
||||
|
||||
En `ha-optimizations` se han eliminado del despliegue los 4 secretos de Secret Manager
|
||||
`MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` (ya no se generan: las IPs de
|
||||
los masters son ahora estáticas y se pasan por metadata). La doc del HA de GCP **no los
|
||||
menciona por nombre en el texto**, así que no hay texto que corregir.
|
||||
|
||||
- Único impacto posible: la captura del Secret Manager en
|
||||
`install.md` (sección "Check deployment outputs in GCP Secret Manager"),
|
||||
imagen `assets/images/platform/self-hosting/shared/gcp/secrets-manager.png`.
|
||||
Esa captura PODRÍA mostrar los 4 secretos de IP que ya no existirán. Es puramente
|
||||
cosmético (el texto solo dice "you will see all the secrets by their name"). Acción
|
||||
OPCIONAL: si se regenera la captura en algún momento, hacerlo tras un despliegue con
|
||||
`ha-optimizations`. NO es bloqueante para este cambio.
|
||||
|
||||
### c.2 — `backup-and-restore.md`: NO TOCAR
|
||||
|
||||
`docs/docs/self-hosting/how-to-guides/backup-and-restore.md` (aprox. líneas 676–692)
|
||||
documenta variables `MASTER_NODE_1_PRIVATE_IP` … `MASTER_NODE_4_PRIVATE_IP` dentro del
|
||||
fichero de configuración on-node `/opt/openvidu/config/node/master-node.env`.
|
||||
|
||||
IMPORTANTE: **esas NO son los secretos eliminados**. Son variables del fichero de
|
||||
configuración que el instalador sigue escribiendo a partir del flag
|
||||
`--master-node-private-ip-list` (que se sigue pasando, ahora derivado de las IPs
|
||||
estáticas). Ese flujo no cambia. **No modifiques `backup-and-restore.md`.**
|
||||
|
||||
### c.3 — `upgrade.md`: NO TOCAR
|
||||
|
||||
`docs/docs/self-hosting/ha/gcp/upgrade.md` (línea ~91) menciona `store_secret.sh` y el
|
||||
secreto `OPENVIDU_VERSION`. Ni el script `store_secret.sh` ni el secreto
|
||||
`OPENVIDU_VERSION` se han tocado. **No requiere cambios.**
|
||||
|
||||
### c.4 — Flujo interno (ALL_SECRETS_GENERATED, handshake de IPs): no documentado
|
||||
|
||||
El mecanismo interno de coordinación (líder master-1 genera secretos y marca
|
||||
`ALL_SECRETS_GENERATED`, polling, etc.) no está documentado públicamente, y su cambio
|
||||
(eliminación del intercambio de IPs por Secret Manager) no altera nada de cara al
|
||||
usuario. No hay texto que actualizar.
|
||||
|
||||
---
|
||||
|
||||
## Checklist de aplicación
|
||||
|
||||
- [ ] En `docs/docs/self-hosting/ha/gcp/install.md`, sustituir "7 to 12 minutes" por
|
||||
"X to Y minutes" en las DOS ocurrencias (~249 y ~308), con la misma cifra en ambas.
|
||||
- [ ] Medir el tiempo real de despliegue con `ov-cloud-tester` sobre `ha-optimizations`
|
||||
y rellenar `X`/`Y`.
|
||||
- [ ] Verificar que NO se han tocado tablas de parámetros ni capturas.
|
||||
- [ ] Verificar que NO se han tocado `backup-and-restore.md`, `upgrade.md`, ni las
|
||||
cifras de tiempo de elastic/azure/otros clouds.
|
||||
- [ ] (Opcional) Regenerar `secrets-manager.png` si se desea reflejar la ausencia de los
|
||||
4 secretos de IP.
|
||||
|
|
@ -20,14 +20,15 @@ resource "google_secret_manager_secret" "openvidu_shared_info" {
|
|||
"OPENVIDU_PRO_LICENSE", "OPENVIDU_RTC_ENGINE", "REDIS_PASSWORD", "MONGO_ADMIN_USERNAME",
|
||||
"MONGO_ADMIN_PASSWORD", "MONGO_REPLICA_SET_KEY", "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY",
|
||||
"DASHBOARD_ADMIN_USERNAME", "DASHBOARD_ADMIN_PASSWORD", "GRAFANA_ADMIN_USERNAME",
|
||||
"GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED",
|
||||
"MASTER_NODE_1_PRIVATE_IP", "MASTER_NODE_2_PRIVATE_IP", "MASTER_NODE_3_PRIVATE_IP", "MASTER_NODE_4_PRIVATE_IP"
|
||||
"GRAFANA_ADMIN_PASSWORD", "ENABLED_MODULES", "OPENVIDU_VERSION", "ALL_SECRETS_GENERATED"
|
||||
])
|
||||
|
||||
secret_id = each.key
|
||||
replication {
|
||||
auto {}
|
||||
}
|
||||
|
||||
depends_on = [google_project_service.secretmanager_api]
|
||||
}
|
||||
|
||||
# GCS buckets for HA deployment
|
||||
|
|
@ -37,6 +38,8 @@ resource "google_storage_bucket" "appdata_bucket" {
|
|||
location = var.region
|
||||
force_destroy = true
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
depends_on = [google_project_service.storage_api]
|
||||
}
|
||||
|
||||
resource "google_storage_bucket" "clusterdata_bucket" {
|
||||
|
|
@ -45,12 +48,16 @@ resource "google_storage_bucket" "clusterdata_bucket" {
|
|||
location = var.region
|
||||
force_destroy = true
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
depends_on = [google_project_service.storage_api]
|
||||
}
|
||||
|
||||
# Service account for the instances
|
||||
resource "google_service_account" "service_account" {
|
||||
account_id = lower("${substr(var.stackName, 0, 12)}-sa")
|
||||
display_name = "OpenVidu instance service account"
|
||||
|
||||
depends_on = [google_project_service.iam_api]
|
||||
}
|
||||
|
||||
# IAM bindings for the service account
|
||||
|
|
@ -58,6 +65,8 @@ resource "google_project_iam_member" "iam_project_role" {
|
|||
project = var.projectId
|
||||
role = "roles/owner"
|
||||
member = "serviceAccount:${google_service_account.service_account.email}"
|
||||
|
||||
depends_on = [google_project_service.cloudresourcemanager_api]
|
||||
}
|
||||
|
||||
# External SSH access to Master Nodes
|
||||
|
|
@ -72,6 +81,8 @@ resource "google_compute_firewall" "external_master_ssh" {
|
|||
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = [lower("${var.stackName}-master-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# External access to Media Nodes (SSH and media traffic)
|
||||
|
|
@ -90,6 +101,8 @@ resource "google_compute_firewall" "external_media_access" {
|
|||
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = [lower("${var.stackName}-media-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Load Balancer health checks and HTTP traffic to Master Nodes
|
||||
|
|
@ -104,6 +117,8 @@ resource "google_compute_firewall" "lb_to_master_http" {
|
|||
|
||||
source_ranges = ["0.0.0.0/0"]
|
||||
target_tags = [lower("${var.stackName}-master-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Master node internal services communication
|
||||
|
|
@ -129,6 +144,8 @@ resource "google_compute_firewall" "master_to_master_internal" {
|
|||
|
||||
source_tags = [lower("${var.stackName}-master-node")]
|
||||
target_tags = [lower("${var.stackName}-master-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Media Nodes to Master Nodes communication
|
||||
|
|
@ -153,6 +170,8 @@ resource "google_compute_firewall" "media_to_master_services" {
|
|||
|
||||
source_tags = [lower("${var.stackName}-media-node")]
|
||||
target_tags = [lower("${var.stackName}-master-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Master Nodes to Media Nodes communication
|
||||
|
|
@ -171,6 +190,8 @@ resource "google_compute_firewall" "master_to_media_services" {
|
|||
|
||||
source_tags = [lower("${var.stackName}-master-node")]
|
||||
target_tags = [lower("${var.stackName}-media-node")]
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -179,6 +200,8 @@ resource "google_compute_address" "nlb_ip" {
|
|||
count = var.publicIpAddress == "" ? 1 : 0
|
||||
name = lower("${var.stackName}-nlb-ip")
|
||||
region = var.region
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Data source for existing IP address when publicIpAddress is provided
|
||||
|
|
@ -206,6 +229,8 @@ resource "google_compute_region_health_check" "tcp_health_check" {
|
|||
timeout_sec = 5
|
||||
healthy_threshold = 3
|
||||
unhealthy_threshold = 4
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Regional backend service for the TCP NLB
|
||||
|
|
@ -289,6 +314,34 @@ locals {
|
|||
is_c4a_instance = startswith(var.masterNodesInstanceType, "c4a-")
|
||||
}
|
||||
|
||||
# Default subnetwork the master instances live in (they use network = "default" with no
|
||||
# explicit subnetwork, so GCP places them in the auto-mode default subnetwork of the region)
|
||||
data "google_compute_subnetwork" "default" {
|
||||
name = "default"
|
||||
region = var.region
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
# Static internal IPs for the master nodes. Reserving them up front lets the media instance
|
||||
# template and every master receive the full list without waiting for the master instances to
|
||||
# boot, which removes the runtime IP-exchange handshake through Secret Manager.
|
||||
resource "google_compute_address" "master_internal_ip" {
|
||||
count = 4
|
||||
name = lower("${var.stackName}-master-node-${count.index + 1}-internal-ip")
|
||||
address_type = "INTERNAL"
|
||||
region = var.region
|
||||
subnetwork = data.google_compute_subnetwork.default.id
|
||||
|
||||
depends_on = [google_project_service.compute_api]
|
||||
}
|
||||
|
||||
locals {
|
||||
# Master node private IPs come from the reserved static internal addresses (index 0 -> master
|
||||
# node 1, ...). Comma-separated to match the installer's --master-node-private-ip-list format.
|
||||
master_node_private_ip_list = join(",", google_compute_address.master_internal_ip[*].address)
|
||||
}
|
||||
|
||||
# Master Node 1
|
||||
resource "google_compute_instance" "openvidu_master_node_1" {
|
||||
name = lower("${var.stackName}-master-node-1")
|
||||
|
|
@ -306,12 +359,14 @@ resource "google_compute_instance" "openvidu_master_node_1" {
|
|||
|
||||
network_interface {
|
||||
network = "default"
|
||||
network_ip = google_compute_address.master_internal_ip[0].address
|
||||
access_config {}
|
||||
}
|
||||
|
||||
metadata = {
|
||||
stackName = var.stackName
|
||||
masterNodeNum = "1"
|
||||
masterNodePrivateIPList = local.master_node_private_ip_list
|
||||
domainName = var.domainName
|
||||
certificateType = var.certificateType
|
||||
ownPublicCertificate = var.ownPublicCertificate
|
||||
|
|
@ -356,12 +411,14 @@ resource "google_compute_instance" "openvidu_master_node_2" {
|
|||
|
||||
network_interface {
|
||||
network = "default"
|
||||
network_ip = google_compute_address.master_internal_ip[1].address
|
||||
access_config {}
|
||||
}
|
||||
|
||||
metadata = {
|
||||
stackName = var.stackName
|
||||
masterNodeNum = "2"
|
||||
masterNodePrivateIPList = local.master_node_private_ip_list
|
||||
domainName = var.domainName
|
||||
certificateType = var.certificateType
|
||||
ownPublicCertificate = var.ownPublicCertificate
|
||||
|
|
@ -387,8 +444,6 @@ resource "google_compute_instance" "openvidu_master_node_2" {
|
|||
node-type = "master"
|
||||
node-num = "2"
|
||||
}
|
||||
|
||||
depends_on = [google_compute_instance.openvidu_master_node_1]
|
||||
}
|
||||
|
||||
# Master Node 3
|
||||
|
|
@ -408,12 +463,14 @@ resource "google_compute_instance" "openvidu_master_node_3" {
|
|||
|
||||
network_interface {
|
||||
network = "default"
|
||||
network_ip = google_compute_address.master_internal_ip[2].address
|
||||
access_config {}
|
||||
}
|
||||
|
||||
metadata = {
|
||||
stackName = var.stackName
|
||||
masterNodeNum = "3"
|
||||
masterNodePrivateIPList = local.master_node_private_ip_list
|
||||
domainName = var.domainName
|
||||
certificateType = var.certificateType
|
||||
ownPublicCertificate = var.ownPublicCertificate
|
||||
|
|
@ -439,8 +496,6 @@ resource "google_compute_instance" "openvidu_master_node_3" {
|
|||
node-type = "master"
|
||||
node-num = "3"
|
||||
}
|
||||
|
||||
depends_on = [google_compute_instance.openvidu_master_node_2]
|
||||
}
|
||||
|
||||
# Master Node 4
|
||||
|
|
@ -460,12 +515,14 @@ resource "google_compute_instance" "openvidu_master_node_4" {
|
|||
|
||||
network_interface {
|
||||
network = "default"
|
||||
network_ip = google_compute_address.master_internal_ip[3].address
|
||||
access_config {}
|
||||
}
|
||||
|
||||
metadata = {
|
||||
stackName = var.stackName
|
||||
masterNodeNum = "4"
|
||||
masterNodePrivateIPList = local.master_node_private_ip_list
|
||||
domainName = var.domainName
|
||||
certificateType = var.certificateType
|
||||
ownPublicCertificate = var.ownPublicCertificate
|
||||
|
|
@ -491,8 +548,6 @@ resource "google_compute_instance" "openvidu_master_node_4" {
|
|||
node-type = "master"
|
||||
node-num = "4"
|
||||
}
|
||||
|
||||
depends_on = [google_compute_instance.openvidu_master_node_3]
|
||||
}
|
||||
|
||||
# ------------------------- scale in resources -------------------------
|
||||
|
|
@ -800,6 +855,12 @@ resource "google_cloudfunctions2_function" "scalein_function" {
|
|||
}
|
||||
service_account_email = google_service_account.service_account.email
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
google_project_service.cloudfunctions_api,
|
||||
google_project_service.cloudbuild_api,
|
||||
google_project_service.run_api
|
||||
]
|
||||
}
|
||||
|
||||
# Cloud Scheduler to trigger the function every 5 minutes
|
||||
|
|
@ -829,6 +890,8 @@ resource "google_cloud_scheduler_job" "scale_scheduler" {
|
|||
service_account_email = google_service_account.service_account.email
|
||||
}
|
||||
}
|
||||
|
||||
depends_on = [google_project_service.cloudscheduler_api]
|
||||
}
|
||||
|
||||
locals {
|
||||
|
|
@ -858,7 +921,7 @@ resource "google_compute_instance_template" "media_node_template" {
|
|||
|
||||
metadata = {
|
||||
stackName = var.stackName
|
||||
masterNodePrivateIPList = "${google_compute_instance.openvidu_master_node_1.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_2.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_3.network_interface[0].network_ip},${google_compute_instance.openvidu_master_node_4.network_interface[0].network_ip}"
|
||||
masterNodePrivateIPList = local.master_node_private_ip_list
|
||||
bucketAppDataName = local.isEmptyAppData ? google_storage_bucket.appdata_bucket[0].name : var.GCSAppDataBucketName
|
||||
bucketClusterDataName = local.isEmptyClusterData ? google_storage_bucket.clusterdata_bucket[0].name : var.GCSClusterDataBucketName
|
||||
region = var.region
|
||||
|
|
@ -880,13 +943,6 @@ resource "google_compute_instance_template" "media_node_template" {
|
|||
lifecycle {
|
||||
create_before_destroy = true
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
google_compute_instance.openvidu_master_node_1,
|
||||
google_compute_instance.openvidu_master_node_2,
|
||||
google_compute_instance.openvidu_master_node_3,
|
||||
google_compute_instance.openvidu_master_node_4
|
||||
]
|
||||
}
|
||||
|
||||
# Managed Instance Group for Media Nodes
|
||||
|
|
@ -904,13 +960,6 @@ resource "google_compute_region_instance_group_manager" "media_node_group" {
|
|||
name = "http"
|
||||
port = 7880
|
||||
}
|
||||
|
||||
depends_on = [
|
||||
google_compute_instance.openvidu_master_node_1,
|
||||
google_compute_instance.openvidu_master_node_2,
|
||||
google_compute_instance.openvidu_master_node_3,
|
||||
google_compute_instance.openvidu_master_node_4
|
||||
]
|
||||
}
|
||||
|
||||
# Autoscaler for Media Nodes
|
||||
|
|
@ -969,11 +1018,8 @@ get_meta() { curl -s -H "Metadata-Flavor: Google" "$${METADATA_URL}/$1"; }
|
|||
# Get master node number from metadata
|
||||
MASTER_NODE_NUM=$(get_meta "instance/attributes/masterNodeNum")
|
||||
|
||||
# Get own private IP
|
||||
PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip")
|
||||
|
||||
# Store current private IP
|
||||
PRIVATE_IP="$(/usr/local/bin/store_secret.sh save MASTER_NODE_$${MASTER_NODE_NUM}_PRIVATE_IP $PRIVATE_IP)"
|
||||
# Get the list of master node private IPs (static internal IPs passed via metadata)
|
||||
MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPList")
|
||||
|
||||
# Check if secrets have been generated
|
||||
ALL_SECRETS_GENERATED=$(gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null || echo "false")
|
||||
|
|
@ -1033,33 +1079,20 @@ if [[ $MASTER_NODE_NUM -eq 1 ]] && [[ "$ALL_SECRETS_GENERATED" == "false" ]]; th
|
|||
ALL_SECRETS_GENERATED="$(/usr/local/bin/store_secret.sh save ALL_SECRETS_GENERATED "true")"
|
||||
fi
|
||||
|
||||
# Wait for all master nodes to store their private IPs
|
||||
while true; do
|
||||
MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP 2>/dev/null || echo "")
|
||||
MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP 2>/dev/null || echo "")
|
||||
MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP 2>/dev/null || echo "")
|
||||
MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP 2>/dev/null || echo "")
|
||||
|
||||
# Check if all master nodes have stored their private IPs
|
||||
if [[ "$MASTER_NODE_1_PRIVATE_IP" != "" ]] &&
|
||||
[[ "$MASTER_NODE_2_PRIVATE_IP" != "" ]] &&
|
||||
[[ "$MASTER_NODE_3_PRIVATE_IP" != "" ]] &&
|
||||
[[ "$MASTER_NODE_4_PRIVATE_IP" != "" ]]; then
|
||||
break
|
||||
# Wait for master-1 to finish generating all shared secrets before reading them.
|
||||
# Bounded to 360 iterations x 5s = 30 minutes.
|
||||
i=0
|
||||
while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do
|
||||
i=$((i + 1))
|
||||
if [ "$i" -ge 360 ]; then
|
||||
echo "Timed out after 30 minutes waiting for master-1 to finish generating secrets" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for master-1 to finish generating secrets..."
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# Wait for master-1 to finish generating all shared secrets before reading them
|
||||
while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do echo "Waiting for master-1 to finish generating secrets..."; sleep 5; done
|
||||
|
||||
# Fetch all values from Secret Manager
|
||||
MASTER_NODE_1_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_1_PRIVATE_IP)
|
||||
MASTER_NODE_2_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_2_PRIVATE_IP)
|
||||
MASTER_NODE_3_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_3_PRIVATE_IP)
|
||||
MASTER_NODE_4_PRIVATE_IP=$(gcloud secrets versions access latest --secret=MASTER_NODE_4_PRIVATE_IP)
|
||||
MASTER_NODE_PRIVATE_IP_LIST="$MASTER_NODE_1_PRIVATE_IP,$MASTER_NODE_2_PRIVATE_IP,$MASTER_NODE_3_PRIVATE_IP,$MASTER_NODE_4_PRIVATE_IP"
|
||||
|
||||
DOMAIN=$(gcloud secrets versions access latest --secret=DOMAIN_NAME)
|
||||
OPENVIDU_PRO_LICENSE=$(gcloud secrets versions access latest --secret=OPENVIDU_PRO_LICENSE)
|
||||
OPENVIDU_RTC_ENGINE=$(gcloud secrets versions access latest --secret=OPENVIDU_RTC_ENGINE)
|
||||
|
|
@ -1082,8 +1115,15 @@ LIVEKIT_API_KEY=$(gcloud secrets versions access latest --secret=LIVEKIT_API_KEY
|
|||
LIVEKIT_API_SECRET=$(gcloud secrets versions access latest --secret=LIVEKIT_API_SECRET)
|
||||
ENABLED_MODULES=$(gcloud secrets versions access latest --secret=ENABLED_MODULES)
|
||||
|
||||
# Build install command
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh)"
|
||||
# Build install command. Download the installer to a file first: process substitution
|
||||
# (sh <(curl ...)) would silently run an empty script and exit 0 on a transient curl failure.
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_master_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_master_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu master node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Common arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -1434,11 +1474,18 @@ EOF
|
|||
|
||||
check_app_ready_script = <<-EOF
|
||||
#!/bin/bash
|
||||
# Bounded to 240 iterations x 5s = 20 minutes.
|
||||
i=0
|
||||
while true; do
|
||||
HTTP_STATUS=$(curl -Ik http://localhost:7880/health/caddy 2>/dev/null | head -n1 | awk '{print $2}')
|
||||
if [ "$HTTP_STATUS" == "200" ]; then
|
||||
break
|
||||
fi
|
||||
i=$((i + 1))
|
||||
if [ "$i" -ge 240 ]; then
|
||||
echo "Timed out after 20 minutes waiting for OpenVidu to become ready" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
EOF
|
||||
|
|
@ -1513,8 +1560,6 @@ CONFIG_S3_EOF
|
|||
|
||||
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
|
||||
|
||||
apt-get update && apt-get install -y
|
||||
|
||||
GCLOUD_VERSION=573.0.0
|
||||
# Install google cli
|
||||
if ! command -v gcloud >/dev/null 2>&1; then
|
||||
|
|
@ -1578,8 +1623,15 @@ MASTER_NODE_PRIVATE_IP_LIST=$(get_meta "instance/attributes/masterNodePrivateIPL
|
|||
STACK_NAME=$(get_meta "instance/attributes/stackName")
|
||||
PRIVATE_IP=$(get_meta "instance/network-interfaces/0/ip")
|
||||
|
||||
# Wait for master nodes to be ready by checking secrets
|
||||
# Wait for master nodes to be ready by checking secrets.
|
||||
# Bounded to 180 iterations x 10s = 30 minutes.
|
||||
i=0
|
||||
while ! gcloud secrets versions access latest --secret=ALL_SECRETS_GENERATED 2>/dev/null | grep -q "true"; do
|
||||
i=$((i + 1))
|
||||
if [ "$i" -ge 180 ]; then
|
||||
echo "Timed out after 30 minutes waiting for master nodes to initialize secrets" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for master nodes to initialize secrets..."
|
||||
sleep 10
|
||||
done
|
||||
|
|
@ -1597,8 +1649,16 @@ if [[ "$OPENVIDU_VERSION" == "none" ]]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# Build install command for media node
|
||||
INSTALL_COMMAND="sh <(curl -fsSL http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh)"
|
||||
# Build install command for media node. Download the installer to a file first: process
|
||||
# substitution (sh <(curl ...)) would silently run an empty script and exit 0 on a transient
|
||||
# curl failure.
|
||||
INSTALLER_SCRIPT="/tmp/install_ov_media_node.sh"
|
||||
curl -fsSL --retry 8 --retry-all-errors --retry-delay 5 -o "$INSTALLER_SCRIPT" "http://get.openvidu.io/pro/ha/$OPENVIDU_VERSION/install_ov_media_node.sh"
|
||||
if [ ! -s "$INSTALLER_SCRIPT" ]; then
|
||||
echo "Downloaded OpenVidu media node installer is empty or missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
INSTALL_COMMAND="sh $INSTALLER_SCRIPT"
|
||||
|
||||
# Media node arguments
|
||||
COMMON_ARGS=(
|
||||
|
|
@ -1698,51 +1758,55 @@ EOF
|
|||
#!/bin/bash -x
|
||||
set -eu -o pipefail
|
||||
|
||||
# Create scripts
|
||||
cat > /usr/local/bin/install.sh << 'INSTALL_EOF'
|
||||
# Check if installation already completed
|
||||
if [ -f /usr/local/bin/openvidu_install_counter.txt ]; then
|
||||
# Launch on reboot
|
||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; }
|
||||
else
|
||||
# Create scripts
|
||||
cat > /usr/local/bin/install.sh << 'INSTALL_EOF'
|
||||
${local.install_script_media}
|
||||
INSTALL_EOF
|
||||
chmod +x /usr/local/bin/install.sh
|
||||
chmod +x /usr/local/bin/install.sh
|
||||
|
||||
cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF'
|
||||
cat > /usr/local/bin/graceful_shutdown.sh << 'GRACEFUL_SHUTDOWN_EOF'
|
||||
${local.graceful_shutdown_script}
|
||||
GRACEFUL_SHUTDOWN_EOF
|
||||
chmod +x /usr/local/bin/graceful_shutdown.sh
|
||||
chmod +x /usr/local/bin/graceful_shutdown.sh
|
||||
|
||||
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
|
||||
echo "DPkg::Lock::Timeout \"-1\";" > /etc/apt/apt.conf.d/99timeout
|
||||
|
||||
apt-get update && apt-get install -y
|
||||
|
||||
GCLOUD_VERSION=573.0.0
|
||||
# Install google cli
|
||||
if ! command -v gcloud >/dev/null 2>&1; then
|
||||
GCLOUD_VERSION=573.0.0
|
||||
# Install google cli
|
||||
if ! command -v gcloud >/dev/null 2>&1; then
|
||||
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
|
||||
apt-get update && apt-get install -y google-cloud-cli=$${GCLOUD_VERSION}-0
|
||||
fi
|
||||
fi
|
||||
|
||||
# Authenticate with gcloud using instance service account
|
||||
gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true
|
||||
gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google")
|
||||
gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
|
||||
# Authenticate with gcloud using instance service account
|
||||
gcloud auth activate-service-account --key-file=/dev/null 2>/dev/null || true
|
||||
gcloud config set account $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email" -H "Metadata-Flavor: Google")
|
||||
gcloud config set project $(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
|
||||
|
||||
export HOME="/root"
|
||||
export HOME="/root"
|
||||
|
||||
# Install OpenVidu Media Node
|
||||
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; }
|
||||
# Install OpenVidu Media Node
|
||||
/usr/local/bin/install.sh || { echo "[OpenVidu] error installing OpenVidu Media Node"; exit 1; }
|
||||
|
||||
# Mark installation as complete
|
||||
echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt
|
||||
# Mark installation as complete
|
||||
echo "installation_complete" > /usr/local/bin/openvidu_install_counter.txt
|
||||
|
||||
# Start OpenVidu
|
||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; }
|
||||
# Start OpenVidu
|
||||
systemctl start openvidu || { echo "[OpenVidu] error starting OpenVidu"; exit 1; }
|
||||
|
||||
# Add cron job to check if instance is abandoned every minute
|
||||
cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF'
|
||||
# Add cron job to check if instance is abandoned every minute
|
||||
cat > /usr/local/bin/check_abandoned.sh << 'CHECK_ABANDONED_EOF'
|
||||
${local.crontab_job_media}
|
||||
CHECK_ABANDONED_EOF
|
||||
chmod +x /usr/local/bin/check_abandoned.sh
|
||||
chmod +x /usr/local/bin/check_abandoned.sh
|
||||
|
||||
echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab -
|
||||
echo "*/1 * * * * /usr/local/bin/check_abandoned.sh > /var/log/openvidu-abandoned-check.log 2>&1" | crontab -
|
||||
fi
|
||||
EOF
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue