openvidu/openvidu-deployment/CHANGELOG.md

Changelog

Every release groups its Added, Improved and Fixed entries by the cloud they apply to:

• All clouds: changes that behave the same on every cloud.
• AWS, Azure, Google Cloud, Oracle Cloud and DigitalOcean: what is specific to that cloud's templates.

The AWS, Azure and Google Cloud templates live in this folder. The Oracle Cloud and DigitalOcean templates live in the openvidu-oracle and openvidu-digitalocean repositories.

3.10.0 (unreleased)

Added

All clouds

• Elastic and High Availability deployments replace an unhealthy Media Node with a new one automatically, keeping the same number of Media Nodes. Before, a broken Media Node stayed in the cluster until it was replaced by hand. Every Media Node runs a health watchdog (the openvidu-media-health systemd service) that checks its media server every 30 seconds and asks the cloud to replace the node when:
  • the media server does not accept connections for 5 minutes,
  • the media server answers with errors or timeouts for 10 minutes (not counted while the node's CPU is saturated, because an overloaded media server briefly reports itself as not ready),
  • the node's installation failed (after a 10 minute wait, so the failure can be inspected and a persistent problem does not create new nodes in a loop),
  • or the node's installation has not finished 90 minutes after it booted.
• Safeguards of the Media Node watchdog:
  • No Media Node is replaced while no Master Node is reachable: a Master Node outage makes every Media Node fail its health check at once, and a new node could not join the cluster either. Once a Master Node is back, the media server gets its full time window to reconnect before the watchdog acts.
  • Failures are not counted during the first 15 minutes after OpenVidu (re)starts on the node.
  • The watchdog pauses itself while a node is drained during a normal scale-in.
  • journalctl -u openvidu-media-health on a Media Node shows what it is doing, and sudo touch /etc/openvidu/media-health.disabled pauses it (for example during manual maintenance); deleting the file resumes it.
• Elastic and High Availability deployments on AWS, Azure and Google Cloud can run a fixed number of Media Nodes, as DigitalOcean and Oracle Cloud already could, with the same parameter: FixedNumberOfMediaNodes on AWS, fixedNumberOfMediaNodes on the other clouds. With a value greater than 0 the deployment runs exactly that many Media Nodes with no autoscaling, and the initial, minimum, maximum and target CPU parameters are ignored. The default, 0, keeps autoscaling. An unhealthy Media Node is still replaced. Changing the number, or switching between the two modes, is a redeploy of the template with the new value. Raising the number adds Media Nodes. On AWS and Google Cloud, lowering it drains the Media Nodes it removes gracefully, as in a scale-in: each one stops accepting new Rooms and is removed when its ongoing Rooms, recordings and ingests end. Azure does not accept a number below the Media Nodes running, because they protect themselves from scale-in: drain the extra ones first (manual scale, then /usr/local/bin/stop_media_node.sh on each, which removes itself the same graceful way) and then redeploy. The administration documentation of each cloud describes the steps.

AWS

• An unhealthy Media Node is marked unhealthy in its Auto Scaling Group, which drains it and launches a new one.

Azure

• An unhealthy Media Node is reimaged in its Virtual Machine Scale Set, so the number of instances never changes.

Google Cloud

• An unhealthy Media Node is recreated in its managed instance group with a fresh disk. The group's native autohealing (TCP health check on port 7880), now also enabled in High Availability deployments, covers a Media Node VM that stops responding altogether.

Oracle Cloud

• An unhealthy Media Node is terminated and its instance pool launches a new one. With a fixed number of Media Nodes, the node is detached from the pool, which launches its replacement, and then terminated.

DigitalOcean

• With autoscaling, an unhealthy Media Node is tagged so the autoscaler deletes it and creates a new one. With a fixed number of Media Nodes, the Droplet is rebuilt in place.

Improved

All clouds

• The single node and the Master Nodes get a 200 GB disk by default on every cloud where the template sets its size. Azure and Google Cloud raise it from 100 GB (single node, Elastic Master Node and High Availability Master Nodes), as do the High Availability Master Nodes on AWS and the Elastic and High Availability Master Nodes on Oracle Cloud. Oracle Cloud single node, which used the size of the image (about 47 GB), now sets 200 GB. AWS single node and Elastic Master Node already had 200 GB, and on DigitalOcean the disk comes with the Droplet size. On Google Cloud and Oracle Cloud single node and the Google Cloud Elastic Master Node, an existing deployment keeps its disk when the template is applied again. Where the size is a parameter, set it to 100 when you apply the new template to an existing deployment (masterNodesDiskSize on Google Cloud, masterNodeDiskSize on Oracle Cloud): Google Cloud recreates a Master Node whose disk size changes.
• Parameter names are the same on every Terraform and Bicep template. The old names still work and take precedence when set, but they are deprecated and will be removed in a future release:
  • Google Cloud and Oracle Cloud single node: RTCEngine is now rtcEngine, as in their Elastic and High Availability templates and on Azure and DigitalOcean.
  • Google Cloud and DigitalOcean High Availability: masterNodesInstanceType is now masterNodeInstanceType, as in every Elastic template and in Azure High Availability.

Google Cloud

• Every machine boots the latest Ubuntu 24.04 image (the ubuntu-2404-lts image family), as on the other clouds. Before, the templates were pinned to an image from December 2024. An existing single node or Master Node is not recreated when a newer image is published. The Media Nodes move to the image family once: the first time the new template is applied to an existing Elastic or High Availability deployment, its Media Nodes are replaced.

Oracle Cloud

• Single node defaults to 2 OCPUs and 8 GB of memory, the size of its own Elastic Master Node and of the 4 vCPU and 8 GB machines of AWS and DigitalOcean. Before, it had 4 OCPUs with only 4 GB of memory.

Fixed

All clouds

• High Availability: publishing to an RTMP ingress failed. OpenVidu returns the ingress URL on port 1945 (rtmps://<your-domain>:1945/rtmp), but the load balancer only accepted RTMP on port 1935. The load balancer now accepts RTMPS on port 1945. Publish to the URL OpenVidu returns when the ingress is created, and allow port 1945 in any firewall in front of your publishers.

AWS

• Single node and Elastic: the Elastic IP was attached only after the instance had started booting, so the domain could be derived from a temporary public IP, or the instance could start without Internet access and fail its installation. The Elastic IP is now known in advance, and the instance waits up to 10 minutes for it before installing anything.

Azure

• Elastic and High Availability: redeploying the template always failed, so no parameter could be changed after the first deployment:

  • InUseSubnetCannotBeDeleted: the Master Nodes' subnets were declared outside the virtual network, so a redeploy tried to delete them while the Master Nodes used them. Every subnet is now declared inside the virtual network.
  • 'Service Uri' must not be empty: Azure Automation returns the URI of a webhook only when it creates it, so the scale-in action group lost it on every redeploy. Every deployment now creates the scale-in webhook under a new name. The webhooks of earlier deployments stay unused in the Automation account until they expire.

  A redeploy still cannot set the fixed, minimum or maximum number of Media Nodes below the Media Nodes running (UnsupportedCapacity); the administration documentation describes how to remove them first.

• Professional single node, Elastic and High Availability: the network security group rule for WebRTC over TCP opened only ports 50000 and 60000 instead of the range between them. It now opens 50000-60000, as the rule for UDP does.

• Single node and Elastic: the network security group still allowed port 9000 (MinIO) from the Internet. MinIO does not run in Azure deployments, which store recordings and application data in Azure Blob Storage, and the other clouds have kept this port closed since 3.9.0. The rule is removed.

Google Cloud

• Single node: the documentation names the media engine variable rtcEngine, but the template declared RTCEngine, so a value set as documented was not applied. Both names work now.

DigitalOcean

• Elastic and High Availability: the OpenVidu v2 compatibility module (v2compatibility) was not enabled by default, unlike on every other cloud and on DigitalOcean's Professional single node. It is now enabled by default.
• Elastic and High Availability: changing fixedNumberOfMediaNodes replaced every fixed Media Node, the ones that stay included, and cut their Rooms, because the Media Nodes' user data carried the number. It now carries only whether the number is fixed, so raising or lowering it creates or removes only the extra Media Nodes. A deployment with more than one fixed Media Node has them replaced once, the first time this version of the template is applied.