Boundless Possibilities Revealed in AI Integrations with Real Time Streaming

As artificial intelligence transforms operations across the video streaming ecosystem decision makers looking for real-time streaming solutions need to be sure they choose a platform that maximizes the AI potential in their use cases.

That’s not always easy to do, given how quickly the latest generation of AI innovations has exploded into the marketplace. But there’s already plenty of evidence to go on when it comes to assessing Red5’s progress toward making AI-enabled solutions readily available for use with Experience Delivery Network (XDN) Architecture.

One can go a long way toward ascertaining how well a real-time streaming platform works with AI if the platform has already been integrated with multiple vendor solutions that heavily rely on the latest advances in AI or older AI and machine-learning (ML) formats. And it’s also a good idea to dig a little deeper for an answer to the most fundamental question, which is whether the platform’s architecture supports the flexibility and functionality essential to getting the most out of AI applications, including those that are meant to support specific use cases running on the platform as well as those that can be applied to enhance performance of the platform itself.

On all counts, decision makers will find that nothing rivals Red5’s XDN Architecture as the foundation for real-time multidirectional streaming that gets the most out of AI.

To see why that’s the case, a good place to start is with a review of Red5’s partnerships with suppliers whose solutions heavily depend on the latest developments in AI. Then we’ll provide a broader perspective on the XDN AI potential by looking at capabilities introduced by key cloud computing partners, the platform’s flexibility in tailoring real-time streaming operations to specific needs, and new ideas that could add efficiencies to platform usage.

The Mutually Beneficial Relationship between AI and Real-Time Streaming

Red5 has long operated on the understanding that full actualization of the symbiotic relationship between real-time streaming and AI is essential to reaching the full potential in both realms. On the one hand, real-time streaming enables AI applications that would be unachievable on traditional one-way streaming infrastructure. At the same time, implementing real-time streaming without support for optimal utilization of AI would undermine achieving all that can be done with real-time streaming.

There are plenty of scenarios where this relationship comes into play:

• AI and advanced applications of ML make it possible through text, object and facial recognition to pull together clips, graphics, stats or any other data files in content production. Real-time streaming makes it possible to put these capabilities to work with sports and other live productions.
• Sports micro-betting uses AI to build moment-to-moment wagering opportunities with betting odds. Real-time streaming moves micro-betting from reliance on second-screen data feeds to integration with primary screen viewing in sync with what’s happening on the field of play.
• Generative AI applying large language model (LLM) algorithms supports instant speech-to-text generation for closed captioning in multilingual environments, while video language model (VLM) enhancements make it possible to produce voice-accurate dubbing of commentary with lip-synced verisimilitude. Real-time streaming can enable simultaneous delivery of live commentary in multiple languages through closed captioning or dubbed presentations from dispersed locations across town or across the globe.
• AI enables human-like voice responsiveness from chatbot avatars. Real-time streaming enables realistic conversational flows between chatbots and users.
• AI has become a major factor in video game playing, often involving use of virtual reality (VR) or augmented reality (AR) technology, with innovations introduced by leading game engine suppliers fueling more realistic reactions by in-game non-player characters (NPCs) to players’ moves, changes in game narratives based on player decisions, and much else. Real-time streaming brings all these innovations into the multiplayer arena without the restrictions on dynamic scene rendering and realism imposed by traditional streaming.
• AI paired with extended reality (XR) applications is creating in-venue blends of virtual and real experience at sporting events, concerts and other big crowd gatherings. Real-time streaming ensures everyone’s on-screen experience syncs up with what’s happening around them.
• AI makes it possible to identify and track objects and faces across multi-screen compilations of camera feeds from expansive fields of real-time surveillance operations. As enabled by the unique capabilities embodied in Red5’s TrueTime MultiView for SurveillanceTM toolset, real-time streaming can be used to compile those camera feeds into multiscreen streams to operations centers, where personnel can instantly call up any feed or subset of feeds for full-screen scrutiny in tandem with the AI analytics feedback.

The list goes on and on touching all the sectors where AI has a role to play in video streaming.

AI Solutions Delivered by Red5 Partners

Red5 has forged partnerships with a wide range of suppliers offering AI-fueled solutions in order to make these kinds of capabilities readily available to customers whether they choose to implement their XDN infrastructures through the fully managed Red5 Cloud software-as-a-service (SaaS) or through independent use of the Red5 Pro portfolio of SDKs and APIs. Whichever approach is taken, XDN infrastructure supports simultaneous connectivity with end-to-end latencies at or below 250ms over any distance across any number of end points.

The current list of Red5 partners making innovative use of AI in these categories include these best-of-breed suppliers:

Content Production

Magnifi

Magnifi is an editing platform that provides an AI-facilitated approach to creating highlights and other enhancements relevant to core production output. Its use of AI makes it far easier and faster for producers to extract material wherever it can be found to create enhancements tuned to each receiving device that serve to deepen fan engagement and drive revenue growth. The company has trained its AI models to process more than 70 types of sports with the ability to automatically personalize and socially optimize elements delivered to end users. And it makes use of AI to identify rights or regulatory compliance issues that might occur with elements in a content stream, thereby alleviating concerns over rights enforcement that have been an impediment to enhancing sports and other programming.

Nomad Media

Nomad’s comprehensive AI-assisted media asset management (MAM) platform is designed to interface with all the types of solutions that go into managing, storing, searching, streaming and analyzing on-demand and live video feeds securely and cost effectively. The company’s use of generative AI supports creation of new metadata, intelligent search and discovery, multi-lingual captioning, and on-the-fly asset refinements, from replacing logos to modifying backgrounds and other aspects of the content itself. The platform’s ability to instantly identify and track content elements is also used in surveillance operations related to public safety and law enforcement, often in cases where Red5’s TrueTime MultiView for Surveillance tools are in play.

Game Engines

The leading game engine suppliers Unreal Engine and Unity are Red5 partners who are making liberal use of AI in their processing mechanisms in and beyond the game development arena. In the gaming world, AI game controllers comprise the underlying fabric developers use to define how everything works, from how virtual characters behave, to how scenes change, to how story lines unfold in reaction to game players’ actions. Game engines are also supporting in-camera visual effects (ICVFX) in virtual M&E productions, image modifications with VFX and other manipulations in postproduction and the rendering of virtual components and entire immersive environments in multiple categories of XR applications.

Developers must be able to exploit the power of these innovations in workflows running on real-time streaming infrastructure without going through laborious integration processes. Red5 has made this possible for its customers, whether they’re running their services on infrastructures created on the Red5 Cloud platform or through their own applications of Red5 Pro tools. In a unique market-leading response to how deeply interwoven game engine functionalities can be in any use case, Red5 has created Unity and Unreal early beta release SDKs to enable customers to employ these partners’ platforms in solution subsets specific to targeted market segments. If you’re interested in being invited to the beta contact us.

PubNub

PubNub is a widely used provider of support for real-time, globally scalable interactive applications tapping the capabilities of generative AI across multiple industry sectors, including live sports, esports and other M&E services, online betting and casino gambling, video games, ecommerce, and other sectors such as health and transportation. Its SDKs and APIs bring its support for interactive engagements to end users at sub-100ms latencies, making it an ideal partner in the real-time streaming space occupied by Red5.

As PubNub puts it in a recent blog, “WebRTC-based low-latency streaming is becoming critical for interactive sports experiences, such as live betting and fan engagement. A pub-sub messaging layer, such as PubNub, can synchronize live data overlays, chat, and real-time fan interactions to enhance the viewing experience without impacting video performance.”  With generative AI PubNub enhances chat applications with real-time multilingual translations and the ability to answer user-generated questions while providing an app development framework customers can employ however they like to increase user engagement, deliver insights into usage behavior, push notifications, support IoT device control and much else.

The Famous Group

Content production specifically targeted to live in-venue user experiences, often with XR components, is now part of the use-case scenario that must be accommodated by real-time streaming platforms. A big step in this direction is Red5’s partnership with The Famous Group, a leading provider of in-venue and remote user experiences that heavily rely on Unreal Engine and AI-assisted image manipulation to seamlessly combine virtual with real elements.

Real-time streaming support from Red5 ensures that the fan engagements are presented on venue video boards in sync with what’s happening on the playing field. By scanning a QR code provided through The Famous Group’s Vixi Suite Platform, fans can open a path for streaming their selfies live or uploading their photos to the video boards, where their input is embellished with virtual team-related or other elements that anyone streaming content from a video board can see.

The AI-enabled capabilities include turning fan photos and the virtual embellishments into fantasy video animations, which can be used to extend the interactive engagement to out-of-venue participants who upload their photos to the video boards. Some use cases have gone so far as to personalize live-streamed content with placement of the viewer’s image in ads or other segments of what they’re watching.

These capabilities create interactive fan engagements that have been widely deployed in more than 175 venues across 15 countries. Productions have involved NFL, MLB, NHL, NBA, MLS, the Premier League, LaLiga, and many other league competitions as well as motor sports, golf, gymnastics and combat sports.

Surveillance

AI-assisted analysis facilitated by Red5 partners is a major factor in the transformation of how surveillance camera inputs from ground-, air- and sea-based observation points are used to track unfolding events across any range of impacted territory. Utilizing AI-aided analytics in combination with Red5 TrueTime MultiView for Surveillance, responders are streaming aggregated video output from devices numbering into the thousands to track criminal activity, fires, traffic accidents, extreme weather conditions, illicit border crossings, battlefield maneuvers and much else.

Virtually any AI application designed to work with video surveillance can be used without any special integrations to parse details in the compilation of camera feeds delivered from the field to command centers in real time by the Red5 platform. But there are some added benefits that can be derived from partners who, like Nomad Media as noted above, are fully integrated with Red5.

Accenture Federal Services

Red5 partner AFS is providing AI-assisted intelligence through a cross-platform web application that enables multiple functions essential to coordinating surveillance operations across dispersed command posts. Through integrated chat services, event/report sharing, and remote sensor control, operators can collaborate in real time to quickly understand what is happening around them in their areas of operation.

Red5 customers in defense, federal law enforcement and government intelligence are employing the AFS platform. For example, U.S. D.O.D. branches are using the real-time multi-intelligence solution to deliver actionable insights derived from sensors and radar as well as video feeds from drones, Humvees, underwater vehicles, body cameras and even canine patrols to remotely dispersed command centers.

Oracle Cloud Infrastructure

The Red5 Cloud SaaS runs on the globally deployed OCI, which allows customers to leverage OCI’s support for AI applications with any use case, as discussed below. In the case of surveillance, Red5 Cloud customers benefit from OCI’s support for integrating live AI-assisted speech-to-text transcriptions and computer vision use cases on its platform. Red5 surveillance customers operating in the OCI domain can identify and rip out relevant transcriptions from the camera feeds at any time to aid in situational analysis.

This support gains added significance from the fact that Oracle has made it possible to extend instantiations of XDN infrastructure beyond the reach of OCI’s fixed datacenter resources, which span 50 geographic regions on six continents. This is accomplished through use of portable Oracle Roving Edge Infrastructure devices, which enables Red5-supported surveillance use cases to be deployed where speed-of-light latencies imposed with use of conventional OCI resources are too high.

Much faster, more knowledgeable responses to emergency and battlefield scenarios can be executed when it’s possible to aggregate and analyze surveillance camera feeds from aerial and maritime drones, body cams, mobile phones and remote fixed locations in relatively close proximity to the action. Points of camera feed aggregations can be positioned in conjunction with ancillary command centers to ensure that no matter how far flung their surveillance assets might be, responders will still be able to maintain real-time vigilance over what’s happening.

DirectAI

DirectAI is offering a solution built on VLMs that is very useful in surveillance as well as applications in M&E and other operations. The cloud-based software allows content producers with no AI skills to use plain language in the creation of object recognition models that can be used to track multiple objects and types of objects simultaneously. In surveillance the platform facilitates identification of faces, license numbers or other elements relevant to a given situation.

Cloud Computing Platforms

Red5 is pre-integrated with multiple cloud platform providers in support of the unique seamless cross-cloud scalability that’s available to customers who utilize Red5 Pro SDKs, toolsets and APIs to implement XDN infrastructures. OCI is among the options available to Red5 Pro users and, as mentioned, also serves as the supplier of cloud compute resources used with the Red5 Cloud service.

Here our focus is on two platforms, OCI and AWS, that provide Red5 customers exceptional support for bringing AI-enabled solutions into play with XDN operations. While these cloud systems have taken divergent approaches to supporting AI, all their AI initiatives revolve around the availability of Nvidia GPU appliances to complement basic CPU support. As a result, by virtue of Red5’s close relationships with both partners, Red5 customers have ready access to the resources they need to bring AI into their operations in whatever ways suit their needs.

AWS

AWS offers customers, including those running XDN infrastructure on its cloud resources, a wide range of tools and services aimed at facilitating use of generative AI, AI foundation models (FMs) and machine-learning processes. AWS, the pioneer in the use of Nvidia GPU acceleration in cloud computing, is now using the latest generation of AI-optimized Nvidia processors to support purpose-built AI services and applications.

The range of AI options is broad and deep, including everything from text-to-speech, speech-to-text and multilingual translation capabilities to support for personalizing content and interactions, obtaining audience insights, building specialized agents, and much more. In addition, Amazon’s SageMaker AI tools enable creation, training and deployment of custom FMs at scale.

OCI

Initially, rather than focusing on services and applications that help customers put AI to work for specific use cases, OCI emphasized making AI-processing resources as supplied by Nvidia available to customers in support of more efficient and flexible use of OCI’s cloud resources in the various stages of AI application development whatever the use case might be. Toward this end Oracle enables customers to earmark use of OCI Superclusters scaling to over 130,000 GPUs for handling demanding AI workloads related to things like generative AI training, development of agentic AI (the functions that allow AI apps to operate as independent decision-making agents) and AI inferences (predictive capabilities of AI apps), and execution of massive projects in scientific research, recommendation operations and other areas.

In March 2025 OCI expanded its AI support with integrations involving Nvidia AI Enterprise software that have made over 160 AI tools and Nvidia microservices available on the OCI customer console. In addition, both companies are collaborating on enabling no-code deployment of their respective AI development blueprints.

Looking Ahead

Of course, the world is just in the early stages of AI adoption. As explained in the foregoing discussion, Red5 has taken a leadership role in ensuring the benefits accruing with ever greater use of AI will be readily available to the growing ecosystem of video-centric operations across all types of enterprises and institutions that depend on real-time multidirectional streaming infrastructure.

As Red5 continues to build partnerships with suppliers who can bring compelling AI-assisted solutions into the real-time streaming domain, we’re also looking at ways we can put AI directly to use in XDN Architecture. While we want to keep our plans under wraps for competitive reasons, we can share a couple of the cost-saving and performance-enhancing uses of AI we’re contemplating.

Both relate to the unique scaling capabilities of XDN Architecture. XDN infrastructure runs on commodity servers in locations that can be dynamically configured by the XDN Stream Manager to seamlessly operate in cloud clusters as Origin, Relay and Edge Nodes. One or more Origin Nodes in a cluster serve to ingest and stream encoded content out to Relay Nodes, each of which serves an array of Edge Nodes that deliver live unicast streams to end points in their assigned service areas.

Origin Node placements can be optimized to accommodate ingestion of massive volumes of streams at minimum latency in interactive scenarios serving all end users through co-locations with XDN Edge Nodes. By leveraging both containerized and virtual machine-based iterations of datacenter virtualization, the XDN platform enables the flexibility and speed of resource utilization that is essential to unlimited scalability and fail-safe redundancy.

In other words, XDN infrastructure can be scaled to accommodate any number of users both as receivers and senders in live streaming sessions. This opens the possibility of employing AI predictive capabilities in our Stream Manager to more precisely tune scaling of resource usage for any given streaming instance by predicting what the requirements will be based on analysis of past usage patterns, demand trends and other indicators. By eliminating some of the guess work associated with meeting scaling requirements, this could cut costs resulting from over-allocating resources as well as the costs that come with delays in resource allocation resulting from underestimating what’s needed.

Another cost-saving concept on our roadmap involves our approach to creating adaptive bitrate (ABR) profile ladders. One of the major distinguishing advantages that come with use of XDN Architecture is our approach to supporting real-time delivery of ABR profiles as a hedge against impediments to stream flows caused by fluctuations in bandwidth.

Customers can automatically implement transcoding of content into multiple bitrate profiles through our software-based Caudron transcoder running on CPU commodity resources prior to ingestion into XDN Origin Nodes. This enables simultaneous streaming of the content in all the assigned bitrates to all Edge Nodes, where awareness of each user’s bandwidth availability ensures the content will be delivered moment by moment at whatever bitrate ensures the best viewing experience for each user.

This approach to ABR streaming over WebRTC contrasts with the common approach taken by the handful of other WebRTC platforms that support ABR, which execute transcoding at their edge points. That approach incurs much higher costs in the utilization of cloud processing resources compared to the one-time transcoding approach taken by XDN Architecture. Moreover, the edge transcoding approach complicates infrastructure scaling and increases the risks that unaddressed malfunctions in transcoding instances will undermine performance.

By bringing AI into XDN ABR processing, it may be possible to employ predictive intelligence to avoid the need to preset ABR profiles. AI could tell us which profiles are actually needed for a specific streaming session, thereby contributing to our market-leading reduction in the use of resources in the transcoding process.

Stay tuned for further information on these possibilities.

Flexibility in Executing AI Applications

Meanwhile, the final point to be made here about current uses of AI technology with XDN Architecture has to do with the flexibility customers have to tailor their use of XDN infrastructure precisely to their needs, whether they’re relying on Red5 Cloud or Red5 Pro instantiations of their XDNs. This is the only real-time streaming platform that allows customers to make use of AI-enabled solutions as they see fit in both a turnkey ready-to-deploy cloud service environment and a DevOps environment where customers have free rein to custom build their own WebRTC infrastructures.

The basic difference between working in the Red5 Cloud and Red5 Pro environments has to do with how the XDN infrastructure is implemented and managed over time. Customers can take a DevOps approach to using XDN Architecture by utilizing the extensive portfolio of Red5 Pro SDKs and APIs over any combination of the multiple cloud computing platforms that have been pre-integrated with XDN Architecture or that can be easily integrated via the widely used Terraform multi-cloud toolset.

In the case of Red5 Cloud, which, as noted runs on the OCI platform, the service configures and maintains real-time interactive infrastructures dedicated exclusively to customers in perfect alignment with their use case requirements. This is a major departure from the shared usage platforms operated by other suppliers of WebRTC cloud services, where pre-formatted use-case applications are offered on a take-it-or-leave-it basis.

In either scenario, XDN Architecture accords customers unparalleled freedom to respond with speed and precision to new opportunities. The architecture’s reliance on open-source technology and APIs together with the availability of application-specific TrueTime toolsets provide all Red5 customers the flexibility they need to tailor AI applications as they see fit.

---

The fulfillment of the potential stemming from the use of AI with real-time multidirectional streaming is now moving full steam ahead. Witnessing what comes next based on where things already stand with the integration of AI-equipped partners on the Red5 platform will certainly be a riveting experience. And there will be much more to come as we expand our partnerships and devise new ways to put AI to use with XDN Architecture.

Good ideas are welcome. New deployment engagements even more so. To keep in touch contact info@red5.net or schedule a call