Millicast 2.0 — Interactive Video & Audio API

In our previous update we introduced the concept of multisource streams and audio multiplexing within our Millicast API and Javascript SDK that enabled the features needed to build your own interactive Clubhouse application with the added benefit of video for the host/presenter.

With our latest Millicast 2.0 release we have deployed more advanced features that enable our customers to add multiple interactive real-time video tracks to their applications, complementing the existing interactive audio functionality.

The new projection feature allows the viewer to choose any of the audio and video sources they want to receive dynamically within a stream. This feature enables Millicast to be used as a massively distributed SFU so you can build your own meeting-like application like Zoom, Meet or Teams.

This article describes the interactive functionality in detail, and the docs are also available here: https://docs.millicast.com/docs/interactivity

Viewer events

In this new release we have added new events that can be received by the viewer application to detect:

  • when a new source has been published within the stream
  • when a source has been unpublished within the stream
  • the simulcast/svc layer information for the published video tracks of each source
  • VAD multiplexing, which was already available.

The docs are available here: https://docs.millicast.com/docs/viewer-events

The events are delivered over the websocket connection established by the viewer application with the Millicast servers.

If you use the Millicast Javascript SDK, you can subscribe to the events when using the “view” command, specifying which events you want to receive:

Source and Layer Selection

So now that you have the ability to publish multiple sources in a stream and detect those events within the viewer, the next step is to specify the source you want to receive.

For each audio and video track you can select and switch sources from the Millicast server and then forward the selected media to the viewer.

This is accomplished using the “project” command. You can specify the “sourceId” that you want to project, use the “trackId” of the source (received on the “active” event”) for each track, and the “mid” of the transceiver object for the media you want to receive.

Here is an example of an “activate” event:

You can send a “project” command to start receiving either of the tracks from the source in one or more of the transceivers you have already setup on the peerconnection:

If you want to stop receiving media from one of the sources, you can either send a “project” command to replace the media with a different source, or use the “unproject” command to stop receiving media for that transceiver:

By default, the Millicast server will choose which is the best Simulcast or SVC layer to forward to the viewer based on the bandwidth estimation calculated by the server.

Real-time AV1 SVC encoding & decoding playback with Millicast

In addition to selecting the origin source for the media, it is also possible to choose the specific Simulcast and/or SVC layer for each video track delivered by the Millicast server. You can do that either by specifying the “layer” attribute on the “project” command or using the “select” command for the main video track:

The layer information available for each video source is provided periodically by the “layers” event as shown above. If you want to switch back to the automatic layer selection, you would just need to send a “project” or “select” command with empty layer information.

Note that Millicast will not limit the number of tracks that a viewer is able to receive, but will limit the maximum bitrate per viewer to a maximum of 4mbps across all media tracks. You should configure the Simulcast/SVC bitrate of all the sources carefully within your Applications so they can receive the desired amount of video tracks in the viewer session.

The docs are available here: https://docs.millicast.com/docs/source-and-layer-selection

Dynamic tracks

The last piece of the puzzle is to be able to receive more than one audio or video track within the same viewer session.

Video & Audio Multiplexing

With the audio multiplexing feature in our previous release, you could create multiple audio tracks when the viewing session was started. However, you couldn’t have multiple video tracks and multiplexed audio from the server. So there was an important feature missing: the ability to add new tracks on demand.

This is now possible with the “addRemoteTrack” command, which will perform a local renegotiation and create the “track” event with the added track and transceiver:

Once the transceiver has been assigned a mid value, you can use it with the “project” command to indicate the media source you want to receive from the Millicast server.

The docs are available here: https://docs.millicast.com/docs/dynamic-viewer-tracks

Start building the future today

To start using these features, sign up for a free developer account and access the Millicast Dashboard to set up API access. Then view the documentation to get started at https://docs.millicast.com/docs/interactivity

You can send an email to beta@millicast.com to request access to the Millicast Interactivity features.

We will enable these capabilities on your Millicast account so you can start building your own real-time interactive streaming applications.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Millicast

Millicast

The Fastest Streaming on Earth. Realtime WebRTC CDN built for large-scale video broadcasting on any device with sub-500ms latency.