Millicast 2.0 — Interactive Video & Audio API
With our latest Millicast 2.0 release we have deployed more advanced features that enable our customers to add multiple interactive real-time video tracks to their applications, complementing the existing interactive audio functionality.
The new projection feature allows the viewer to choose any of the audio and video sources they want to receive dynamically within a stream. This feature enables Millicast to be used as a massively distributed SFU so you can build your own meeting-like application like Zoom, Meet or Teams.
This article describes the interactive functionality in detail, and the docs are also available here: https://docs.millicast.com/docs/interactivity
In this new release we have added new events that can be received by the viewer application to detect:
- when a new source has been published within the stream
- when a source has been unpublished within the stream
- the simulcast/svc layer information for the published video tracks of each source
- VAD multiplexing, which was already available.
The docs are available here: https://docs.millicast.com/docs/viewer-events
The events are delivered over the websocket connection established by the viewer application with the Millicast servers.
Source and Layer Selection
So now that you have the ability to publish multiple sources in a stream and detect those events within the viewer, the next step is to specify the source you want to receive.
For each audio and video track you can select and switch sources from the Millicast server and then forward the selected media to the viewer.
This is accomplished using the “project” command. You can specify the “sourceId” that you want to project, use the “trackId” of the source (received on the “active” event”) for each track, and the “mid” of the transceiver object for the media you want to receive.
Here is an example of an “activate” event:
You can send a “project” command to start receiving either of the tracks from the source in one or more of the transceivers you have already setup on the peerconnection:
If you want to stop receiving media from one of the sources, you can either send a “project” command to replace the media with a different source, or use the “unproject” command to stop receiving media for that transceiver:
In addition to selecting the origin source for the media, it is also possible to choose the specific Simulcast and/or SVC layer for each video track delivered by the Millicast server. You can do that either by specifying the “layer” attribute on the “project” command or using the “select” command for the main video track:
The layer information available for each video source is provided periodically by the “layers” event as shown above. If you want to switch back to the automatic layer selection, you would just need to send a “project” or “select” command with empty layer information.
Note that Millicast will not limit the number of tracks that a viewer is able to receive, but will limit the maximum bitrate per viewer to a maximum of 4mbps across all media tracks. You should configure the Simulcast/SVC bitrate of all the sources carefully within your Applications so they can receive the desired amount of video tracks in the viewer session.
The docs are available here: https://docs.millicast.com/docs/source-and-layer-selection
The last piece of the puzzle is to be able to receive more than one audio or video track within the same viewer session.
With the audio multiplexing feature in our previous release, you could create multiple audio tracks when the viewing session was started. However, you couldn’t have multiple video tracks and multiplexed audio from the server. So there was an important feature missing: the ability to add new tracks on demand.
This is now possible with the “addRemoteTrack” command, which will perform a local renegotiation and create the “track” event with the added track and transceiver:
Once the transceiver has been assigned a mid value, you can use it with the “project” command to indicate the media source you want to receive from the Millicast server.
The docs are available here: https://docs.millicast.com/docs/dynamic-viewer-tracks
Start building the future today
To start using these features, sign up for a free developer account and access the Millicast Dashboard to set up API access. Then view the documentation to get started at https://docs.millicast.com/docs/interactivity
You can send an email to email@example.com to request access to the Millicast Interactivity features.
We will enable these capabilities on your Millicast account so you can start building your own real-time interactive streaming applications.