Subject: [rtcweb] Synchronization

From: Martin Thomson <[email-masked]>
Date: Sat, 3 Nov 2012 08:11:01 +0000

The discussion in the Web RTC (or was it media capture?) meeting this
week reached the following conclusion:

For a given MediaStream instance, the MediaStreamTracks that are part
of that MediaStream are synchronized for playback.  This is the only
synchronization guarantee.  Inter-MediaStream synchronization is
entirely at browser discretion.  No guarantees are required between
MediaStream instances, even for MediaStreams that are composed of
exactly the same set of tracks.

>From this, I also note that the "playback" guarantees necessarily
apply to the application of RTP timestamps, such that it would be
possible to ensure that the tracks could be successfully synchronized
on a remote peer when they are extended over the network.   However,
this does not imply that the browser place the packets on the network
at the same time.

Though synchronizing packet construction would be a desirable
property, this neither required nor sensible.  Nor does it especially
help given that different tracks might traverse the network in
dramatically different ways (due to different network paths, different
QoS treatment or other factors).  A receiver is therefore required to
pay attention to the timestamps and account for any skew that might
have occurred.  In practice, delaying any synchronization adjustment
(i.e., buffering) until playback allows for less effort in the case
where the MediaStream is immediately disassembled.

A possible recommendation would be to have packets sent as soon as is
practical with no particular regard to synchronization of
transmission.  Let the playback device provide any buffering necessary
for synchronized playback.

We can construct an example with tracks t1, t2 and media streams m1
(comprised of t1 and t2, denoted m1[t1] and m2[t2]) and m2 (comprised
of m2[t1] and m2[t2]).  In this example, playback of m1 causes m1[t1]
and m1[t2] to be synchronized.  It may naturally occur that when
playing both m1 and m2, the tracks m1[t1] and m2[t1] also playback
synchronized, but the browser is not required to ensure that this
happens.

This is made clearer when a third track is added to m2.  Replacing
m2[t1] with m2[t3] requires that the playback of m2 causes m2[t2] and
m2[t3] to be synchronized.  t3 could vastly different timing
characteristics to t1 and t2, maybe it is a copy of t1 that has been
sent through the network and has been returned with a significant
delay.  This results in a need for playback of m2 to be delayed until
t2 and t3 are synchronized.  The browser is required to buffer m2[t2]
until its timing corresponds with m2[t3]; it is not required to
synchronize m1[t2] and m2[t2].

Though t2 is part of the same MediaStream as t1 and t3, there is no
transitive synchronization such that t1 and t3 become synchronized.
It is the browsers choice entirely.  Because the browser is not
obliged to adhere to any particular latency constraints, it could
choose to delay playback of m1 to align with m2 as a matter of
optimization.  This causes t1 and t3 to become synchronized if that
suits the implementation constraints of the browser.  It is
conceivable that synchronizing playback of the two MediaStreams could
- for example - allow the browser to share decoder resources for t2.
This would result in t1 and t3 being synchronized, but that remains
entirely an implementation choice.

This would, on face value, seem to be fairly dire consequences for
interoperability.  Consider CNAME.  A WebRTC-aware implementation is
able to follow these rules and produce behavior consistent with the
WebRTC source.  The actual CNAMEs assigned to tracks as they traverse
the network are not important as it pertains to WebRTC-aware
receivers.

An RTP implementation that follows specification will use CNAME to
determine which RTP streams (aka WebRTC tracks, aka MediaStreamTrack
instances) require synchronization.  But even in the simple example
given above, it is not clear how each RTP stream is assigned CNAME.
The only two choices that seem viable are to have either just one
CNAME for all RTP streams or one CNAME for each RTP stream.  Had there
been no "sharing" of tracks, each MediaStream could be given a
different CNAME and everything would work out.  As soon as sharing
occurs, there is a choice:
   a) replicate the RTP stream so that the one CNAME per MediaStream
guarantee is met
   b) consider breaking synchronization for peers that are not aware
of WebRTC synchronization contexts (aka MediaStream)
The former is safe, but wasteful, the latter has potential
interoperability and user experience ramifications.

Who makes this choice is another choice we need to make:
   a) we could decide and require browsers to implement one or other choice
   b) we could allow browsers to decide, with or without a recommendation
   c) we could require browser to allow applications to decide, with
or without a default, where the default may be required, recommended
or up to browsers to choose

Furthermore, it is unclear how this synchronization extends to
multiple inbound RTP streams on the same session.  The implied design
is one where MediaStreams are created as a result of receiving offers
or answers.  Offers and answers that contain the necessary msid
markings can be used to pre-allocate MediaStreams and MediaStreamTrack
instances based on the provided description.

There are several potential failure modes in this design: the
signaling may not provide the necessary markings (for instance, it
might not be possible to know what SSRC will be chosen for a given
track at the time that signaling is created), additional streams may
arrive, streams could arrive prior to the necessary signaling, or
streams could undergo an SSRC reassignment.  In all these cases,
streams are arriving that do not have known synchronization
characteristics.

In RTP, the expectation is that a sender report will arrive with CNAME
information that enables the receiver to learn the synchronization
properties of streams.  Prior to this happening (which could be a very
long time), the receiver is forced to play streams in a synchronized
fashion.

There are clearly several designs currently in mind for how this
synchronization operates.  It would seem that none of these are
perfectly adequate.

On the other hand, at the point where unsynchronized tracks are
noticeable, it is also likely that the delay imposed by
synchronization would be noticeable to the point of being worse.  In
the end, it may be that all this complexity doesn't turn out to be
useful...

--Martin

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 04 Nov 2012 13:53:05 +0100

Thanks for the thorough analysis!

I have been thinking about synchronization in a reasonably ad-hoc 
fashion; I want to ensure that we can support the following scenarios:

- Audio and video for one source arrives in one MediaStream, over one 
PeerConnection. They are played out synchronized.

- Multiple MediaStreams arrive, with very varying jitters, over the same 
PeerConnection. It should be possible to play out those streams that do 
not have much jitter without having their timebase disrupted by streams 
that need heavy dynamic adaptation because of jitter (varying-speed 
playback can sound quite odd at times).

If there is no stream duplication, a sender can choose to do this in a 
backwards-compatible fashion by assigning one CNAME for each MediaStream.

It will also achieve backwards compatibility by using one CNAME across 
all MediaStreams; this will not permit old gateways to break 
synchronization in the way that an RTCWEB browser is permitted to do (if 
we decide that such breaking is permissible).

If we only want to send each MediaStreamTrack once, RTP doesn't allow 
more than one CNAME, so the sender that wants to be backwards compatible 
is faced with the choice of either sending it more than once (achieving 
compatibility by paying a bandwidth tax) or using the same CNAME for 
multiple MediaStreamTracks (achieving compatibility by imposing a 
synchronization requirement on the traditional RTP recipient).

This is made somewhat more complex by the fact that making this decision 
when tracks get added to MediaStreams might be less than trivial; 
changing the CNAME of an RTP stream seems to be something that isn't 
exactly obvious how to do in RTP and SDP; it's obvious how to send the 
new name, but it's not obvious what the behaviour is intended to be when 
a name changes, or whether this imposes timing constraints between 
signalling and SR sending.

Are there cases in current SDP/RTP usage when one changes a CNAME?

On 11/03/2012 09:11 AM, Martin Thomson wrote:
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Stefan Hakansson LK <[email-masked]>
Date: Mon, 5 Nov 2012 09:26:02 +0100

Martin,

thanks for this write up.

I totally agree to that it makes most sense to send packets as soon as 
possible, and let the playback device handle synchronization.

And perhaps we should change the notion of a MediaStream from being 
"play back synchronized" to something like "these tracks belong 
together", but leave it to the playback device (browser) to decide if it 
gives better user experience to play them synchronized, or (which is 
likely if one or more of the tracks have much larger latency than other 
track(s)) unsynchronized.

Regarding CNAMEs, to me it seems natural that all MediaStreamTracks (RTP 
streams) originating from the same browser should have the same CNAME. 
The audio and video is presumable captured in the same location at the 
same time. This would enable, if the network latency is short, the 
playback device to synch them even for situations when not all tracks go 
in the same MediaStream (but capture the same scene).

But MediaStreamTracks from other sources should have another CNAME.

So, as example: assume you (for some reason) create a MediaStream 
containing MediaStreamTracks locally generated (by camera and mike) as 
well as MediaStreamTracks received from a PeerConnection. If this 
MediaStream is in turn sent over a PeerConnection, it seems most natural 
to me if the locally generated tracks have the same CNAME, and the 
forwared tracks have another (presumably the one given by the browser 
they origin from).

Stefan

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 05 Nov 2012 12:12:36 +0100

On 11/05/2012 09:26 AM, Stefan Hakansson LK wrote:
> together", but leave it to the playback device (browser) to decide if 
> it gives better user experience to play them synchronized, or (which 
> is likely if one or more of the tracks have much larger latency than 
> other track(s)) unsynchronized.
If I were a stickler for precision, I think I'd claim that 
"synchronized" only means that "these streams come in with consistent 
timestamps", there's nobody going to whap you over the head with a wet 
noodle if you play them out unsynchronized.

>
> Regarding CNAMEs, to me it seems natural that all MediaStreamTracks 
> (RTP streams) originating from the same browser should have the same 
> CNAME. The audio and video is presumable captured in the same location 
> at the same time. This would enable, if the network latency is short, 
> the playback device to synch them even for situations when not all 
> tracks go in the same MediaStream (but capture the same scene).
>
> MediaStream is in turn sent over a PeerConnection, it seems most 
> natural to me if the locally generated tracks have the same CNAME, and 
> the forwared tracks have another (presumably the one given by the 
> browser they origin from).
An example of a real use case where this track arrangement might be 
natural is one where the remotely generated streams are of a politician 
giving a speech, and a locally added video channel gives interpretation 
in sign language.... those streams should not be allowed to drift in 
relation to each other, so it makes sense to claim that they're 
synchronized (and for the relaying node to make sure clock drift doesn't 
happen, by whatever means he can).

But they don't have the same origin.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 5 Nov 2012 12:26:21 +0000

On 5 November 2012 11:12, Harald Alvestrand <[email-masked]> wrote:
>> And perhaps we should change the notion of a MediaStream from being "play
>> back synchronized" to something like "these tracks belong together", but
>> leave it to the playback device (browser) to decide if it gives better user
>> experience to play them synchronized, or (which is likely if one or more of
>> the tracks have much larger latency than other track(s)) unsynchronized.
>
> If I were a stickler for precision, I think I'd claim that "synchronized"
> only means that "these streams come in with consistent timestamps", there's
> nobody going to whap you over the head with a wet noodle if you play them
> out unsynchronized.

Yes, user experience will be the most important consideration.
Whatever that means for synchronization.  Obviously, synchronization
is desirable, but it needs to be balanced against a lot of other
concerns.

Bernard and I discussed this, as well as other scenarios where you
have synthetic, blob-sourced and remote streams.  In those cases, it
might make sense to progress the clock consistently, but it is
possible that true synchronization is nonsensical.

> But they don't have the same origin.

I'm not sure how that would be relevant.  If you are talking about
marking tracks for their source, that is true.  Streams consisting of
tracks from mixed sources would not come directly from
RTCPeerConnection, they would be assembled locally.  An authenticated
RTCPeerConnection would not be able to provide a mix of authenticated
an unauthenticated streams.   That is, unless we add considerably more
detail to identity assertions.

---

From: Stefan Hakansson LK <[email-masked]>
Date: Mon, 5 Nov 2012 14:16:51 +0100

On 11/05/2012 01:26 PM, Martin Thomson wrote:
> RTCPeerConnection, they would be assembled locally.

At least if used without identity assertion, you can very well forward 
the locally assembled MediaStream over a RTCPeerConnection. Then I think 
that the tracks in that assembler MediaStream can have different origin, 
and they should use different CNAME in the transmission.

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 05 Nov 2012 15:02:53 +0100

> detail to identity assertions.
Or if the other end of the RTCPeerConnection is a mixer - it would then 
vouch for the identity of the streams it's providing as saying "these 
were definitely provided by me", but that wouldn't be proof of their 
original source.

Whether anyone wants to do some action based on what the origin for the 
stream is a different question.

---

From: Randell Jesup <[email-masked]>
Date: Wed, 07 Nov 2012 02:04:51 -0500

> is desirable, but it needs to be balanced against a lot of other

Agreed.

> possible that true synchronization is nonsensical.

Yes, those are some of the obvious cases of synchronization between 
different MediaStreams is nonsensical - if a stream sourced from a 
<video> object is sent along with live video from the camera, you 
*really* don't want to stall the live video because the <video> element 
got starved or hiccupped.

-- 
Randell Jesup
[email-masked]