Subject: [rtcweb] Time-synchronized data channels

From: Randell Jesup <[email-masked]>
Date: Thu, 17 Nov 2011 18:25:54 -0500

A proposal for time-linked data streams in WebRTC.  Justin challenged me 
on this, and Matthew indicated some of the utility provided by the 
un-named plugin, so after a little back-and-forth I propose this:

Requirements: deliver datagrams in a fashion synchronized(*) to a media 
stream or set of synchronized media streams.  Datagrams may be reliable 
or unreliable depending on the underlying data channel; note that 
reliable datagrams in particular may be delivered after the 
synchronization point, but any datagram may be delivered 'late', 
depending on congestion control and other factors.  The datagrams SHOULD 
NOT be delivered before the media stream reaches the equivalent 
synchronization point, but note that device and playout driver overhead 
may not be included in the calculations.

* Note that this can never be truly synchronous, since the JS code runs 
asynchronously to the media streams, though 'web workers' can help make 
lower-latency possible.

Instead of building an infrastructure inside of WebRTC to support this 
directly, I suggest we expose the needed functionality to implement this 
and allow the app and/or library authors to implement it.

Required additional functionality:

1) Ability to request the "current" time of a media track being sent or 
played back (will not be current when returned!)
2) Ability to request a callback or event when a particular time of a 
media track is played out, or on the sending side when it's "current" 
(which lets you do deltas relative to an event)

I believe those are generally-useful functionalities with other uses, so 
there's little added work.

Pseudo-code:

To send time-linked data:
          datagram.time = mediatrack.GetCurrentTime();
          datastream.Send(datagram);
While it would force the layout, you could combine these into one 
function (SendTimedData or whatever).  That might provide lower latency 
for the data.  You could also send an event linked to a time in the 
future, especially using the first API (both could co-exist easily).

NOTE: timed_buffer below would be an application/library object, not 
part of WebRTC.  (Or if people prefer we could easily include it; the 
basic design wouldn't be very complex; mostly housekeeping.)  Or just 
include source in an example.  Building it in *might* give some added 
latency improvements; we should consider if that's true and if so if 
it's important.

On reception:
          timed_buffer.Insert(datagram,datagram.time);
(don't make assumptions on internal format of datagram, unless we use 
SendTimedData/whatever)

In timed_buffer.Insert:
          if (time < this.mediatrack.current_time)
               this.onDataAvailable(datagram, time);
          else {
               var position = this.insertAt(datagram, time);
               if (position == 0)
                    this.mediatrack.NotifyAt(this.head.time);
          }

In timed_buffer.onNotifyAt()
          var old_head = Remove(this.head);
          this.mediatrack.NotifyAt(this.head.time);
          this.onDataAvailable(old_head.datagram, old_head.time);

We assume that the application has indicated in some fashion to both 
sides which media track and which data stream are associated with each 
other.

There may be some housekeeping to make sure it doesn't grow unbounded, 
handle associations, etc, but it's all straightforward.

-- 
Randell Jesup
[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 18 Nov 2011 08:08:01 +0800

I like the idea of making this a JS overlay rather than a core protocol 
feature.

It's of course trivial to do a bufferbloat attack:

while (1) {
    data.timestamp = <very large number>
    send(data)
}

and having the result be a JS failure rather than a browser failure (or 
having to write complicated rules in the browser spec for how to handle 
such) seems good to me.

On 11/18/2011 07:25 AM, Randell Jesup wrote:
> A proposal for time-linked data streams in WebRTC.  Justin challenged 
> me on this, and Matthew indicated some of the utility provided by the 
> Requirements: deliver datagrams in a fashion synchronized(*) to a 
> media stream or set of synchronized media streams.  Datagrams may be 
> reliable or unreliable depending on the underlying data channel; note 
> that reliable datagrams in particular may be delivered after the 
> depending on congestion control and other factors.  The datagrams 
> SHOULD NOT be delivered before the media stream reaches the equivalent 
> synchronization point, but note that device and playout driver 
> overhead may not be included in the calculations.
>
> * Note that this can never be truly synchronous, since the JS code 
> runs asynchronously to the media streams, though 'web workers' can 
> help make lower-latency possible.
>
>
> directly, I suggest we expose the needed functionality to implement 
> this and allow the app and/or library authors to implement it.
>
> 1) Ability to request the "current" time of a media track being sent 
> or played back (will not be current when returned!)
> I believe those are generally-useful functionalities with other uses, 
> so there's little added work.
>
>
> Pseudo-code:
>
> function (SendTimedData or whatever).  That might provide lower 
> latency for the data.  You could also send an event linked to a time 
> in the future, especially using the first API (both could co-exist 
> easily).
>
>