Subject: [rtcweb] JSEP - Removing and modifying streams

From: Cullen Jennings <[email-masked]>
Date: Fri, 27 Jan 2012 08:22:32 -0700

Imagine that two browsers (A and B)  have set up audio and video streams and are happily chatting away. At this point side A unplugs the camera. What happens next? how does this get notified ? how does the PeerConnection end up telling the JS that a new offer / answer exchange needs to be done to indicate this video stream is gone? It seems like a callback indicating something like this needed would be an good way to solve this. 

I think this come up in another context where the browser wants to change the the video resolution of frame rate due to congestion control. We have not sorted out how the congestion control story works but some of the nicer systems end up significantly renovating video where there is a substantial change of bandwidth. You can imagine that you are at home, have 100% of a DSL link for incoming HD video, then someone starts iTunes and it downloading 3 TV shows and suddenly you go from 100% of DSL to 25% of DSL.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 27 Jan 2012 07:57:37 -0800

> Imagine that two browsers (A and B)  have set up audio and video streams and are happily chatting away. At this point side A unplugs the camera. What happens next? how does this get notified ? how…

Yes; we need something associated with a MediaStream to tell the app - 
this appears to be the 'removetrack' event (3.2.1) on the 
MediaStreamTrackList.  That should trigger the application to 
renegotiate the connection to tell the other side.  I should note this 
is not removestream on PeerConnection.

> I think this come up in another context where the browser wants to change the the video resolution of frame rate due to congestion control. We have not sorted out how the congestion control story w…

I'll respond to this later today.

-- 
Randell Jesup
[email-masked]

---

From: Justin Uberti <[email-masked]>
Date: Fri, 27 Jan 2012 14:49:03 -0500

On Fri, Jan 27, 2012 at 10:57 AM, Randell Jesup <[email-masked]>wrote:

>> Imagine that two browsers (A and B)  have set up audio and video streams
>> and are happily chatting away. At this point side A unplugs the camera.
>> What happens next? how does this get notified ? how does the PeerConnection
>> end up telling the JS that a new offer / answer exchange needs to be done
>> to indicate this video stream is gone? It seems like a callback indicating
>> something like this needed would be an good way to solve this.
>>
>
> MediaStreamTrackList.  That should trigger the application to renegotiate
> the connection to tell the other side.  I should note this is not
> removestream on PeerConnection.

Agree.

>
>
>
>> I think this come up in another context where the browser wants to change
>> the the video resolution of frame rate due to congestion control. We have
>> not sorted out how the congestion control story works but some of the nicer
>> systems end up significantly renovating video where there is a substantial
>> change of bandwidth. You can imagine that you are at home, have 100% of a
>> DSL link for incoming HD video, then someone starts iTunes and it
>> downloading 3 TV shows and suddenly you go from 100% of DSL to 25% of DSL.
>>
>

The encoder already is going to get feedback from the bandwidth estimator
to tell it to adjust its output bitrate, typically by changing QP. In cases
where a suitable QP cannot be maintained, I imagine the encoder pipeline
will internally start reducing resolution and framerate.

While I generally like the idea of the app being able to control these
sorts of things, I think we'd need to be able to expose more info to the
app (e.g. QP values) in order to allow it to do a good job.

Note that a similar issue exists where the app wants to reduce resolution
or framerate due to CPU overload, and should probably be solved similarly.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 27 Jan 2012 17:12:43 -0800

On 1/27/2012 11:49 AM, Justin Uberti wrote:
>
>
> On Fri, Jan 27, 2012 at 10:57 AM, Randell Jesup 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>         Imagine that two browsers (A and B)  have set up audio and
>         video streams and are happily chatting away. At this point
>         side A unplugs the camera. What happens next? how does this
>         get notified ? how does the PeerConnection end up telling the
>         JS that a new offer / answer exchange needs to be done to
>         indicate this video stream is gone? It seems like a callback
>         indicating something like this needed would be an good way to
>         solve this.
>
>
>     Yes; we need something associated with a MediaStream to tell the
>     app - this appears to be the 'removetrack' event (3.2.1) on the
>     renegotiate the connection to tell the other side.  I should note
>     this is not removestream on PeerConnection.
>
>
> Agree.

Note that I believe this means that the app will be required (pretty 
much) to monitor every MediaStreamTrackList for removetrack events; so 
that's another block of "mandatory" code the app needs to include.

>
>         I think this come up in another context where the browser
>         wants to change the the video resolution of frame rate due to
>         congestion control. We have not sorted out how the congestion
>         control story works but some of the nicer systems end up
>         significantly renovating video where there is a substantial
>         change of bandwidth. You can imagine that you are at home,
>         have 100% of a DSL link for incoming HD video, then someone
>         starts iTunes and it downloading 3 TV shows and suddenly you
>         go from 100% of DSL to 25% of DSL.
>
>
> The encoder already is going to get feedback from the bandwidth 
> estimator to tell it to adjust its output bitrate, typically by 
> changing QP. In cases where a suitable QP cannot be maintained, I 
> imagine the encoder pipeline will internally start reducing resolution 
> and framerate.

I agree.  Bitrate (and thus QP for video) will adjust automatically, and 
the browser is allowed to change the resolution within whatever 
negotiated parameters exist to provide a good experience (and I would 
encourage it to adjust resolution).

> sorts of things, I think we'd need to be able to expose more info to 
> the app (e.g. QP values) in order to allow it to do a good job.
>
> Note that a similar issue exists where the app wants to reduce 
> resolution or framerate due to CPU overload, and should probably be 
> solved similarly.

Agreed.

I'm going to try to throw together a strawman proposal for the app's 
interface to the congestion control parameters for the Interim...  at 
least a starting point for really discussing it, instead of waving hands 
as we have been, including me.

Sooo...  (And this is rough, and a first cut, but we need to start 
somewhere)

I propose we add a callback on PeerConnection and some attributes on it 
and MediaTracks:

{
...
     attribute unsigned long maxBitrate; // allow app to limit bitrate 
used for the PeerConnection
     attribute readonly unsigned long currentBitrate;  // [Open issue] - 
how much smoothing?
     attribute readonly long deltaBitrate; // maybe drop this

     attribute Function? onBitrateChange;
     attribute Function? onResolutionChange;  // not strictly needed

     // These are global settings that can be overridden on a per-track 
basis
     attribute float minFramerate;
     attribute float maxFramerate; // might not be needed
     attribute boolean preferMotion;  // false means preference for 
Resolution
}

I'm considering a 'hysterisis' value you can set to suppress minor 
changes in bandwidth from notifying you all the time.

To each MediaTrack:

{
...
     // If the app bumps up the currentBitrate of one track and doesn't 
lower another track, the
     // system will make cuts to keep the total bitrate within bounds.
     attribute unsigned long currentBitrate;
     attribute short priority;  // for allocating bits

     // The rest of these are for video (is short guaranteed enough, 
always?  Probably)
     attribute unsigned short width;  // Let app override the values 
chosen automatically
     attribute unsigned short height;
     attribute float framerate;
     attribute float minFramerate;
     attribute float maxFramerate; // might not be needed
     attribute boolean preferMotion;  // false means preference for 
Resolution

     attribute readonly unsigned short videoQuality;
     // audioQuality??

}

Should some of these be echoed in MediaTrackLists or MediaStreams?

-- 
Randell Jesup
[email-masked]

---

From: Justin Uberti <[email-masked]>
Date: Fri, 27 Jan 2012 20:37:38 -0500

On Fri, Jan 27, 2012 at 8:12 PM, Randell Jesup <[email-masked]>wrote:

>> On Fri, Jan 27, 2012 at 10:57 AM, Randell Jesup <[email-masked]<mailto:
>> [email-masked]**>> wrote:
>>
> Note that I believe this means that the app will be required (pretty much)
> to monitor every MediaStreamTrackList for removetrack events; so that's
> another block of "mandatory" code the app needs to include.
>

The app will probably need to monitor this regardless, if for no other
reason than to not display frozen video in its preview windows when the
track dies.

> the browser is allowed to change the resolution within whatever negotiated
> parameters exist to provide a good experience (and I would encourage it to
> adjust resolution).
>
> interface to the congestion control parameters for the Interim...  at least
> a starting point for really discussing it, instead of waving hands as we
> have been, including me.
>

>    attribute unsigned long maxBitrate; // allow app to limit bitrate used
> for the PeerConnection
>    attribute readonly unsigned long currentBitrate;  // [Open issue] - how
> much smoothing?
>

I assume this is total Tx bitrate, including FEC and RTX?

>    attribute readonly long deltaBitrate; // maybe drop this

What does |deltaBitrate| measure?

I think we also want estimated send and recv bandwidth, RTX bitrate, and
"regular" Tx bitrate.

>    attribute Function? onBitrateChange;

This will probably be changing frequently, so it might just be best to let
the app poll it.

>    // These are global settings that can be overridden on a per-track basis

This is may be difficult to use because the camera framerate may fluctuate
significantly due to lighting.

>
> I'm considering a 'hysterisis' value you can set to suppress minor changes
> in bandwidth from notifying you all the time.
>
> To each MediaTrack:
>
> {
> ...

I tend to think for v1 the app should choose priorities but let the PC
allocate bits.

>
>    // The rest of these are for video (is short guaranteed enough, always?
>  Probably)
>    attribute unsigned short width;  // Let app override the values chosen
> automatically

Is this QP, or something that maps to it?

> Should some of these be echoed in MediaTrackLists or MediaStreams?

I think this, as is, is a good start.

>
>
> --
> Randell Jesup
> [email-masked]
>
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>

---

From: Randell Jesup <[email-masked]>
Date: Fri, 27 Jan 2012 23:27:32 -0800

On 1/27/2012 5:37 PM, Justin Uberti wrote:
>
>
> On Fri, Jan 27, 2012 at 8:12 PM, Randell Jesup <[email-masked]
> <mailto:[email-masked]>> wrote:

>     much) to monitor every MediaStreamTrackList for removetrack events;
>     so that's another block of "mandatory" code the app needs to include.
>
>

And so that the other side doesn't show a frozen image.  We should 
(perhaps are part of example code) pull together all the "you really 
should have this" code blocks.

>     interface to the congestion control parameters for the Interim...
>       at least a starting point for really discussing it, instead of
>     waving hands as we have been, including me.
>
>     I propose we add a callback on PeerConnection and some attributes on
>     it and MediaTracks:
>
>     {
>     ...
>         attribute unsigned long maxBitrate; // allow app to limit
>     bitrate used for the PeerConnection
>         attribute readonly unsigned long currentBitrate;  // [Open
>     issue] - how much smoothing?
>
>

Yes.

> What does |deltaBitrate| measure?

It was an idea that probably (as I went through it) I should have 
removed - the idea was deltabitrate since last onbitratechange.  If the 
app cares, it's easy for it to calculate that itself.

> I think we also want estimated send and recv bandwidth, RTX bitrate, and

Probably, though that's more a function of a statistics module than 
input to the app for congestion reaction reasons (maybe RTX is useful).

> This will probably be changing frequently, so it might just be best to
> let the app poll it.

I hate polling...  The app may be relatively idle most of the time in 
stable, in-call scenarios otherwise.  The frequent changes are the 
reason I suggested some (settable) hysteresis before notification.

> This is may be difficult to use because the camera framerate may
> fluctuate significantly due to lighting.

In some cases it may not be possible to meet this requirement - but that 
doesn't mean you shouldn't try.  It also helps the system know that the 
app wants us to start trading off other things (resolution, color 
rendition, noise, etc) for framerate at a certain point, even if the 
setting is for preferResolution.

For most "video-call-like" uses I'd rather the camera stay over 15fps on 
low light (preferably over 20 or 25), and damn the resolution/noise/etc. 
  I realize some webcams are shitty and won't let us do that.

>         // If the app bumps up the currentBitrate of one track and
>     doesn't lower another track, the

Maybe.  I don't see this as major problem to implement (given the other 
items), but I'd be ok with it as readonly to start.  (I think apps will 
start playing "chase the desired codec bitrate" by fiddling priorities 
and watching the results.)

> Is this QP, or something that maps to it?

Yes; nominally QP, though that's not really a quality measurement.

> I think this, as is, is a good start.

Thanks.  This is basically what I've (we've) been suggesting from the 
start, and people generally seemed to like - letting the app have more 
input into how to react the bandwidth changes - who loses bits, who 
gains them, and when thresholds are reached removing (or adding) video 
streams, changing resolutions, etc.

-- 
Randell Jesup
[email-masked]

---

From: Stefan Hakansson LK <[email-masked]>
Date: Sat, 28 Jan 2012 17:18:24 +0100

On 01/28/2012 02:37 AM, Justin Uberti wrote:
>         <[email-masked] <mailto:[email-masked]>
>         <mailto:[email-masked]
>         <mailto:[email-masked]>__>> wrote:
>

My understanding was that if the camera is unplugged this would mean 
that the associated track enters state "ended" and the app is notified 
via the "ended" event.

>         changing QP. In cases where a suitable QP cannot be maintained,
>         I imagine the encoder pipeline will internally start reducing
>         resolution and framerate.
>
>
>     I agree.  Bitrate (and thus QP for video) will adjust automatically,
>     and the browser is allowed to change the resolution within whatever
>     negotiated parameters exist to provide a good experience (and I
>     would encourage it to adjust resolution).
>
>         While I generally like the idea of the app being able to control
>         these sorts of things, I think we'd need to be able to expose
>         more info to the app (e.g. QP values) in order to allow it to do
>         a good job.
>
>         resolution or framerate due to CPU overload, and should probably
>         be solved similarly.
>
>
>     Agreed.
>
>     [email-masked] <mailto:[email-masked]>
>
>     _________________________________________________
>     rtcweb mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/__listinfo/rtcweb
>     <https://www.ietf.org/mailman/listinfo/rtcweb>
>
>