Subject: Re: [rtcweb] Comments on draft-ietf-rtcweb-use-cases-and-requirements-07

From: Cullen Jennings <[email-masked]>
Date: Fri, 8 Jun 2012 08:23:15 -0400

On May 30, 2012, at 5:06 , Göran Eriksson AP wrote:

>> 
>> -----Original Message-----
>> From: [email-masked] 
>> [mailto:[email-masked]] On Behalf Of Cullen Jennings
>> Sent: den 16 maj 2012 03:31
>> To: [email-masked]
>> Subject: [rtcweb] Comments on 
>> draft-ietf-rtcweb-use-cases-and-requirements-07
>> 
>> 
>> Like to add case where application has one two video streams 
>> but one is far more important than the other. Should be a way 
>> to make sure that preferential treatment is given the the 
>> important stream over the less important streams. 
> 
> I agree that this is a relevant area to address to secure that WebRTC is competitive compared to
> apps done in native OS's, especiallin in enterprise context.
> 
> The current use case document states something like "being able to use priority functions in network nodes".
> This is vague since it touches many matters such as whether to put audio and video on the same IP-flow or have them in different
> ones, which is essential when considering mapping on LTE radio channels; the potential use of DS to identify flows, which
> may be relevant in enterprise context, etc, how to cater for treament of stuff on the datagram channel, multihoming consequences, etc.
> 
> I am not sure however how far into such matters we should go in this WG, but given the importance of making a
> good solution working across OS's and browser vendors and access technologies, I am leaning towards support a discussions about such
> details in this WG for instance using the use case document.
> 
> What is Your take of this? 

I think we need to discuss it in the spec. Let's imagine the case where we want the brows to set one DSCP for audio and a different one for video. I don't think we can have setting the DSCP 100% under control of the JS App as that has security problems. But the browser should set the DSCP to something and if there are multiple video streams, I think it is important to be able to indicate some video streams are less important than others - only the JS app knows which one is less important.

---

From: Göran Eriksson AP <[email-masked]>
Date: Fri, 8 Jun 2012 14:51:14 +0200

> From: Cullen Jennings [mailto:[email-masked]] 
> Sent: den 8 juni 2012 14:23
> To: Göran Eriksson AP
> Cc: [email-masked]
> Subject: Re: [rtcweb] Comments on 
> >> Like to add case where application has one two video 
> streams but one 
> >> is far more important than the other. Should be a way to make sure 
> >> that preferential treatment is given the the important stream over 
> >> the less important streams.
> > 
> > I agree that this is a relevant area to address to secure 
> that WebRTC 
> > is competitive compared to apps done in native OS's, 
> especiallin in enterprise context.
> > 
> > The current use case document states something like "being 
> able to use priority functions in network nodes".
> > This is vague since it touches many matters such as whether to put 
> > audio and video on the same IP-flow or have them in different ones, 
> > which is essential when considering mapping on LTE radio 
> channels; the potential use of DS to identify flows, which 
> may be relevant in enterprise context, etc, how to cater for 
> treament of stuff on the datagram channel, multihoming 
> consequences, etc.
> > 
> > I am not sure however how far into such matters we should 
> go in this 
> > WG, but given the importance of making a good solution 
> working across 
> > OS's and browser vendors and access technologies, I am 
> leaning towards support a discussions about such details in 
> this WG for instance using the use case document.
> > 
> I think we need to discuss it in the spec. 
>Let's imagine the 
> case where we want the brows to set one DSCP for audio and a 
> different one for video. I don't think we can have setting 
> the DSCP 100% under control of the JS App as that has 
> security problems. But the browser should set the DSCP to 
> something and if there are multiple video streams, I think it 
> is important to be able to indicate some video streams are 
> less important than others - only the JS app knows which one 
> is less important. 

Agree that the JS controlling the DSCP marking is an issue- I am more leaning towards
looking to solutions putting the "origin" in control. Also, the server side of the "app" should also be able to in a position to understand the relative importance on the different
streams/flows in an application. But this is of course up for discussion.

Another set of use cases could be considering the data on the data channel- we have scenarios where these have a significant impact on the UX and it could therefore be
relevant to consider these flows as well- after all, it may be nice talking to/looking at someone but if we also expect some shared data, I am pretty sure we will spend time complaining on the "darn" network more than doing whatever was the original purpose, :-)!

Going forward, I think Your proposal for adding use cases detailing requirements in this
area is a very good one and a good, next step. I expect this will be brought up in next week interim meeting for a discussion about how to proceed.

---

From: "Hutton, Andrew" <[email-masked]>
Date: Sun, 10 Jun 2012 15:33:17 +0000

Hi,

Some comments on draft-ietf-rtcweb-use-cases-and-requirements-07.

Section 4.1.1.1 (Simple Video Communication Service) contains the statement "Each user can also pause sending of media (audio, video, or both) and mute incoming media".  I think this and the associated requirements need some clarification for example whether the requirement is to control the direction of the media and therefore requires the generation of a new SDP offer with for example a direction of "recvonly/inactive".  The only associated requirement seems to be A8 but this does not help clarify what is meant by "pause" in the use case. It needs to be clarified whether the pause/resume/mute are actions within the browser or are actions which result in notifications to the remote user.

Maybe we could add a use case which involves controlling the direction of multiple streams to emulate switching between different users in a multiparty call scenario.

The statement "It is essential that the communication cannot be eavesdropped" appears in all use cases so could be moved to section 4.1 but I think it needs to be reworded in terms of the media always being encrypted as I don't like the term "eavesdropped" which is not defined.

Regards
Andy

---

From: Ted Hardie <[email-masked]>
Date: Sun, 10 Jun 2012 18:48:20 +0200

> Section 4.1.1.1 (Simple Video Communication Service) contains the statement "Each user can also pause sending of media (audio, video, or both) and mute incoming media".  I think this and the associated requirements need some clarification for example whether the requirement is to control the direction of the media and therefore requires the generation of a new SDP offer with for example a direction of "recvonly/inactive".

Howdy,

As a clarifying question, are you thinking that it is not clear
because non-signaling methods could achieve some of these?  That is
"muting" incoming media could also be accomplished entirely receive
side, by simply not rendering it?

regards,

Ted

 The only associated requirement seems to be A8 but this does not help
clarify what is meant by "pause" in the use case. It needs to be
clarified whether the pause/resume/mute are actions within the browser
or are actions which result in notifications to the remote user.
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Hutton, Andrew" <[email-masked]>
Date: Sun, 10 Jun 2012 20:03:07 +0000

Hi Ted,

See below.

Regards
Andy

> From: Ted Hardie [mailto:[email-masked]]
> Sent: 10 June 2012 18:48
> To: Hutton, Andrew
> Cc: [email-masked]
> Subject: Re: [rtcweb] More Comments on draft-ietf-rtcweb-use-cases-and-
> requirements-07
> 
> > Section 4.1.1.1 (Simple Video Communication Service) contains the
> statement "Each user can also pause sending of media (audio, video, or
> both) and mute incoming media".  I think this and the associated
> requirements need some clarification for example whether the
> requirement is to control the direction of the media and therefore
> requires the generation of a new SDP offer with for example a direction
> of "recvonly/inactive".
> 
> Howdy,
> 

Yes I guess so it is not clear whether the use case results in a local action in the browser or it whether it impacts the media and SDP. I want to make sure that it is possible to control the media direction in SDP (sendrecv/recvonly/sendonly/inactive) and have the browser behave appropriately.

> > Maybe we could add a use case which involves controlling the
> direction of multiple streams to emulate switching between different
> users in a multiparty call scenario.
> >
> >
> > The statement "It is essential that the communication cannot be
> eavesdropped" appears in all use cases so could be moved to section 4.1
> but I think it needs to be reworded in terms of the media always being
> encrypted as I don't like the term "eavesdropped" which is not defined.
> >
> > Regards
> > Andy
> >
> >
> >

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 10 Jun 2012 23:50:43 +0200

> The statement "It is essential that the communication cannot be eavesdropped" appears in all use cases so could be moved to section 4.1 but I think it needs to be reworded in terms of the media alw…
If we want to define the term, we might want to use the term 
"wiretapping", and refer to RFC 2804 for the definition.

(having been involved in that discussion, I've got some personal 
fondness for that definition...)

---

From: "Hutton, Andrew" <[email-masked]>
Date: Tue, 12 Jun 2012 20:39:34 +0000

Hi,

My suggestion would be to remove the statements from each use case regarding eavesdropping and replace it with bullets in section 4.1 for considerations which apply to all use cases. These should state that all media streams and data channels must be integrity and confidentiality protected.

Regards
Andy

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Harald Alvestrand
> Sent: 10 June 2012 23:51
> To: [email-masked]
> of "recvonly/inactive".  The only associated requirement seems to be A8
> but this does not help clarify what is meant by "pause" in the use
> case. It needs to be clarified whether the pause/resume/mute are
> actions within the browser or are actions which result in notifications
> to the remote user.
> >

---

From: Stefan Hakansson LK <[email-masked]>
Date: Wed, 13 Jun 2012 15:36:51 +0200

On 06/10/2012 10:03 PM, Hutton, Andrew wrote:
> Hi Ted,
>
> See below.
>
> Regards Andy
>
>> -----Original Message----- From: Ted Hardie
>> [mailto:[email-masked]] Sent: 10 June 2012 18:48 To: Hutton,
>> Andrew Cc: [email-masked] Subject: Re: [rtcweb] More Comments on
>> draft-ietf-rtcweb-use-cases-and- requirements-07
>>
>> statement "Each user can also pause sending of media (audio, video,
>> or both) and mute incoming media".  I think this and the
>> associated requirements need some clarification for example whether
>> the requirement is to control the direction of the media and
>> therefore requires the generation of a new SDP offer with for
>> example a direction of "recvonly/inactive".
>>
>> Howdy,
>>
>> because non-signaling methods could achieve some of these?  That
>> is "muting" incoming media could also be accomplished entirely
>> receive side, by simply not rendering it?
>>
>
> Yes I guess so it is not clear whether the use case results in a
> local action in the browser or it whether it impacts the media and
> SDP. I want to make sure that it is possible to control the media
> direction in SDP (sendrecv/recvonly/sendonly/inactive) and have the
> browser behave appropriately.

This is my personal understanding:

- the API (and this part has been there for a long time) allows the 
application to disable/enable individual tracks of MediaStreams
- So say you have a MediaStream (one audio track, one video track in 
this example) that is rendered using a video element; disabling audio 
would lead to silence, disabling video to that the video stops
- The question is: what should happen if the MediaStream is connected to 
a PeerConnection and sent to a peer?
- using recvonly does not seem appropriate as that would stop all ssrc:s 
belonging to the same m-line - and the app only disabled one
- one option could be to simply stop sending the RTP packets for this 
track. I don't know how reliable that would be
- I think there has been proposals signaling this in SDP or in RTCP, but 
I don't think there was ever any conclusion

For the other part, muting incoming media, this is simple: the app (at 
receiving browser) could just disable the tracks it wants in the 
incoming MediaStreams. A related question is if this should be signaled 
back to the sender (so it could stop sending). This is essentially a 
question of efficiency/optimization, and nothing stops the app from 
signaling that now using an app specific protocol (perhaps carried over 
the data channel), but perhaps this should be brought into SDP or RTCP.

>
>
>> regards,
>>
>> Ted
>>
>> The only associated requirement seems to be A8 but this does not
>> help clarify what is meant by "pause" in the use case. It needs to
>> be clarified whether the pause/resume/mute are actions within the
>> browser or are actions which result in notifications to the remote
>> user.
>>>
>> direction of multiple streams to emulate switching between
>> different users in a multiparty call scenario.
>>>
>>>
>> eavesdropped" appears in all use cases so could be moved to section
>> 4.1 but I think it needs to be reworded in terms of the media
>> always being encrypted as I don't like the term "eavesdropped"
>> which is not defined.
>>>
>>> Regards Andy
>>>
>>>
>>>
>>> _______________________________________________ rtcweb mailing
>>> list [email-masked]
> _______________________________________________ rtcweb mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/rtcweb

---

From: "Hutton, Andrew" <[email-masked]>
Date: Thu, 14 Jun 2012 09:15:12 +0000

On 13 June 2012 15:37 Stefan Hakansson wrote:

> track. I don't know how reliable that would be.

In the case of outgoing media then I think mute should simply disconnect the source (e.g. microphone) and there is no impact on RTP other than depending on the codec silence will be transmitted or SID frame etc. There is also no impact on RTCP which continues.

This is different to setting the direction to be recvonly which would completely stop the outgoing RTP packets and would normally be signaled to the remote party. This should of course be achieved using setLocalDescription.

> I don't think there was ever any conclusion

If we are talking about a local mute then I don't see any need to signal this but again this is different to changing the direction attribute in SDP.

> the data channel), but perhaps this should be brought into SDP or RTCP.

If the application wants to ask the remote party to pause sending of RTP then it should surely do this in SDP by setting the direction attribute appropriately (i.e. sendonly or inactive).

Regards
Andy

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 14 Jun 2012 11:21:34 +0200

Changing the subject line again, since we're discussing how to do mute, 
not whether to do it.....

> This is different to setting the direction to be recvonly which would completely stop the outgoing RTP packets and would normally be signaled to the remote party. This should of course be achieved…

Just to add to the complexity of the "mute" case:

When an audio stream is muted, should there be comfort noise sent?
If yes, what should determine the noise level?

(people who've never had to deal with comfort noise can safely tune out 
at this point)

---

From: Stefan Hakansson LK <[email-masked]>
Date: Thu, 14 Jun 2012 15:10:22 +0200

On 06/14/2012 11:21 AM, Harald Alvestrand wrote:
> Changing the subject line again, since we're discussing how to do
> mute, not whether to do it.....
>
>>> - the API (and this part has been there for a long time) allows
>>> the application to disable/enable individual tracks of
>>> MediaStreams - So say you have a MediaStream (one audio track,
>>> one video track in this example) that is rendered using a video
>>> element; disabling audio would lead to silence, disabling video
>>> to that the video stops - The question is: what should happen if
>>> the MediaStream is connected to a PeerConnection and sent to a
>>> peer? - using recvonly does not seem appropriate as that would
>>> stop all ssrc:s belonging to the same m-line - and the app only
>>> disabled one - one option could be to simply stop sending the RTP
>>> packets for this track. I don't know how reliable that would be.
>> In the case of outgoing media then I think mute should simply
>> disconnect the source (e.g. microphone) and there is no impact on
>> RTP other than depending on the codec silence will be transmitted
>> or SID frame etc. There is also no impact on RTCP which continues.
>>
>> This is different to setting the direction to be recvonly which
>> would completely stop the outgoing RTP packets and would normally
>> be signaled to the remote party. This should of course be achieved
>> using setLocalDescription.
>
> When an audio stream is muted, should there be comfort noise sent? If
> yes, what should determine the noise level?

To me it feels as that ideally we should stop sending RTP (and make the 
other end know that the track is disabled). If we go to comfort noise 
state, the track in the MediaStream at the other end would still be 
active (even though what is played could be very low level comfort 
noise), but to reflect the local state the remote track should go to 
"muted", shouldn't it?

>
> (people who've never had to deal with comfort noise can safely tune
> out at this point)
>
>>
>>
>>> - I think there has been proposals signaling this in SDP or in
>>> RTCP, but I don't think there was ever any conclusion
>> If we are talking about a local mute then I don't see any need to
>> signal this but again this is different to changing the direction
>> attribute in SDP.
>>
>>> For the other part, muting incoming media, this is simple: the
>>> app (at receiving browser) could just disable the tracks it wants
>>> in the incoming MediaStreams. A related question is if this
>>> should be signaled back to the sender (so it could stop sending).
>>> This is essentially a question of efficiency/optimization, and
>>> nothing stops the app from signaling that now using an app
>>> specific protocol (perhaps carried over the data channel), but
>>> perhaps this should be brought into SDP or RTCP.
>>>
>> If the application wants to ask the remote party to pause sending
>> of RTP then it should surely do this in SDP by setting the
>> direction attribute appropriately (i.e. sendonly or inactive).
>>
>> Regards Andy
>>
>> list [email-masked] https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Justin Uberti <[email-masked]>
Date: Thu, 14 Jun 2012 15:36:55 -0400

For "mute", stopping sending of RTP may trigger concealment on the remote
side. We currently send silent audio when muted to prevent this.
For "hold", you are notifying the remote side (e.g. a=inactive), so it
makes sense to stop sending RTP.

On Thu, Jun 14, 2012 at 9:10 AM, Stefan Hakansson LK <
[email-masked]> wrote:

> state, the track in the MediaStream at the other end would still be active
> (even though what is played could be very low level comfort noise), but to
> reflect the local state the remote track should go to "muted", shouldn't it?
>
>
>
>
>>> ______________________________**_________________ rtcweb mailing
>>> list [email-masked] https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>>>
>>
>> ______________________________**_________________ rtcweb mailing list
>> [email-masked] https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>>
>
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>

---

From: "Hutton, Andrew" <[email-masked]>
Date: Thu, 14 Jun 2012 20:01:10 +0000

I agree with Justin that mute is a local action and should not stop the RTP or change the state of the remote track.

Regards
Andy

On 14 Jun 2012, at 20:37, "Justin Uberti" <[email-masked]<mailto:[email-masked]>> wrote:

For "mute", stopping sending of RTP may trigger concealment on the remote side. We currently send silent audio when muted to prevent this.
For "hold", you are notifying the remote side (e.g. a=inactive), so it makes sense to stop sending RTP.

On Thu, Jun 14, 2012 at 9:10 AM, Stefan Hakansson LK <[email-masked]<mailto:[email-masked]>> wrote:
On 06/14/2012 11:21 AM, Harald Alvestrand wrote:
Changing the subject line again, since we're discussing how to do
mute, not whether to do it.....

To me it feels as that ideally we should stop sending RTP (and make the other end know that the track is disabled). If we go to comfort noise state, the track in the MediaStream at the other end would still be active (even though what is played could be very low level comfort noise), but to reflect the local state the remote track should go to "muted", shouldn't it?

Regards Andy

_______________________________________________ rtcweb mailing
list [email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/rtcweb

_______________________________________________ rtcweb mailing list
[email-masked]<mailto:[email-masked]> https://www.ietf.org/mailman/listinfo/rtcweb

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Martin Thomson <[email-masked]>
Date: Thu, 14 Jun 2012 13:13:49 -0700

> sense to stop sending RTP.

Since signaling takes a little while to transit the network, it makes
sense for hold to act as mute (send silence) until it is accepted.  I
think that most users will expect hold to take immediate effect, at
least from the perspective of sending of media.

---

From: "Hutton, Andrew" <[email-masked]>
Date: Fri, 15 Jun 2012 08:17:31 +0000

> least from the perspective of sending of media.

This is debatable and relates to the discussion about when to apply any change to the session description really the change should probably not be applied until the answer is received otherwise it might be necessary to restore the previous session description depending on the result of the offer/answer exchange.  I would say don't apply the change until the answer is received.

Regards
Andy

---

From: Stefan Hakansson LK <[email-masked]>
Date: Fri, 15 Jun 2012 12:34:31 +0200

On 06/14/2012 10:01 PM, Hutton, Andrew wrote:
> I agree with Justin that mute is a local action and should not stop the
> RTP or change the state of the remote track.
>
> Regards
> Andy

First, a short explanation of (my understanding of) what is in the API now:

* each track has a muted/unmuted and a enabled/disabled state
* enable state can be controlled by the application
* muted state can only be controlled by the browser
* muted overrides enabled (so a track that is enabled will still not 
play if it is muted)

The idea is that "mute" state is out of the control of the app (so the 
browser can override - and the user if the browser chrome allows). There 
has been some discussion indicating that the above is not sufficient and 
that we need to add more.

I think we're talking about different "mutes" in this thread.

One that is that you mute using e.g. a button on your headset. This 
would lead to audio samples with amplitude of zero being generated. This 
should not change any state on the track (locally or remotely) IMO, and 
silence/cn should be sent in the RTP.

Another is that the app has a "mute" button. Clicking this would 
(depending of the app) lead to the corresponding track being "disabled" 
in the local MediaStream. For the remote MediaStream, this is basically 
the same as "mute in the browser chrome" in the local case (since the 
source stops producing data), and I think the state there should change 
to "muted".

A third is that you mute the device via the browser chrome (I assume 
this possibility will exist). In this case the track state changes to 
muted, and I really think this should propagate to the other end of a 
PeerConnection.

Stefan

>> For "mute", stopping sending of RTP may trigger concealment on the
>> remote side. We currently send silent audio when muted to prevent this.
>> On Thu, Jun 14, 2012 at 9:10 AM, Stefan Hakansson LK
>> <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
>>     To me it feels as that ideally we should stop sending RTP (and
>>     make the other end know that the track is disabled). If we go to
>>     comfort noise state, the track in the MediaStream at the other end
>>     would still be active (even though what is played could be very
>>     low level comfort noise), but to reflect the local state the
>>     remote track should go to "muted", shouldn't it?
>>
>>
>>
>>
>>             _________________________________________________ rtcweb
>>             mailing
>>             list [email-masked] <mailto:[email-masked]>
>>             https://www.ietf.org/mailman/__listinfo/rtcweb
>>             <https://www.ietf.org/mailman/listinfo/rtcweb>
>>
>>
>>         _________________________________________________ rtcweb
>>         mailing list
>>         [email-masked] <mailto:[email-masked]>
>>         https://www.ietf.org/mailman/__listinfo/rtcweb
>>         <https://www.ietf.org/mailman/listinfo/rtcweb>
>>
>>
>>     _________________________________________________
>>     rtcweb mailing list
>>     [email-masked] <mailto:[email-masked]>
>>     https://www.ietf.org/mailman/__listinfo/rtcweb
>>     <https://www.ietf.org/mailman/listinfo/rtcweb>
>>
>>
>> [email-masked] <mailto:[email-masked]>

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 15 Jun 2012 12:34:46 +0200

I'd say it's an implementation issue. If I want to put a call on hold, I will stop sending media no matter what the response to the updated offer is.

Also, as we discussed in SIPCORE when working on the re-INVITE handling spec, clients may not even be able to fallback to the before-the-new-offer-was-sent state (probably not an issue for hold, but in general).

However,  I don't think it's up to us to specify how services work :) We should focus on what happens when no media is sent - for whatever reason.

Regards,

Christer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Hutton, Andrew
Sent: 15. kesäkuuta 2012 11:18
To: Martin Thomson; Justin Uberti
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments on draft-ietf-rtcweb-use-cases-and-requirements-07)

Regards
Andy

---

From: Christer Holmberg <[email-masked]>
Date: Fri, 15 Jun 2012 12:36:32 +0200

I think we DO, however, need to specify what happens if a new offer is rejected, and if the browser is not able to fallback to the previous state.

Regards,

Christer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christer Holmberg
Sent: 15. kesäkuuta 2012 13:35
To: Hutton, Andrew; Martin Thomson; Justin Uberti
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments on draft-ietf-rtcweb-use-cases-and-requirements-07)

Regards,

Christer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Hutton, Andrew
Sent: 15. kesäkuuta 2012 11:18
To: Martin Thomson; Justin Uberti
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments on draft-ietf-rtcweb-use-cases-and-requirements-07)

Regards
Andy

---

From: "DRAGE, Keith (Keith)" <[email-masked]>
Date: Fri, 15 Jun 2012 12:50:37 +0200

I agree it is important to agree on what resources are released (and therefore made available for some other purpose) as a result of the action. This confusion has existed ever since the PSTN invented a hold service.

Is the function merely to mute the transducers locally, or is the function to make resources (any of media, signaling, transducers, etc) available for some other purpose.

Regards

Keith

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Stefan Hakansson LK
Sent: 15 June 2012 11:35
To: Hutton, Andrew
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments on draft-ietf-rtcweb-use-cases-and-requirements-07)

Stefan

---

From: Martin Thomson <[email-masked]>
Date: Fri, 15 Jun 2012 13:22:11 -0700

On 15 June 2012 03:34, Christer Holmberg <[email-masked]> wrote:
> I don't think it's up to us to specify how services work :) We should focus on what happens when no media is sent - for whatever reason.

Take that and frame it.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 15 Jun 2012 16:42:08 -0400

> makes sense to stop sending RTP.

I would also suggest that for video-mute that instead of stopping video 
or sending black, the application should send a "mute image".  This 
could be done by changing the video source to be a fixed image (or small 
looped animation from a <video> element.  This greatly reduces confusion 
for users ("you just went black!" "where did you go?", etc)

This is largely an application issue, so long as you can redirect the 
source of input MediaStream for the outgoing video.

See the message stream around 2/13/2012 titled "Mute and MediaStream 
repointing".  One message:
http://www.w3.org/mid/[email-masked]

A quote:
------------------------------------

Since Hold is a slightly different concept, I believe 'muting' an 
outgoing stream should generate silence for audio, and some type of 
'muted' video message.  We used a camera icon with a circle-slash. 
Bandwidth for a static image can be dropped almost as much as you can 
for black.  I dislike (strongly) using 'black' to signal Mute because of 
codecs and the like in the path (and things like insertions of video 
'bugs' or other manipulation after the MediaStream generation (such as 
through the MediaStream Processing API proposal) might mess it up).

** This brings up an important point about repointing a MediaStream **

However, exactly what should be sent for Mute is probably an application 
issue - for video mute, the application should probably replace one 
source with a static (or not) image.  This speaks more to the 
MediaStream Processing API, since you don't want to renegotiate in order 
to change the tracks - you want to change the source of an existing track.

So, basically you want a MediaStream from getUserMedia() (call it 
MSCamera), a MediaStream from a static image or pre-recorded video (call 
it MSMute), and a MediaStream that can be either (MSMerged).  We can't 
send a track, just a stream, and again we don't want to renegotiate, so 
we need to create MSMerged from either MSCamera or MSMute.  The 
MediaStream Processing API should be able to handle this, much as it can 
handle smooth cross-fades of audio and stitching video clips together - 
see demos at:

http://robert.ocallahan.org/2012/01/mediastreams-processing-demos.html

This could be a canned video (or audio), a fixed image, or something 
algorithmically generated by the application using Workers ("You are 
second in line" "Your wait time is 3 minutes" "Please see our new fall 
line at http://bigstore.com/"), and equivalently-generated audio.

This would allow you to change the outgoing video source without 
triggering re-negotiations, which is very important.

Alternatively we'd need a way to override the source for individual 
tracks (and restore the default value), and a way to create a track as a 
discrete object, etc.

---------------------------------
end quote

-- 
Randell Jesup
[email-masked]

---

From: Martin Thomson <[email-masked]>
Date: Fri, 15 Jun 2012 13:49:47 -0700

On 15 June 2012 13:42, Randell Jesup <[email-masked]> wrote:
> I would also suggest that for video-mute that instead of stopping video or
> sending black, the application should send a "mute image".

The question I have is whether there is: a) a fixed image/video that
is serialized to RTP b) video "comfort noise", or c) local mute
image/video playback.  If it can't be c, then you might need to
explain yourself.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 15 Jun 2012 16:50:47 -0400

On 6/14/2012 4:13 PM, Martin Thomson wrote:
> On 14 June 2012 12:36, Justin Uberti<[email-masked]>  wrote:

Absolutely any Hold operation should Mute audio and video until the 
actual offer/answer is complete.  Been there, done that. :-)

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Fri, 15 Jun 2012 17:18:36 -0400

On 6/15/2012 4:49 PM, Martin Thomson wrote:
> On 15 June 2012 13:42, Randell Jesup<[email-masked]>  wrote:

Either we build it into the system (which I dislike - SetMuteImage() or 
whathaveyou, though we could), or we put it in the application domain 
and suggest or assist implementation.

The simple way to handle Mute is to revector the input to a "mute" or 
"hold" source, and they may not be the same (hold would have "Please 
Hold" and music, perhaps; mute would have silence and "Muted" or a 
camera with a circle/slash, etc).  You can also signal that out of band 
to let the other end take smart action (remove muted video image, 
display "muted" next to a name, etc).  This can be done via 
DataChannels, for example.  You may then reinvite for Hold to make it 
one-way, but that's mostly optional.

-- 
Randell Jesup
[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 16 Jun 2012 21:56:04 +0200

On 06/15/2012 11:18 PM, Randell Jesup wrote:
> Either we build it into the system (which I dislike - SetMuteImage() 
> or whathaveyou, though we could), or we put it in the application 
> domain and suggest or assist implementation.
>
> camera with a circle/slash, etc).  You can also signal that out of 
> band to let the other end take smart action (remove muted video image, 
The resulting requirements seem to me to be:

1) To be able to use a still image as source for a video stream
2) To be able to change the source of a video stream after its creation

Both are requirements on the GetUserMedia API. Can we link these to a 
specific use case, or do we need to modify the use case document so that 
the requirements are clear?

              Harald

---

From: "Jim Barnett" <[email-masked]>
Date: Sat, 16 Jun 2012 13:07:41 -0700

Harald,
  Your description below makes me nervous:  with getUserMedia, the user
grants permission to _specific_ media streams (for example audio, or
front camera, but not rear).  If the system wants " to change the source
of a video stream after its creation", shouldn't it be required to ask
permission again?  In this case,  common sense says that it's not
necessary, but that's because we're not changing to an arbitrary
alternative the source (e.g., suddenly switching on the rear camera) but
rather substituting some sort of 'null' or 'dummy' source.  Can we think
of a different way to phrase it?

- Jim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Harald Alvestrand
Sent: Saturday, June 16, 2012 3:56 PM
To: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments
ondraft-ietf-rtcweb-use-cases-and-requirements-07)

On 06/15/2012 11:18 PM, Randell Jesup wrote:
>>> I would also suggest that for video-mute that instead of stopping 
>>> video or sending black, the application should send a "mute image".
>>

              Harald

---

From: Harald Alvestrand <[email-masked]>
Date: Sat, 16 Jun 2012 22:34:10 +0200

On 06/16/2012 10:07 PM, Jim Barnett wrote:
> front camera, but not rear).
Yes, he grants permission to use of the camera. That permission persists 
until <we have to define this point in time>. There's some interesting 
language there now about "strong" and "weak" references to video streams.

>    If the system wants " to change the source
There are multiple ways to implement a requirement, as always.

One possibility is the simple one:

getUserMedia(..., function(strream) {
       outgoingstream = stream;
})
getStreamFromPicture(image, function(stream) {
       
outgoingstream.videoStreams(0).replaceContent(stream.videoStreams(0));
}..)

whatever the "replace" call does.
There may be a requirement that we keep a reference around that keeps 
the video stream, so that we can restore it, in order that the camera 
not close and can be used later:

getUserMedia(..., function(strream) {
       copyOfCameraStream = stream;
       outgoingstream = MediaStream(stream.audioTracks, stream.videoTracks);

})

getStreamFromPicture(image, function(stream) {
       
outgoingstream.videoStreams(0).replaceContent(stream.videoStreams(0));
}..)

restorePicture = function() {
      
outgoingtstream.videoStreams(0).replaceContent(copyOfCameraStream.video.Streams(0));
}

But there are surely multiple other ways to define it. What I want to 
make sure we have done first is to capture the requirement accurately.

---

From: "Jim Barnett" <[email-masked]>
Date: Sat, 16 Jun 2012 14:58:34 -0700

It's capturing the requirement accurately that I'm concerned with.  I
think "To be able to change the source of a video stream after its
creation" is too broad, in the sense that someone might interpret it to
mean that the system could shift from the front to the rear camera
without asking permission.  For mute and hold, we want something like
"switch to a dummy source and back".  The source that we switch to in
cases like this is not a 'first-class' video source, but rather a
place-holder.  That's very different from switching to another live
source.

- Jim

-----Original Message-----
From: Harald Alvestrand [mailto:[email-masked]] 
Sent: Saturday, June 16, 2012 4:34 PM
To: Jim Barnett
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments

On 06/16/2012 10:07 PM, Jim Barnett wrote:
> Harald,
>    Your description below makes me nervous:  with getUserMedia, the 
> user grants permission to _specific_ media streams (for example audio,

> or front camera, but not rear).
Yes, he grants permission to use of the camera. That permission persists
until <we have to define this point in time>. There's some interesting
language there now about "strong" and "weak" references to video
streams.

>    If the system wants " to change the source of a video stream after 
> its creation", shouldn't it be required to ask permission again?  In 
> this case,  common sense says that it's not necessary, but that's 
> because we're not changing to an arbitrary alternative the source 
> (e.g., suddenly switching on the rear camera) but rather substituting 
> some sort of 'null' or 'dummy' source.  Can we think of a different 
> way to phrase it?

       

getUserMedia(..., function(strream) {
       copyOfCameraStream = stream;
       outgoingstream = MediaStream(stream.audioTracks,
stream.videoTracks);

})

      
outgoingtstream.videoStreams(0).replaceContent(copyOfCameraStream.video.
Streams(0));
}

>> band to let the other end take smart action (remove muted video 
>> image, display "muted" next to a name, etc).  This can be done via 

> specific use case, or do we need to modify the use case document so 
> that the requirements are clear?
>
>                Harald
>
>
>
>

---

From: Randell Jesup <[email-masked]>
Date: Sat, 16 Jun 2012 20:38:30 -0400

> place-holder.  That's very different from switching to another live

Which, by the way, we should be able to do.

MediaStreams need to be able to be repointed, switched, processed, etc, 
OR you need to be able to replace a MediaStream in the inputs to 
PeerConnection with another one transparently without renegotiation (and 
that misses a bunch of uses).

The MediaStream Processing provides a mechanism to do this for both 
audio and video, plus the ability to do more complex operations 
(cross-fades audio-and-video, frame-accurate source switches, etc).

If you don't have or use that, you need the ability to replace a track 
in a MediaStream with another track.

All we're talking about are mechanisms to shuffle existing or creatable 
tracks/streams; a new camera source would require a call to 
getUserMedia() and a dialog.  Once you had front and back, you could 
switch sources without effectively shutting down all media and restarting.

Simple one camera -> one video stream mechanisms are insufficient for 
application using these features.  If you force people, they'll roll 
their own using canvases, etc.

I want something generic and usable in lots of interesting ways, not a 
few special-purpose pre-coded mechanisms (for example, a 
track.setMuteImage() method).

-- 
Randell Jesup
[email-masked]

---

From: "Sunyang (Eric)" <[email-masked]>
Date: Sun, 17 Jun 2012 06:48:04 +0000

You said:  "MediaStreams need to be able to be repointed, switched, processed, etc, OR you need to be able to replace a MediaStream in the inputs to PeerConnection with another one transparently without renegotiation (and that misses a bunch of uses)."

But if the MediaStream  you choose to replace the existing one contain codecs that not supported by remote end, I think we still need renegotiation.
Or you can make sure that the browser can choose the same codecs as default with the replaced one, I think browser can not do this.

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Randell Jesup
发送时间: 2012年6月17日 8:39
收件人: [email-masked]
主题: Re: [rtcweb] Mute implementations (Re: More Comments ondraft-ietf-rtcweb-use-cases-and-requirements-07)

-- 
Randell Jesup
[email-masked]

---

From: Randell Jesup <[email-masked]>
Date: Sun, 17 Jun 2012 03:47:18 -0400

> Or you can make sure that the browser can choose the same codecs as default with the replaced one, I think browser can not do this.

It's possible that renegotiation might be required in *some* cases with
pre-encoded data from the camera, but not many (if any), since we need
to rate-control the data. We would need to decode the camera stream
(H.264 typically), then re-encode at the correct bitrate. Or we'll need
to change the camera bitrate targets on a frame-by-frame basis (unlikely).

-- 
Randell Jesup
[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 17 Jun 2012 10:00:58 +0200

On 06/16/2012 11:58 PM, Jim Barnett wrote:
> without asking permission.
People certainly want to shift from the front to the rear camera, and 
people want to have the ability to get all that asking-for-permissions 
stuff out of the way before that.

I recommend the Media Task Force's "additional scenarios" for more 
usages of "local media".
Jjim, I know you know where this is, but for others:
http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/scenarios.html

>    For mute and hold, we want something like
ObLangDisagreement: Files, images and the output of video processors are 
very real sources, and should not be "second class".

Switching between live streams (that we have permission to access) is 
going to be required, but I think this belongs on the Media TF mailing list.

> until<we have to define this point in time>. There's some interesting

---

From: "Sunyang (Eric)" <[email-masked]>
Date: Sun, 17 Jun 2012 08:45:13 +0000

Most pad and smarthphone, front and back camera has different quality.
Not to metion the high resolution local video file (720p?)
If we switch mediastream between them, and like what you said, you decoded-encoded-control-frame-by-frame,
the system may be slow down, if you use codecs in browser, it will be more slower.

The point is that we accept the renegoation rather then decode-encode operation to fit for previous SDP.
Renegoation is not serious, we can accept it, but we can not accept the performance degradation.

-----邮件原件-----
发件人: [email-masked] [mailto:[email-masked]] 代表 Randell Jesup
发送时间: 2012年6月17日 15:47
收件人: [email-masked]
主题: Re: [rtcweb] 答复: Mute implementations (Re: More Comments ondraft-ietf-rtcweb-use-cases-and-requirements-07)

-- 
Randell Jesup
[email-masked]

---

From: "Sunyang (Eric)" <[email-masked]>
Date: Sun, 17 Jun 2012 09:23:57 +0000

Do we need add one more scenario:

The same video B2B communication, and switch video stream between front and rear webcam?
The video streams displayed in the same video element, but source changed.

There is on relative scenario in media capture scenarios, “video diary….”, but not not related with above mentioned scenario.

Should we add it?

发件人: [email-masked] [mailto:[email-masked]] 代表 Harald Alvestrand
发送时间: 2012年6月17日 16:01
收件人: Jim Barnett
抄送: [email-masked]
主题: Re: [rtcweb] Mute implementations (Re: More Comments ondraft-ietf-rtcweb-use-cases-and-requirements-07)

without asking permission.
People certainly want to shift from the front to the rear camera, and people want to have the ability to get all that asking-for-permissions stuff out of the way before that.

I recommend the Media Task Force's "additional scenarios" for more usages of "local media".
Jjim, I know you know where this is, but for others:
http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/scenarios.html

source.
ObLangDisagreement: Files, images and the output of video processors are very real sources, and should not be "second class".

Switching between live streams (that we have permission to access) is going to be required, but I think this belongs on the Media TF mailing list.

- Jim

-----Original Message-----

To: Jim Barnett

Cc: [email-masked]<mailto:[email-masked]>

Harald,

streams.

way to phrase it?

       outgoingstream = stream;

})

}..)

stream.videoTracks);

})

}..)

Streams(0));

}

- Jim

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On

To: [email-masked]<mailto:[email-masked]>

On 15 June 2012 13:42, Randell Jesup<[email-masked]><mailto:[email-masked]>

wrote:

explain yourself.

2) To be able to change the source of a video stream after its

creation

               Harald

rtcweb mailing list

---

From: Randell Jesup <[email-masked]>
Date: Sun, 17 Jun 2012 10:16:22 -0400

> Renegoation is not serious, we can accept it, but we can not accept the performance degradation.

I'm not certain what you're saying here. If the camera is encoded we
likely are having to decode and re-encode anyways (perhaps with a scale,
crop or composite operation as well, which require decoded video as well).

The point is that some operations (mute, hold, etc) *need* to be "on the
next frame" or close. This can be a temporary thing (mute video until
renegotiate as recv-only for hold), but it must not wait for a
renegotiation.

The system can and will scale the new video source down to the same
outgoing resolution already selected, and if it wants to it can scale it
up to the previous resolution if it's lower (that probably isn't a good
idea, but can be done).

-- 
Randell Jesup
[email-masked]

---

From: "Jim Barnett" <[email-masked]>
Date: Sun, 17 Jun 2012 07:17:36 -0700

I still think that there is a significant difference between
files/images and live sources.  A live source provides a lot of
information about my environment - what's going on, whether I have
shaved, etc.  The file or image source that we use for hold is designed
to provide _no_ information about my environment.  In a user's mind,
showing "please wait" or a  images of a waterfall is very different from
showing anything _live_ from his environment, and we may want to reflect
that difference in the requirements.  For example, does the user have to
give explicit permission to show the "Please wait" source when the call
is put on hold, or is that configured as a default in the browser?  (The
latter, I'd  think, which makes it different from the front and rear
cameras.)

-          Jim

P.S. I agree that the system can switch from front to rear cameras once
it has permission for both, though it should make it clear that it is
doing so.   

From: Harald Alvestrand [mailto:[email-masked]] 
Sent: Sunday, June 17, 2012 4:01 AM
To: Jim Barnett
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments
ondraft-ietf-rtcweb-use-cases-and-requirements-07)

I recommend the Media Task Force's "additional scenarios" for more
usages of "local media".
Jjim, I know you know where this is, but for others:
http://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/scenarios.ht
ml

Switching between live streams (that we have permission to access) is
going to be required, but I think this belongs on the Media TF mailing
list.

 
- Jim
 
-----Original Message-----
From: Harald Alvestrand [mailto:[email-masked]] 
Sent: Saturday, June 16, 2012 4:34 PM
To: Jim Barnett
Cc: [email-masked]
Subject: Re: [rtcweb] Mute implementations (Re: More Comments

	Harald,
	   Your description below makes me nervous:  with getUserMedia,
the 
	user grants permission to _specific_ media streams (for example
audio,

	   If the system wants " to change the source of a video stream
after 
	its creation", shouldn't it be required to ask permission again?
In 
	this case,  common sense says that it's not necessary, but
that's 
	because we're not changing to an arbitrary alternative the
source 
	(e.g., suddenly switching on the rear camera) but rather
substituting 
	some sort of 'null' or 'dummy' source.  Can we think of a
different 
	way to phrase it?

 

 
 
})

 

	 
	- Jim
	 
	-----Original Message-----
	From: [email-masked] [mailto:[email-masked]]
On 

			On 15 June 2012 13:42, Randell
Jesup<[email-masked]> <mailto:[email-masked]> 

wrote:

				I would also suggest that for video-mute
that instead of stopping 
				video or sending black, the application
should send a "mute image".

			The question I have is whether there is: a) a
fixed image/video that

			is serialized to RTP b) video "comfort noise",
or c) local mute 
			image/video playback.  If it can't be c, then
you might need to 
			explain yourself.

		Either we build it into the system (which I dislike -
SetMuteImage() 
		or whathaveyou, though we could), or we put it in the
application 
		domain and suggest or assist implementation.
		 
		The simple way to handle Mute is to revector the input
to a "mute" or

		"hold" source, and they may not be the same (hold would
have "Please 
		Hold" and music, perhaps; mute would have silence and
"Muted" or a 
		camera with a circle/slash, etc).  You can also signal
that out of 
		band to let the other end take smart action (remove
muted video 
		image, display "muted" next to a name, etc).  This can
be done via 
		DataChannels, for example.  You may then reinvite for
Hold to make it

	 
	 
	Both are requirements on the GetUserMedia API. Can we link these
to a 
	specific use case, or do we need to modify the use case document
so 
	that the requirements are clear?
	 
	               Harald

---

From: Stefan Hakansson LK <[email-masked]>
Date: Sun, 17 Jun 2012 16:33:08 +0200

For the fun of it I thought a little while about what options that are 
currently available with the APIs that are in the W3C docs as is. The 
list is in no way exhaustive, there are other ways to do it.

Perhaps this is not sufficient for what people want to do, but at least 
there are some things you can do already:

1. Replace video with the display of an image
=============================================
One solution: Use the “poster” attribute of the video element. Quote 
from the html spec: “The poster attribute gives the address of an image 
file that the user agent can show while no video data is available.”. We 
can spec it such that if no video track of a MediaStreamis active, that 
counts “as no data available”. This way, as soon as the user “mutes” the 
video, an image will replace it at display. The site can supply the 
image (either a default by the site or one “profile” pic selected by the 
user), or it can be provided over the data channel (as a blob).

Another solution: A signal is sent on the data channel (possibly 
accompanied by the image) on the data channel, the peer’s app overlays 
the image on top of the video element as result (and switches back on 
another signal).

A third (similar) solution: The app at the receiving end listens to 
“onunmute” of the video track in question; once it fires an image is 
overlayed.

A fourth: The peer’s app analyzes the video using canvas, and overlays 
an image when there is no data in the video.

2. Switch between front and rear camera
========================================
One solution: create a MediaStream (e.g. by calling getUserMedia twice 
immediately after each other, or by combining existing tracks into a new 
MediaStream) containing two video tracks (front+back cam). The video 
shown is toggled by enabling/disabling video tracks (the video element 
plays the first active one in the case of several video tracks).

Another: use add/removeTrack to switch video track being sent.

3. Play music on hold
=====================
One solution (audio only case): The site provides an URL to a (music) 
file to play when on hold. The peer’s app registers an event listener 
that listens to “onmute” on the incoming audio track. When “onmute” 
fires, the app changes the source of the audio element to the URL of the 
music file.

One solution if the communication is audio+video, and the intent is to 
just replace the audio (but continue playing video): assume a 
MediaStream with one audio and one video track is used. Play the audio 
in an audio element, the video in a (muted) video element. At on hold, 
replace the source of the audio element with the music file.

A solution if the communication is audio+video, and the intent is to 
replace audio and video: either “mute” the tracks (at sending side); 
receiving end gets “onmute” event and replaces the .src of the video 
element with the URL to a video file, or signal on the data channel 
something that makes the other end replace the src of the video element 
with an URL to a file

Another solution is that the sending end does “removeStream” from 
PeerConnection. This fires “onremovestream” at the receiving side; that 
event can trigger the change of src to the video element. This solution 
frees up resources.

Note that once the recording function has been specced (in such a way 
that the recorded file can be the source of a MediaStream) more options 
will become available.

--Stefan

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 17 Jun 2012 16:44:04 +0200

On 06/17/2012 04:17 PM, Jim Barnett wrote:
> information about my environment -- what's going on, whether I have 
> shaved, etc.  The file or image source that we use for hold is 
> designed to provide _/no/_ information about my environment.  In a 
> user's mind, showing "please wait" or a  images of a waterfall is very 
> different from showing anything _/live/_ from his environment, and we 
> may want to reflect that difference in the requirements.  For example, 
> does the user have to give explicit permission to show the "Please 
> wait" source when the call is put on hold, or is that configured as a 
> default in the browser?  (The latter, I'd  think, which makes it 
> different from the front and rear cameras.)
>
>
I agree with respect to the needed permissions; this is mostly a 
language issue.

I think we will have some applications (like the shared YouTube viewer 
in hangouts) where a stored medium is the most important piece of the 
app - but I don't think that means we need special access procedures for 
it (the "file upload" dialog for getting permissions to access a file 
should be sufficient), and for resources that are already accessible to 
the Web app, like images loaded to the page - I see absolutely no reason 
to put extra requirements on it.

But as I said - Media task force.

---

From: Randell Jesup <[email-masked]>
Date: Sun, 17 Jun 2012 12:02:20 -0400

CCing Media-capture as much of this touches on issues there.  For 
context, you can read the recent rtcweb archives.  The general topic is 
"Mute" (and Hold) and both replacing video with an image or pre-recorded 
video clip (repeating if short), and related issue of replacing a 
front-camera video with a rear-camera video without operations that take 
significant time (renegotiation of the peerconnection).  Basically 
immediate source switches.

On 6/17/2012 10:17 AM, Jim Barnett wrote:
> information about my environment – what’s going on, whether I have
> to provide _/no/_ information about my environment.  In a user’s mind,
> showing “please wait” or a  images of a waterfall is very different from
> showing anything _/live/_ from his environment, and we may want to
> reflect that difference in the requirements.  For example, does the user
> have to give explicit permission to show the “Please wait” source when
> the call is put on hold, or is that configured as a default in the
> browser?  (The latter, I’d  think, which makes it different from the
> front and rear cameras.)

Selecting an image or video from the user's filesystem would require the 
user's direct permission (ala <input type="file">).  Sourcing a 
mediastream from a web source (i.e. part of the app) or from a 
app-generated canvas (with "please wait" in it) would not.

If the app requests both front and rear camera streams and gets them 
(two getUserMedia() calls), then permission is already given and the app 
should be able to do anything with those streams.

If the app hasn't requested the rear camera yet, requesting would have 
to be done first and would engender a request to the user to permit access.

Once the app has both streams, it should be able to switch between the 
streams on demand (front/back UI button in the app), with no delay, and 
certainly no renegotiation of the call.

If there's a reason it could renegotiate after switching.  For (a 
contrived) example, if it had negotiated a call with a low maximum 
resolution, it could switch (and downscale the new hires stream) until a 
renegotiation allowed sending the full resolution.  That's a contrived 
example, but there might be some real-world examples, and that's up to 
the app.

The same things apply to switching between video (live or recorded) and 
still images.

These speak to how MediaStream work and can be processed, and how a 
track in a MediaStream can be used to represent different things, and 
how that's different than a MediaStream with two tracks.  Some of this 
impacts the interface and assumptions on MediaStreams and tracks in 
PeerConnection.  These are issues because PeerConnection, <video> (and 
others) may play a particular track of a MediaStream.

I'll further respond to Stefan's detailed email.

-- 
Randell Jesup
[email-masked]

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 17 Jun 2012 18:17:54 +0200

> But if the MediaStream  you choose to replace the existing one contain codecs that not supported by remote end, I think we still need renegotiation.
Alert.... a MediaStream doesn't contain a codec.

http://wiki.alvestrand.no/w/index.php/The_Byte_Stream_Fallacy

A MediaStream may be linked to a source that provides data in a given
codec format; it may be linked to a sink that accepts data in the given
codec format, and as a result the work that goes on to support the
moving of data from the source to the sink may be more or less complex.

But saying that a MediaStream "contains a codec" is (to my mind) not a
good idea.

---

From: Randell Jesup <[email-masked]>
Date: Sun, 17 Jun 2012 12:32:43 -0400

CC'd to both lists, as this affects both Media Capture (MediaStreams) 
and rtcweb protocol issues (and PeerConnection API issues, but I'm not 
yet including the WebRTC list).

> 1. Replace video with the display of an image

This is a receiver-side action and thus is not very useful for Mute and 
Hold operations.  This would depend on those stopping transmitting 
(which could cause other issues) and would infer that lack of reception 
means mute or hold - while it could also mean a temporary or permanent 
loss of connectivity, etc.

> the image on top of the video element as result (and switches back on

I've done this (using RTCP APP as a data channel), and it works well 
*if* both sides are the same application, and it still requires that the 
sources immediately mute (though black would be ok in this case). 
However, if you ever need to talk to something that doesn't understand 
this out-of-band "I'm muted" signal, then you really want to Mute/Hold 
with an image (and sound for Hold).  (And I've had to do this as well.)

> “onunmute” of the video track in question; once it fires an image is

How does "onmute" work?  This similarly presupposes agreement on the 
out-of-band message, and also has the same issue when in a heterogeneous 
environment.

> an image when there is no data in the video.

a) ugh, what a waste of CPU cycles
b) Has the same problem as the first - the receiver doesn't know why 
it's getting no video (or black video).  Detecting no incoming video and 
overlaying makes sense (though timeouts can be tricky to avoid false 
positives), but it's not a replacement for Mute and Hold from the source.

> 2. Switch between front and rear camera

This means toggling would require a renegotiation end-to-end before the 
switch occurs.  Massively annoying (delay, perhaps long) and means more 
complexity required for receiving - I should be able to switch cameras 
and your simple receiver app shouldn't need to do a bunch of footwork.

> Another: use add/removeTrack to switch video track being sent.

Ditto.

> 3. Play music on hold

Again, this depends on the "onmute" notification somehow getting across, 
and there may be a silence gap until it does (unless it's sent inband in 
the RTP somehow (header extensions?), and then there are redundancy needs.

If one were developing a single, non-heterogeneous app one might do 
this, but relying on this in a heterogeneous world has problems.  (For 
example, connection to a PSTN gateway - you'd have to build this 
behavior into the WebRTC<->PSTN gateway and tightly specify it at the 
rtcweb level.)  This amounts to build a specification for an 
application, not for a set of tools and capabilities others can use to 
build apps.

> replace the source of the audio element with the music file.

stream = getUserMedia();
audio.src = stream;
video.src = stream;
video.mute();
audiostream = audio.captureStreamUntilEnded();
videostream = video.captureStreamUntilEnded();
mergedstream = new MediaStream(audio track from audiostream, video track 
from videostream);
pc.addStream(mergedstream);

onmute: audio.src = music;

Problem (other than this is a bit painful and wasteful): the audio and 
video streams are now unsynchronized.

> something that makes the other end replace the src of the video element

There's no transmittal of these "onmute" events you're talking about 
across the RTP layer.

See the previous arguments about receiver-side display, timing, etc.

> event can trigger the change of src to the video element. This solution

This requires a full renegotiation (which causes onremovestream to fire 
once complete).

> that the recorded file can be the source of a MediaStream) more options

I'm already assuming something like that 
(video.captureStreamUntilEnded()) - see the MediaStream Processing API 
proposal.

-- 
Randell Jesup
[email-masked]

---

From: Stefan Hakansson LK <[email-masked]>
Date: Mon, 18 Jun 2012 14:08:00 +0200

Hi Randell,

some feedback inline.

But overall, this was done more as a thought exercise. I'm not at all 
against being able to replace the source of a MediaStream(track) in 
operation. But I don't know when that should be added.

Br,
Stefan

> loss of connectivity, etc.

I think that a track should enter state "muted" (in JS) if no data is 
received, as this is in line with how "muted" is defined. If no data is 
received due to connection loss this will be detected on lack of 
ICE-messages (or RTCP?) I guess and become known to the app via the 
PeerConnection state changing from "open" to "connecting".

That said, I think the changed state of the track should be explicitly 
signaled. One proposal I have heard is to use the "sending" attribute in 
http://datatracker.ietf.org/doc/draft-lennox-mmusic-sdp-source-selection/?include_text=1, 
another (that I like better because timing should be better) is to use 
something like what is in 
http://datatracker.ietf.org/doc/draft-westerlund-avtext-rtp-stream-pause/ but 
for the send direction (basically "pause-indication" sent in RTCP). But 
perhaps in a first version the lack of RTP packets for a certain time 
could work as "mute" indication.

> with an image (and sound for Hold).  (And I've had to do this as well.)

Right, this requires the applications to understand each other. But that 
is already a prerequisite, isn't it? An arbitrary app would not 
understand that a PeerConnection object should be created just because 
an SDP offer was pushed to it from the server and so on.

> out-of-band message, and also has the same issue when in a heterogeneous

Re. "onmute", see above. Agree on that it presupposes agreement, but 
some kind of agreement is needed anyway.
> positives), but it's not a replacement for Mute and Hold from the source.

Agree to that it will consume CPU cycles.

> and your simple receiver app shouldn't need to do a bunch of footwork.

I don't agree fully. The entire MediaStream (in this case consisting of 
one audio and two video tracks) is negotiated at the same time. But only 
one of the video tracks is actively transmitting RTP most of the time.

>> Another: use add/removeTrack to switch video track being sent.

Here I agree that there would be a re-negotiation.

> the RTP somehow (header extensions?), and then there are redundancy needs.

Yep, "onmute" must be trigged by something.

> application, not for a set of tools and capabilities others can use to

Again, I don't think you could develop two different clients in 
isolation anyway. Have not thought about the gateway issues.

> video streams are now unsynchronized.

I guess this part is a bit underspecified yet. It is said that tracks in 
a MediaStream should be synchronized, but not what happens when 
different tracks are played in different elements.

> There's no transmittal of these "onmute" events you're talking about

I think we need something like that added (as discussed above) eventually.

> This requires a full renegotiation (which causes onremovestream to fire

Yes.

> (video.captureStreamUntilEnded()) - see the MediaStream Processing API

It should get into the spec once designed.