Subject: [rtcweb] Basic scenario 'impossible?' to achieve with the actual API

From: José Luis Millán <[email-masked]>
Date: Tue, 2 Jul 2013 18:04:00 +0200

Hi,

Please, let me know how this normal use case could be solved with the
current API.

Alice and Bob have accessed to a Web page offering a basic WebRTC
application   that they use to speak and see each other. They are in the
middle of a media session right now.

Alice presses the "Video" button in her web application. A SDP offer with
video description is sent to Bob:

Bob doesn't want to see her, so his web application rejects the media by
using any valid method for that:
-- a=inactive
-- m=video 0

Alice web application received a SDP answer which was consumed successfully
by the RTCPeerConnection. Alice thinks she is sharing her video, but she is
not.

The application "Video" button is disabled since the video is already being
shared (theoretically)

The application "self-Video" HTML5 video element displays Alice's face. She
thinks that Bob is seeing her face too, but he did indeed rejected so.

I hope this common scenario can be achieved somehow with the current API. I
can't find how to do it though.

Thanks in advance

-- 
José Luis Millán

---

From: Enrico Marocco <[email-masked]>
Date: Tue, 2 Jul 2013 18:13:48 +0200

> Please, let me know how this normal use case could be solved with the

Just off the top of my head:

Audio-only call establishment:

A --{"action":"CALL","sdp":"v=0..."}-> B
B --{"action":"ACCEPT CALL","sdp":"v=0..."}-> A

Rejected audio+video upgrade proposal:

A --{"action":"UPGRADE","sdp":"v=0..."}-> B
B --{"action":"REJECT UPGRADE"}-> A

Alice rolls back and removes local video display.

I may be missing something, but doesn't look like rocket science to me...

Enrico

---

From: José Luis Millán <[email-masked]>
Date: Tue, 2 Jul 2013 18:30:16 +0200

Thanks Enrico,

Yes, you are right. I was missing a detail.

Now imagine you are not using your own wire protocol but a standard one,
where accepting the request doesn't imply accepting or rejecting the SDP
offer, since SDP has its own methods to accept or reject the call as
commented before.

A --{"action":"CALL","sdp":"v=0..."}-> B
B --{"action":"ACCEPT CALL","sdp":"v=0\r\nm=video 0..."}-> A

2013/7/2 Enrico Marocco <[email-masked]>

> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>
>

-- 
José Luis Millán

---

From: José Luis Millán <[email-masked]>
Date: Tue, 2 Jul 2013 18:35:01 +0200

2013/7/2 José Luis Millán <[email-masked]>

> offer, since SDP has its own methods to accept or reject the call as

I meant 'accept or reject the SDP offer or part of it'  where is said
'accept or reject the call'

-- 
José Luis Millán

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 2 Jul 2013 18:47:43 +0200

> B --{"action":"REJECT UPGRADE"}-> A

SDP itself is a media negotiation protocol so SDP (and its WebRTC
"API") should provide a way to determine whether the remote has
accepted or not our new stream/track. But how? Current API lacks any
kind of event for the case in which the remote replies a SDP rejecting
our new local strem/track.

--
Iñaki Baz Castillo
<[email-masked]>

---

From: Jim Barnett <[email-masked]>
Date: Tue, 2 Jul 2013 17:20:04 +0000

When the JS calls setRemote with the response (which rejects the video), the UA will know that the video has been rejected.  Is the idea that the UA will let the user know via the chrome?  Do we think it is the JS applications job to notify the user of success/failure, or the UA's?  I don't recall a discussion of this.

- Jim

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Iñaki Baz Castillo
Sent: Tuesday, July 02, 2013 12:48 PM
To: Enrico Marocco
Cc: [email-masked]
Subject: Re: [rtcweb] Basic scenario 'impossible?' to achieve with the actual API

2013/7/2 Enrico Marocco <[email-masked]>:
> A --{"action":"CALL","sdp":"v=0..."}-> B B --{"action":"ACCEPT 
> CALL","sdp":"v=0..."}-> A
>
> A --{"action":"UPGRADE","sdp":"v=0..."}-> B B --{"action":"REJECT 
> UPGRADE"}-> A

SDP itself is a media negotiation protocol so SDP (and its WebRTC
"API") should provide a way to determine whether the remote has accepted or not our new stream/track. But how? Current API lacks any kind of event for the case in which the remote replies a SDP rejecting our new local strem/track.

---

From: José Luis Millán <[email-masked]>
Date: Tue, 2 Jul 2013 20:05:22 +0200

In order to avoid the exposed circumstance, where a user has to 'suppose'
she is sharing her media because there is no way to certainly know, yes,
the JS application should be able to, at least, check if a localMediaStream
is being transmitted out of the RTCPeerConnection.

This would be a simple task if each mediaStream, or mediaStreamTrack had
it's own associated (even if logical because of Boundle) transport. In that
case, such connectivity status would give the information and much more
control over each media element.

2013/7/2 Jim Barnett <[email-masked]>

> When the JS calls setRemote with the response (which rejects the video),
> the UA will know that the video has been rejected.  Is the idea that the UA
> will let the user know via the chrome?  Do we think it is the JS
> applications job to notify the user of success/failure, or the UA's?  I
> don't recall a discussion of this.
>
> - Jim
>
> "API") should provide a way to determine whether the remote has accepted
> or not our new stream/track. But how? Current API lacks any kind of event
> for the case in which the remote replies a SDP rejecting our new local
> strem/track.
>
> --
> Iñaki Baz Castillo
> <[email-masked]>

-- 
José Luis Millán

---

From: Martin Thomson <[email-masked]>
Date: Tue, 2 Jul 2013 11:05:52 -0700

> Alice rolls back and removes local video display.

This is fine for the case that has been proposed, but it doesn't work
as well if A and B are initiating a session.  In that case, you would
need two rounds of negotiation to get the functionality you describe.

There are two separate problems here:

1. How do I reject a proposed stream?

2. How do I learn that my proposed stream was rejected?

I'm sure that there are controls for this already...  Maybe.  You
could, as Enrico suggests, add streams one-by-one using multiple
offer/answer rounds - then you would be able to isolate what was
rejected. That doesn't really scale.

Note: These questions are only relevant with offer/answer.  Choosing
not to do offer/answer means that the application is in control, so
the question is moot.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 2 Jul 2013 11:06:41 -0700

> When the JS calls setRemote with the response (which rejects the video), the UA will know that the video has been rejected.  Is the idea that the UA will let the user know via the chrome?  Do we th…

Notifications in Chrome aren't sufficient.  The application needs to know too.

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 2 Jul 2013 20:07:50 +0200

On Tue, 2 Jul 2013 17:20:04 +0000

> When the JS calls setRemote with the response (which rejects the video), the UA will know that the video has been rejected.  Is the idea that the UA will let the user know via the chrome?  Do we th…

I think that success/failure in such a case would be a quite complex thing to notify, mostly because you'd first have to define/understand what you're trying to change: are you just trying to add video? are you changing the port? are you muting one direction? what if something was accepted, and something else not? etc. etc.

Anyway, I agree with you that the UA knows about that. In the case of Alice trying to add video and Bob rejecting it, there may be no callback as of now, but the JS application should already have a way to know the video has been rejected nevertheless. It might just look at the updated MediaStream instance: if nothing changed, video was rejected (unless in case Bob is only receiving video and not sending it, e.g., because it has no webcam, there would be no video track as well?). Not saying it's not ugly, or that a callback isn't needed, but it may work.

Lorenzo

-- 
Lorenzo Miniero, COB

Meetecho s.r.l.
Web Conferencing and Collaboration Tools
http://www.meetecho.com

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 2 Jul 2013 20:11:27 +0200

On Tue, 2 Jul 2013 18:13:48 +0200

> I may be missing something, but doesn't look like rocket science to me...

For the same reasons I just wrote in response to Jim's mail, I'm not sure some kind of a REJECT-UPGRADE action would be enough to address such a scenario, especially in case, for instance, part of the update attempt was accepted and the rest rejected. Considering several different aspects of a session may be subject to change, such a mechanism should be aware of that.

Lorenzo

-- 
Lorenzo Miniero, COB

Meetecho s.r.l.

---

From: José Luis Millán <[email-masked]>
Date: Tue, 2 Jul 2013 20:29:02 +0200

2013/7/2 Martin Thomson <[email-masked]>
> I'm sure that there are controls for this already...  Maybe.

Apart from the statistics API there aren' t, as far as I know. That would
solve the exposed issue.

> You

-- 
José Luis Millán

---

From: Enrico Marocco <[email-masked]>
Date: Tue, 2 Jul 2013 22:11:53 +0200

> rejected. That doesn't really scale.

Come on, Martin, what Enrico suggested was one of a probably unlimited
number of possible approaches for one very specific supposedly
unaddressable use case. It was put together and sent within 9' since the
initial email hit the list -- don't build your case on it!

FWIW I agree that, in more complex scenarios where streams are
frequently added and/or removed, requiring a single O/A for each
operation is going to be a problem. My preference -- a reasonable
tradeoff at this point in time -- would be to limit SDP O/A to
negotiation of ICE params, crypto material and payload types, and do
stream manipulation in an app-specific way. The latter is no-plan, that
incidentally carries my name on it. The former looks pretty much like
plan A.

Enrico

---

From: Roman Shpount <[email-masked]>
Date: Tue, 2 Jul 2013 17:00:22 -0400

> incidentally carries my name on it. The former looks pretty much like
Since ICE needs to be set via API methods in order to support trickle ICE
the only thing that really needs to be in the initial O/A is crypto. Move
this into separate API methods and O/A can be gone.
_____________
Roman Shpount

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 2 Jul 2013 23:16:03 +0200

2013/7/2 Lorenzo Miniero <[email-masked]>:
> Anyway, I agree with you that the UA knows about that. In the case of Alice trying to add video and Bob rejecting it, there may be no callback as of now, but the JS application should already have a way to know the video has been rejected nevertheless. It might just look at the updated MediaStream instance: if nothing changed, video was rejected

That is *really* ugly, don't you think?

--
Iñaki Baz Castillo
<[email-masked]>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 3 Jul 2013 07:30:00 +0000

On 7/2/13 8:05 PM, José Luis Millán wrote:
> In order to avoid the exposed circumstance, where a user has to
> 'suppose' she is sharing her media because there is no way to certainly
> know, yes, the JS application should be able to, at least, check if a
> localMediaStream is being transmitted out of the RTCPeerConnection.

There is a proposal that addresses this:

http://lists.w3.org/Archives/Public/public-webrtc/2013Jan/att-0005/PrioAPI.pdf

(Note that the constraint part would be simplified per the recent 
agreements in the Media Capture TF)

Stefan

> it's own associated (even if logical because of Boundle) transport. In
> that case, such connectivity status would give the information and much
> more control over each media element.
>
>
> 2013/7/2 Jim Barnett <[email-masked]
> <mailto:[email-masked]>>
>
>     When the JS calls setRemote with the response (which rejects the
>     video), the UA will know that the video has been rejected.  Is the
>     idea that the UA will let the user know via the chrome?  Do we think
>     it is the JS applications job to notify the user of success/failure,
>     or the UA's?  I don't recall a discussion of this.
>
>     - Jim
>
>     From: [email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] On
>     Behalf Of Iñaki Baz Castillo
>     Cc: [email-masked] <mailto:[email-masked]>
>     2013/7/2 Enrico Marocco <[email-masked]
>     <mailto:[email-masked]>>:
>     kind of event for the case in which the remote replies a SDP
>     rejecting our new local strem/track.
>
>     --
>     Iñaki Baz Castillo
>     <[email-masked] <mailto:[email-masked]>>
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Stefan Håkansson LK <[email-masked]>
Date: Wed, 3 Jul 2013 07:38:50 +0000

On 7/2/13 6:04 PM, José Luis Millán wrote:
> Alice presses the "Video" button in her web application. A SDP offer
> with video description is sent to Bob:
>

My personal view is that even if we use SDP for carrying the association 
between RTP SSRCs and the MediaStream and MediaStreamTrack structures as 
well as negotiating codecs, we should not use it to negotiate what 
streams and tracks to use. It should only be used to set up the streams 
and tracks over the networks that is decided by the application.

One way in this specific case could be that Bob's application (since Bob 
does not want to see Alice) simply does not show the video. Bob's 
application could in addition signal this to Alice's application (in 
some app proprietary way) and Alice's app could remove the video stream 
(or track) from PeerConnection to save bandwidth and processing.

Stefan

>
> Alice web application received a SDP answer which was consumed
> successfully by the RTCPeerConnection. Alice thinks she is sharing her
> video, but she is not.
>
> The application "Video" button is disabled since the video is already
> being shared (theoretically)
>
> The application "self-Video" HTML5 video element displays Alice's face.
> She thinks that Bob is seeing her face too, but he did indeed rejected so.
>
> I hope this common scenario can be achieved somehow with the current
> API. I can't find how to do it though.
>
> Thanks in advance
>
> --
> José Luis Millán

---

From: Enrico Marocco <[email-masked]>
Date: Wed, 3 Jul 2013 10:23:17 +0200

> offer, since SDP has its own methods to accept or reject the call as

I assume you are talking about browser-to-browser signalling. If it was
a deliberate choice to pick SIP rather than a custom protocol properly
designed for the semantics of you application, then it is a
self-inflicted pain you cannot blame on the API.

If on the other hand you are talking about interworking with SIP
networks, I'm sympathetic with you, as there is no way for your
gatewaying function (regardless of whether it is on an external element
or wired directly into the JS) to avoid SDP parsing and handling of the
42 different ways it defines for signalling stream rejection. My advice
in such case: hang on in there! Eventually your customers will be happy
to pay you good money for the commendable effort.

Enrico

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 3 Jul 2013 10:58:52 +0200

> (or track) from PeerConnection to save bandwidth and processing.

Let me understand it:

So we are forced to use SDP in any WebRTC application but we cannot
use SDP for common features (i.e. detecting that the remote peer has
rejected our new offered stream/track). Thus regardless we use SDP we
cannot do SIP, right? and instead we must use a proprietary signaling
mechanism for those so common features already present in SIP and SDP,
right?

So live with SDP but don't use SDP, am I wrong?

So be free to implement any custom signaling protocol (proprietary
protocol) but not a standard like SIP since SIP's SDP usage cannot be
implemented in WebRTC, right?

Opss, I forgot, I can parse my new generated SDP at JS level (the SDP
with my new video stream/track offer) and then also parse the received
SDP, and detect wheter there is "a=inactive" or "m" line with port 0
to detect that the remote has rejected my stream. Am I really right?

Regards.

--
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Wed, 3 Jul 2013 13:58:54 +0200

2013/7/2 Lorenzo Miniero <[email-masked]>:
> In the case of Alice trying to add video and Bob rejecting it, there may be no callback as of now, but the JS application should already have a way to know the video has been rejected nevertheless. It might just look at the updated MediaStream instance: if nothing changed, video was rejected (unless in case Bob is only receiving video and not sending it, e.g., because it has no webcam, there would be no video track as well?).

One more comment about this:

The fact that Alice offers video to Bob does not mean that Bob also
should offer video to Alice. It could perfectly occur that Alice
offers video to Bob, Bob accepts receiving it but Bob does not send
video to Alice (i.e. "a=recvonly").

The problem described in this issue is the fact that Alice has no way
to realize whether Bob has accepted receiving her video or not. Bob
has replied to the video offer with "m=video 0" or "a=inactive", which
means that there won't be video RTP from Alice to Bob. But there is no
way for Alice's JS to realize of that so Alice's will expect that Bob
is receiving her video (when he is not). There is no API callback to
know whether the remote has accepted our flows or not.

--
Iñaki Baz Castillo
<[email-masked]>

---

From: Lorenzo Miniero <[email-masked]>
Date: Wed, 3 Jul 2013 14:31:58 +0200

Il giorno Wed, 3 Jul 2013 13:58:54 +0200
Iñaki Baz Castillo <[email-masked]> ha scritto:

> > In the case of Alice trying to add video and Bob rejecting it,
> > there may be no callback as of now, but the JS application should
> > already have a way to know the video has been rejected
> > nevertheless. It might just look at the updated MediaStream
> > instance: if nothing changed, video was rejected (unless in case
> > Bob is only receiving video and not sending it, e.g., because it
> > has no webcam, there would be no video track as well?).
> 

Yes, that's what I meant by the "unless...": I'm not sure a remote
MediaStream would look different in case of no video at all or a
recvonly video. If not, I agree that not even the ugly approach I
mentioned would work.

L.