Subject: [rtcweb] Open issue: Signalling muted streams

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Jan 2012 13:18:17 +0100

This came up in an off-list discussion, but I think it should be on our 
open issues list:

When a sender mutes a stream, what happens at the receiving end?

Either the stream will continue to send data (black/silence), or it doesn't.
If it continues, no problem.
If it doesn't:

Either the recipient knows that the non-data is intentional, or he doesn't.
If he doesn't, we need to make sure nothing times out or gets out of 
whack because of the lack of data.
If he knows it's intentional, he has to learn it somehow.

Either the information comes over the media path, or over the signalling 
path.
If it's the media path, there needs to be an RTP-level definition of 
what's sent to indicate it. (RTCP message, comfort noise / video 
equivalent, other...)
If it's the signalling path, there needs to be a defined API to get the 
information passed up to the JS layer at the sender, and passed down 
from the JS layer at the recipient.

Thoughts from the community?

                        Harald

---

From: Stefan Hakansson LK <[email-masked]>
Date: Tue, 17 Jan 2012 14:12:48 +0100

> Thoughts from the community?

Off the top of my head:

1. I think we should do this in the media plane (RTP/RTCP)
2. Initially, the browser could interpret the absence of data (RTP 
and/or RTCP reports) to determine if the other end has stopped sending, 
and then change the state of the corresponding MediaStreamTrack(s) to 
"muted". This is more or less the same way as when adding a new 
MediaStream to PeerConnection: the MediaStream object at the receiving 
PeerConnection is created immediately with all MediaStreamTracks in 
muted state, once RTP is coming in the state is changed to not muted.
3. Perhaps special RTP or RTCP packets can be added later to indicate 
"muted" later - I suspect this will involve other WGs so it might take 
some time.

Stefan

>
>                          Harald
>
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Wolfgang Beck <[email-masked]>
Date: Tue, 17 Jan 2012 14:38:17 +0100

On Tue, Jan 17, 2012 at 1:18 PM, Harald Alvestrand <[email-masked]> wrote:
> This came up in an off-list discussion, but I think it should be on our open
> issues list:
>

It's the signalling discussion again.

If you tunnel SDP through HTTP and some some server-to-server
protocol, you just use the SDP mechanism to mute one side.

Without server-to-server protocol, you only need an API call to pause
sending of RTP packets. How this is signaled to the remote party is up
the JS client and server.

Moving parts of the SDP functionality to RTP would break compatibility
with existing systems. Which was the main argument for sticking with
SDP..

Wolfgang Beck

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Jan 2012 14:51:53 +0100

On 01/17/2012 02:38 PM, Wolfgang Beck wrote:
> On Tue, Jan 17, 2012 at 1:18 PM, Harald Alvestrand<[email-masked]>  wrote:
Which SDP mechanism are you recommending? (RFC and section)
> with existing systems. Which was the main argument for sticking with
So your answer is "API, and use an SDP mechanism". Thanks for the clarity!

---

From: "Ravindran, Parthasarathi" <[email-masked]>
Date: Tue, 17 Jan 2012 14:28:45 +0000

Harald,

Hope the point of discussion is "Signaling Hold" and it will be solved by SDP mechanism (by inactive direction attribute) as you mentioned below.

AFAIK, Mute operation normally does not involve any signaling and stop sending the media from the endpoint (browser in WebRTC). In case mute operation is required in WebRTC, ICE shall solve keep-alive mechanism in the media path and mute API is required for stop transmitting the media from the local browser.

Thanks
Partha

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Harald Alvestrand
>Sent: Tuesday, January 17, 2012 7:22 PM
>To: Wolfgang Beck
>Cc: [email-masked]
>Subject: Re: [rtcweb] Open issue: Signalling muted streams
>
>>> This came up in an off-list discussion, but I think it should be on
>>> our open issues list:
>>>

---

From: Varun Singh <[email-masked]>
Date: Tue, 17 Jan 2012 16:44:19 +0200

Hi,

Comments inline.

On Tue, Jan 17, 2012 at 14:18, Harald Alvestrand <[email-masked]> wrote:
> If he doesn't, we need to make sure nothing times out or gets out of whack
> because of the lack of data.
> If it's the media path, there needs to be an RTP-level definition of what's
> sent to indicate it. (RTCP message, comfort noise / video equivalent,
> other...)

The sender will send RTCP sender reports to keep the session alive and if the
"sender's packet count" does not change between two consecutive reports the
receiver can guess that the stream is muted. Caveat is that the RTCP reports
are not sent as often as RTP, so a detection based on RTCP SRs will
take an RTCP Interval.

> information passed up to the JS layer at the sender, and passed down from
> the JS layer at the recipient.
>

Defining an API may be more appropriate.

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Roman Shpount <[email-masked]>
Date: Tue, 17 Jan 2012 10:16:58 -0500

Can CN packets be used to indicate muted stream on the media plane?
_____________
Roman Shpount

On Tue, Jan 17, 2012 at 8:51 AM, Harald Alvestrand <[email-masked]>wrote:

>> On Tue, Jan 17, 2012 at 1:18 PM, Harald Alvestrand<harald@alvestrand.**no<[email-masked]>>
>>  wrote:
>>
>>> open
> ______________________________**_________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/rtcweb<https://www.ietf.org/mailman/listinfo/rtcweb>
>

---

From: Randell Jesup <[email-masked]>
Date: Tue, 17 Jan 2012 10:23:14 -0500

>> When a sender mutes a stream, what happens at the receiving end?

I'll note that it's critical that muting of audio and video occur 
immediately, and not require any handshake with the recipient.  This can 
be black/silence, though in practice I've found that black video is 
disconcerting to recipients - a visual indication of Mute is much better.

In the past I've used either a textual message, or more recently a 
symbol of a camera with a red circle & slash.

>> If it continues, no problem.

In interop scenarios, it's always unclear how a receiver will react to 
stopping of media.  Some will end a call after 10-30 seconds of no media 
(though hopefully RTCP reports would preclude that; they did in my 
implementations).

>> whack because of the lack of data.

Right.

>> what's sent to indicate it. (RTCP message, comfort noise / video

The original Ojo used a bit in an RTCP APP status message sent when the 
other side muted (obviously only worked between two of our phones).  We 
resent it several times at increasing delays to avoid packet loss 
issues, a bit like RFC 2833.

* You could define a new RTCP message
* You could define a 'status' payload ala DTMF in RTP
* You could mandate that media stoppage occurs via CN, and then only if 
the other side agreed (audio only)
* For video you could mandate sending a black IDR periodically (video 
rough equivalent of comfort noise).  Or sending a "Mute" image by IDR 
periodically.

(And of course your could continue to send silent/black/mute-image).

There is an advantage in conferencing (and some other) scenarios to 
recover the bandwidth during mute.

> muted state, once RTP is coming in the state is changed to not muted.

Mute->non-mute is easier (though note that 'muted' reception tracks is 
not the same as 'muted' at the far end, and so shouldn't be used for UI 
purposes to indicate mute).

Non-Mute->mute as described could also be triggered by a network 
interruption.  If the delay is longer, it would be slow at triggering.

> "muted" later - I suspect this will involve other WGs so it might take

See above.

I favor silence for audio, and a mute image periodically sent by IDR. 
If we can agree on signalling mute, I would want it in the media plane. 
  I'm open as to RTCP vs RTP but prefer RTP; RTP is faster and there 
would be explicit up-front negotiation as to whether the other side 
supports it (and if not, you just continue to send media as normal 
silence and a mute image or black, which should reduce bandwidth).

-- 
Randell Jesup
[email-masked]

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 17 Jan 2012 07:26:00 -0800

Hi Harald,

On 1.17.2012 04:18 , "Harald Alvestrand" <[email-masked]> wrote:

>[Š]
>
>Either the information comes over the media path, or over the signaling
>path.

This is not necessarily an "either, or" (exclusive or) decision.  For
example, the ITU video conferencing system Recs since H.320 (and the
associated video codecs) include a mechanism that allows for picture-exact
video mute (in the form of a Supplementary Enhancement Information (SEI)
message inside the video bitstream, and a video unmute in the signaling
protocol.  

Historically, this design choice was motivated by video switching MCUs and
video encoders incapable of coding full intra pictures (using intra walk
around in inter pictures).  However, the ability of sending a mute right
in the media path, and thereby synchronized with the bitstream and not
subject to delays that may or may not occur on the signaling path, has
been shown to be very useful outside these limited scenarios.  The case
for precise synchronization of an unmute with the media is not quite as
strong, but has been made as well.

OTOH, media is susceptible to losses, and a lost "mute" can lead to rather
embarrassing (or worse) situations.

Therefore, I would suggest to add a mute/unmute capability, per media
path, to the media plane related "signaling" (for example using the
RTP/RTCP related standards you mentioned--I'm not pushing for those SEI
messages).  If the designer wishes to put the same signaling also on the
signaling path, well and good.  AFAICT, it's relatively simple to deal
with conflicting information over both paths in a receiver implementation.
 Traditional video conferencing systems have dealt with this for decades.

Stephan

---

From: Wolfgang Beck <[email-masked]>
Date: Tue, 17 Jan 2012 17:59:44 +0100

> Which SDP mechanism are you recommending? (RFC and section)
RfC 3264 section 8.4 recommends a=sendonly, which turned out to be
problematic if you need to distinguish between 'I will send RTP but
ignore your RTP' and 'I won't send RTP and ignore your RTP'.

> So your answer is "API, and use an SDP mechanism". Thanks for the clarity!
If we want to go the SDP over ROAP path, yes.

If we had an API which was loosely modeled after SDP, we wouldn't need
SDP at all. We've discussed that.
The only good reason to use SDP is compatibility with existing implementations.

 Wolfgang Beck

---

From: Roman Shpount <[email-masked]>
Date: Tue, 17 Jan 2012 12:13:13 -0500

On Tue, Jan 17, 2012 at 11:59 AM, Wolfgang Beck <
[email-masked]> wrote:

> This section was actually a fairly bad decision on behalf of RFC authors.
It is worded this way since a=sendonly corresponds to using 0.0.0.0 address
in c line, but should almost never be used in normal call flows. For hold
you typically use a=inactive. So use a=sendonly if you are planning to send
RTP, but don't want to receive any media (reverse mute), a=recvonly if you
put a line on mute, but still want to receive RTP, and a=inactive if you
put a line on hold and plan to do neither.
_____________
Roman Shpount

---

From: Randell Jesup <[email-masked]>
Date: Tue, 17 Jan 2012 12:17:44 -0500

On 1/17/2012 11:59 AM, Wolfgang Beck wrote:
> On Tue, Jan 17, 2012 at 2:51 PM, Harald Alvestrand<[email-masked]>  wrote:

Please note, because people are often sloppy about this:

Mute != Hold

They are very closely related, but there are two primary differences:

1) In Mute, you expect to continue to receive *and* play audio and/or 
video from the other side.  In Hold, any incoming media is thrown away.
2) Mute often may affect only audio or only video, though it could 
affect both/all (this is a UI decision).  Hold would affect all streams.

Hold is what was being discussed in RFC 3264 8.4, and they suggested 
re-INVITE with a=sendonly PLUS muting local audio immediately and 
blocking playback immediately.

Hold often implies replacing audio and/or video with some sort of 
"you're on hold" message (audio and/or video), which also tells the 
far-end party that they don't have to worry (much) about talking.

-- 
Randell Jesup
[email-masked]

---

From: Igor Faynberg <[email-masked]>
Date: Tue, 17 Jan 2012 12:29:55 -0500

At the first glance, it appears to me that sending "muted data" is 
wasteful. I don't fully understand why sending nothing would confuse the 
recipient (or, more important, the recipient's system), but to deal with 
it I think sending a "muted" bit through signaling is the cleanest 
solution.

Wolfgang's suggestion of using SDP is doable I think.

Igor

On 1/17/2012 7:18 AM, Harald Alvestrand wrote:
> Either the information comes over the media path, or over the 
> signalling path.
> If it's the signalling path, there needs to be a defined API to get 
> the information passed up to the JS layer at the sender, and passed 
> down from the JS layer at the recipient.
>

---

From: Aron Rosenberg <[email-masked]>
Date: Tue, 17 Jan 2012 11:27:44 -0800

For "muting" of outbound video, the best real-world interop method that we
have found is to just have your encoder constantly reencode the last frame
of video prior to the user requesting mute. This method keeps the media
going, doesn't trip far-end timeout logic and has the nice side effect that
the image quality improves on the frozen frame and it works with any video
codec (VP8 doesn't support SEI). Once steady state is hit, the bitrate used
while in mute state is practically zero.

On the audio side, we do something similar where we degrade/convert the
audio signal to CN over one or two 20ms audio frames (so no popping
or squeaks occur). After that initial period, CN is passed to the audio
codec to keep compressing. This also works with any audio codec.

Both methods still allow for signaling if you decide to let the far end
know about the mute. On the signaling side, I have seen RTCP based methods
and SIP INFO based methods. RTCP is nice because it has a better
association with a particular stream.

For video, our testing showed that freezing the last image is far better
than sending a black/gray/other color frame.

Aron Rosenberg
Sr. Director, Engineering,
LifeSize, a division of Logitech

On Tue, Jan 17, 2012 at 9:29 AM, Igor Faynberg <
[email-masked]> wrote:

> recipient (or, more important, the recipient's system), but to deal with it
> I think sending a "muted" bit through signaling is the cleanest solution.
>

---

From: Venkatesh <[email-masked]>
Date: Tue, 17 Jan 2012 14:50:25 -0800

Section 8.4 of RFC 3264 talks about SDP usage to indicate streams on hold.
Although this is in the context of SIP, the mechanism is probably
applicable in this context.

Thanks,
Venkatesh

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Jan 2012 10:07:08 +0100

On 01/17/2012 06:13 PM, Roman Shpount wrote:
>
>
> On Tue, Jan 17, 2012 at 11:59 AM, Wolfgang Beck 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> This section was actually a fairly bad decision on behalf of RFC 
> authors. It is worded this way since a=sendonly corresponds to using 
> 0.0.0.0 address in c line, but should almost never be used in normal 
> call flows. For hold you typically use a=inactive. So use a=sendonly 
> if you are planning to send RTP, but don't want to receive any media 
> (reverse mute), a=recvonly if you put a line on mute, but still want 
> to receive RTP, and a=inactive if you put a line on hold and plan to 
> do neither.

An interesting wrinkle is that RFC 3264 section 8.4 seems to assume that 
the dominant use case of mute is the recipient wanting to not receive 
media (so it needs to be receiver-initiated), while the most-discussed 
use case in this group has been a sender turning off his microphone or 
camera (so it needs to be sender-initiated).

Is a=sendonly / a=recvonly / a=inactive manipulation appropriate for 
both cases?

                Harald

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Jan 2012 10:08:48 +0100

> to receive RTP, and a=inactive if you put a line on hold and plan to 
Oops.... I suddenly realized:

These mechanisms (a=sendonly and so on) are at the RTP session level. 
They apply to all SSRCs in the RTP session.

The mute functions we've been discussing at the API level are at the 
MediaStreamTrack level. That maps to wanting to mute a single SSRC.

How does that affect things?

                Harald

---

From: Roman Shpount <[email-masked]>
Date: Wed, 18 Jan 2012 12:42:03 -0500

On Wed, Jan 18, 2012 at 4:07 AM, Harald Alvestrand <[email-masked]>wrote:

> **
> the dominant use case of mute is the recipient wanting to not receive media
> (so it needs to be receiver-initiated), while the most-discussed use case
> in this group has been a sender turning off his microphone or camera (so it
> needs to be sender-initiated).
>
> Is a=sendonly / a=recvonly / a=inactive manipulation appropriate for both
> cases?
>
>
It is appropriate in all three cases:

   - Hold with Music: pausing remote media and playing some sort of hold
   music (or hold signal) to remote party
   - Mute: muting local audio and stopping camera, but still receiving
   remote media
   - Hold: pausing both sent and received media

_____________
Roman Shpount

---

From: Roman Shpount <[email-masked]>
Date: Wed, 18 Jan 2012 12:45:02 -0500

On Wed, Jan 18, 2012 at 4:08 AM, Harald Alvestrand <[email-masked]>wrote:

> These mechanisms (a=sendonly and so on) are at the RTP session level. They
> apply to all SSRCs in the RTP session.
>
a=sendonly / a=recvonly / a=inactive attributes are per "m" line in SDP, so
each media line (audio, video, etc) can be put on hold or muted separately.

P.S. SDP in general does not work with SSRC, it operates with source and
destination ports. Technically speaking, SSRC for the same
source/destination pair can change during the call.
_____________
Roman Shpount

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Jan 2012 18:58:21 +0100

On 01/18/2012 06:45 PM, Roman Shpount wrote:
>
> On Wed, Jan 18, 2012 at 4:08 AM, Harald Alvestrand 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     These mechanisms (a=sendonly and so on) are at the RTP session
>     level. They apply to all SSRCs in the RTP session.
>
>     The mute functions we've been discussing at the API level are at
>     the MediaStreamTrack level. That maps to wanting to mute a single
>     SSRC.
>
> a=sendonly / a=recvonly / a=inactive attributes are per "m" line in 
> SDP, so each media line (audio, video, etc) can be put on hold or 
> muted separately.

I don't think you understood my question.

If we have one SDP session, consisting of two RTP sessions (one for 
audio and one for video), and each RTP session is carrying two tracks 
(one for participant A and one for participant B), and we want to mute 
the audio of participant A, but do not want to mute participant B, how 
should we signal it?

>
> P.S. SDP in general does not work with SSRC, it operates with source 
> and destination ports. Technically speaking, SSRC for the same 

Yes, that's why RFC 5576 specified the "previous-ssrc" attribute.

---

From: Roman Shpount <[email-masked]>
Date: Wed, 18 Jan 2012 13:04:09 -0500

On Wed, Jan 18, 2012 at 12:58 PM, Harald Alvestrand <[email-masked]>wrote:

> If we have one SDP session, consisting of two RTP sessions (one for audio
> and one for video), and each RTP session is carrying two tracks (one for
> participant A and one for participant B), and we want to mute the audio of
> participant A, but do not want to mute participant B, how should we signal
> it?
>

I am not sure what you mean here. Are you saying that an end point is
getting media from two different remote parties on the same IP and port
from the same remote IP and port? Is this due to remote mixer or due to
multicast?
_____________
Roman Shpount

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Jan 2012 19:13:23 +0100

On 01/18/2012 07:04 PM, Roman Shpount wrote:
>
> On Wed, Jan 18, 2012 at 12:58 PM, Harald Alvestrand 
>     If we have one SDP session, consisting of two RTP sessions (one
>     for audio and one for video), and each RTP session is carrying two
>     tracks (one for participant A and one for participant B), and we
>     want to mute the audio of participant A, but do not want to mute
>     participant B, how should we signal it?
>
>
> getting media from two different remote parties on the same IP and 
> port from the same remote IP and port? Is this due to remote mixer or 
> due to multicast?
It's a permitted configuration for a PeerConnection. The discussions 
around the need to minimize the number of separate ICE setups has led it 
to be a preferred one.

Use cases include multiple cameras (section 4.2.9 in the requirements 
doc) and central-server video conferencing (section 4.2.10).

             Harald

---

From: Roman Shpount <[email-masked]>
Date: Wed, 18 Jan 2012 13:35:47 -0500

On Wed, Jan 18, 2012 at 1:13 PM, Harald Alvestrand <[email-masked]>wrote:

> around the need to minimize the number of separate ICE setups has led it to
> be a preferred one.
>
> Use cases include multiple cameras (section 4.2.9 in the requirements doc)
> and central-server video conferencing (section 4.2.10).
>
>
I was under the impression that we planned to reuse the same ip/port with
multiple m=lines and different ssrc attributes in SDP for this. In this
case end point can refuse each line, negotiate codecs, and support
direction attributes for each line separately. It is often needed for both
of those use cases anyway, since multiple cameras (or video conference
participants) can use different codecs, video resolutions and such. Also,
in case of central mixing server, you typically signal the mixing server to
put the individual participant on hold via some sort of side channel.
_____________
Roman Shpount

---

From: "Roni Even" <[email-masked]>
Date: Wed, 18 Jan 2012 21:53:14 +0200

Hi,
If the sender stops sending data but keeps the RTP session it will still
send RTCP sender reports allowing the receiver to know that it temporary not
sending data.
If the sender wants to stop the RTP session he should send an RTCP BYE to
close the channel. 
If receiver do not get data, RTCP sender reports or RTCP BYE he should time
out and view the RTP session as closed
Roni Even

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Harald Alvestrand
> Sent: Tuesday, January 17, 2012 2:18 PM
> To: [email-masked]
> Subject: [rtcweb] Open issue: Signalling muted streams
> 
> equivalent, other...) If it's the signalling path, there needs to be a
> defined API to get the information passed up to the JS layer at the
> sender, and passed down from the JS layer at the recipient.
>

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Jan 2012 21:04:43 +0100

On 01/18/2012 07:35 PM, Roman Shpount wrote:
>
> On Wed, Jan 18, 2012 at 1:13 PM, Harald Alvestrand 
>     It's a permitted configuration for a PeerConnection. The
>     discussions around the need to minimize the number of separate ICE
>     setups has led it to be a preferred one.
>
>     Use cases include multiple cameras (section 4.2.9 in the
>     requirements doc) and central-server video conferencing (section
>     4.2.10).
>
>
> I was under the impression that we planned to reuse the same ip/port 
> with multiple m=lines and different ssrc attributes in SDP for this. 
> In this case end point can refuse each line, negotiate codecs, and 
> support direction attributes for each line separately. It is often 
> needed for both of those use cases anyway, since multiple cameras (or 
> video conference participants) can use different codecs, video 
> resolutions and such.
That wasn't my impression. The ultimate goal of the "reuse" discussions, 
as I see it, is to send all the audio and all the video over a single 
IP/port pair. If we can't get reuse, we'll revert to doing one RTP 
session for all the audio and one RTP session for all the video.

> Also, in case of central mixing server, you typically signal the 
> mixing server to put the individual participant on hold via some sort 
> of side channel.
And how does the mixing server tell the end-user that a channel has been 
put on hold?

---

From: Colin Perkins <[email-masked]>
Date: Wed, 18 Jan 2012 22:11:53 +0000

On 17 Jan 2012, at 12:18, Harald Alvestrand wrote:
> This came up in an off-list discussion, but I think it should be on our open issues list:
> 
> If he doesn't, we need to make sure nothing times out or gets out of whack because of the lack of data.
> Either the information comes over the media path, or over the signalling path.
> If it's the media path, there needs to be an RTP-level definition of what's sent to indicate it. (RTCP message, comfort noise / video equivalent, other…)

RTCP SR packets include a "sender's packet count" field. A sender which stops transmitting RTP data packets will keep sending RTCP packets, and the receiver can tell from the non-increasing sender packet count that the sender stopped transmitting, rather than the path losing the packets. This takes a couple of reports before the receiver is sure, of course, so can be relatively slow (although using AVPF to send an SR just after the last video packet, coupled with a reasonably short reporting interval to get a second SR, should inform the receiver reasonably quickly, and we'll need a reasonably short reporting interval to make the congestion control stable most likely).

Colin

> If it's the signalling path, there needs to be a defined API to get the information passed up to the JS layer at the sender, and passed down from the JS layer at the recipient.
> 

-- 
Colin Perkins
http://csperkins.org/

---

From: Christer Holmberg <[email-masked]>
Date: Thu, 19 Jan 2012 07:23:05 +0100

Hi Harald,

>> I was under the impression that we planned to reuse the same ip/port with multiple m=lines and 
>> different ssrc attributes in SDP for this. In this case end point can refuse each line, negotiate 
>> codecs, and support direction attributes for each line separately. It is often needed for both 
>> of those use cases anyway, since multiple cameras (or video conference participants) can use 
>> different codecs, video resolutions and such.
>
> That wasn't my impression. The ultimate goal of the "reuse" discussions, as I see it, is to send all 
> the audio and all the video over a single IP/port pair. If we can't get reuse, we'll revert to doing 
> one RTP session for all the audio and one RTP session for all the video.

As you know, BUNDLE allows the sending of all the audio and all the video over a single IP/port pair, still using multiple m= lines :)

I guess the question here is whether a single m= line could contain multiple media tracks, and you want to be able to control them individualy, in which case you could not use direction attributes etc which affect the whole m= line.

Regards,

Christer

---

From: Roman Shpount <[email-masked]>
Date: Thu, 19 Jan 2012 11:05:23 -0500

On Thu, Jan 19, 2012 at 1:23 AM, Christer Holmberg <
[email-masked]> wrote:

>
> I guess the question here is whether a single m= line could contain
> multiple media tracks, and you want to be able to control them individualy,
> in which case you could not use direction attributes etc which affect the
> whole m= line.
>
>
Using multiple media tracks with a single m line is tricky. For instance it
would require dynamically to create and remove windows for each remote
video participant. And you are obviously loosing a lot of control over
individual participants such as direction attributes or codec settings.

Out of curiosity, would current WebRTC implementation even support multiple
simultaneous sources over the same m=line for video?
_____________
Roman Shpount

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 19 Jan 2012 17:13:33 +0100

On 01/19/2012 05:05 PM, Roman Shpount wrote:
>
> On Thu, Jan 19, 2012 at 1:23 AM, Christer Holmberg 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     I guess the question here is whether a single m= line could
>     contain multiple media tracks, and you want to be able to control
>     them individualy, in which case you could not use direction
>     attributes etc which affect the whole m= line.
>
>
> Using multiple media tracks with a single m line is tricky. For 
> instance it would require dynamically to create and remove windows for 
> each remote video participant. And you are obviously loosing a lot of 
> control over individual participants such as direction attributes or 
> codec settings.
>
> Out of curiosity, would current WebRTC implementation even support 
> multiple simultaneous sources over the same m=line for video?
I can only speak to the one WebRTC implementation I know about (Chrome), 
and it only supports one video stream in one PeerConnection. But that's 
a temporary limitation.

We intend to support multiple simultaneous sources over the same m= 
line; in fact we have no intention of ever generating multiple m= lines 
for video sources - we see no need to.

However, that requires some work on signaling the association between 
media carried on SSRCs and media presented as MediaStreamTracks inside 
MediaStreams at the API; that's what draft-alvestrand-rtcweb-msid, which 
I mentioned on this list a week ago, is all about.

                   Harald

---

From: Roman Shpount <[email-masked]>
Date: Thu, 19 Jan 2012 11:29:36 -0500

On Thu, Jan 19, 2012 at 11:13 AM, Harald Alvestrand <[email-masked]>wrote:

> **
> We intend to support multiple simultaneous sources over the same m= line;
> in fact we have no intention of ever generating multiple m= lines for video
> sources - we see no need to.
>
> MediaStreams at the API; that's what draft-alvestrand-rtcweb-msid, which I
> mentioned on this list a week ago, is all about.
>
> I guess if we are changing the way things normally work, we can put
together a draft that defines directional attributes as an source level
attributes.

Something like:
a=ssrc:12345 sendonly
a=ssrc:12345 recvonly
a=ssrc:12345 inactive

BTW, have anybody considered the implications of carrying multiple audio or
video sources over the same m= line on interop? This would probably require
a very non-trivial signaling logic in order to interop with anything that
does not support this, since adding a media source in this case will not
require a signaling transaction. Because of this, something like an SBC
will need to monitor RTP and then issue signaling transactions to add or
remove additional m-lines.
_____________
Roman Shpount

---

From: Venkatesh <[email-masked]>
Date: Thu, 19 Jan 2012 08:52:15 -0800

Harold:
If you are going to be signaling the SSRC's in the SDP rather than using
multiple m lines, may be, we could extend it to be able to
provide directional attributes on a per SSRC basis?

Venkatesh

On Thu, Jan 19, 2012 at 8:13 AM, Harald Alvestrand <[email-masked]>wrote:

> Using multiple media tracks with a single m line is tricky. For instance
> it would require dynamically to create and remove windows for each remote
> and it only supports one video stream in one PeerConnection. But that's a
> temporary limitation.
>

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 19 Jan 2012 18:43:24 +0100

On 01/19/2012 05:29 PM, Roman Shpount wrote:
> On Thu, Jan 19, 2012 at 11:13 AM, Harald Alvestrand 
>     We intend to support multiple simultaneous sources over the same
>     m= line; in fact we have no intention of ever generating multiple
>     m= lines for video sources - we see no need to.
>
>     However, that requires some work on signaling the association
>     between media carried on SSRCs and media presented as
>     MediaStreamTracks inside MediaStreams at the API; that's what
>     draft-alvestrand-rtcweb-msid, which I mentioned on this list a
>     week ago, is all about.
>
> together a draft that defines directional attributes as an source 
> level attributes.
>
> Something like:
> a=ssrc:12345 sendonly
> a=ssrc:12345 recvonly
> a=ssrc:12345 inactive
>
> BTW, have anybody considered the implications of carrying multiple 
> audio or video sources over the same m= line on interop? This would 
> probably require a very non-trivial signaling logic in order to 
> interop with anything that does not support this, since adding a media 
> source in this case will not require a signaling transaction. Because 
> of this, something like an SBC will need to monitor RTP and then issue 
> signaling transactions to add or remove additional m-lines.
See draft-westerlund-avtcore-max-ssrc-00 for one discussion of this.

FWIW, Google Hangouts works just fine with a large number of SSRCs per 
RTP session.

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 20 Jan 2012 09:15:52 +0100

On 01/19/2012 05:52 PM, Venkatesh wrote:
> Harold:
> If you are going to be signaling the SSRC's in the SDP rather than 
> using multiple m lines, may be, we could extend it to be able to 
Since SSRCs are defined to be one-way, "on" or "off" should be enough.
> Venkatesh
> On Thu, Jan 19, 2012 at 8:13 AM, Harald Alvestrand 
>>         contain multiple media tracks, and you want to be able to
>>         control them individualy, in which case you could not use
>>         direction attributes etc which affect the whole m= line.
>>
>>
>>     instance it would require dynamically to create and remove
>>     windows for each remote video participant. And you are obviously
>>     loosing a lot of control over individual participants such as
>>     direction attributes or codec settings.
>>
>>     Out of curiosity, would current WebRTC implementation even
>>     support multiple simultaneous sources over the same m=line for
>>     video?
>     I can only speak to the one WebRTC implementation I know about
>     (Chrome), and it only supports one video stream in one
>     PeerConnection. But that's a temporary limitation.
>
>     [email-masked] <mailto:[email-masked]>

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 20 Jan 2012 14:24:28 +0100

On 2012-01-18 23:11, Colin Perkins wrote:
> 
> RTCP SR packets include a "sender's packet count" field. A sender
> which stops transmitting RTP data packets will keep sending RTCP
> packets, and the receiver can tell from the non-increasing sender
> packet count that the sender stopped transmitting, rather than the
> path losing the packets. This takes a couple of reports before the
> receiver is sure, of course, so can be relatively slow (although
> using AVPF to send an SR just after the last video packet, coupled
> with a reasonably short reporting interval to get a second SR, should
> inform the receiver reasonably quickly, and we'll need a reasonably
> short reporting interval to make the congestion control stable most
> likely).
> 

As Indidivual,

Fully agree with this as a basic approach to indicating the
(temporarily) end of transmission indication. I think this actually are
a common problem that hasn't been dealt with in the RTP level yet. In
RTSP 2.0 there is a Server to client signaling message for End_of_Stream
indication. This tells you that the last RTP packet for this stream is
the following RTP sequence number. That way one also avoids the issues
with packet retransmission and other transport mechanism from reacting
on the lack of packets.

>From my perspective I think there are the following categories of solutions.

1. Let the Web APP deal with telling the other end that it happened.
(may require API functions)
2. Use the signaling channel between the browser instances to indicate it.
3. Use media plane signaling

I think it is also important to remember which use case we are talking
about.
A. Telling the receiver that the sender stopped sending intentionally
B. Have the receiver tell the sender that it doesn't need a particular
media stream.

For B3, I presented a potential solution in Taipei's AVTEXT meeting
https://datatracker.ietf.org/doc/draft-westerlund-avtext-rtp-stream-pause/

That got the interesting feedback that TMMBR=0 (RFC 5104) is used in
video conferencing (not fully ok according to normative statements and
semantically). There is clearly other cases where signalling plane
solutions are argued for.

But, in the WebRTC context doing a B1 solution may be the simplest.

I think A3 should be pursued because I certain that WebRTC is not the
only application that would like an indication that one stopped sending.
And that an indication following the media plane so all middleboxes that
can access becomes aware of it. But, this is clearly work for AVTCORE or
AVTEXT.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------