Subject: [rtcweb] Issue with "priority" defined in rtcweb-transports

From: Taylor Brandstetter <[email-masked]>
Date: Wed, 11 Oct 2017 10:29:43 -0700

(Mainly copied from https://github.com/w3c/webrtc-pc/issues/1625)

When defining priority, rtcweb-transports says:

The WebRTC prioritization model is that the application tells the
WebRTC endpoint about the priority of media and data that is
controlled from the API.
...
The priority settings affect two pieces of behavior: Packet send
sequence decisions and packet markings. Each is described in its own
section below.

The sections below are "Local prioritization" and "Usage of Quality of
Service - DSCP and Multiplexing". The "local prioritization" section gives
this guidance:

When an WebRTC endpoint has packets to send on multiple streams that
are congestion-controlled under the same congestion control regime,
the WebRTC endpoint SHOULD cause data to be emitted in such a way
that each stream at each level of priority is being given
approximately twice the transmission capacity (measured in payload
bytes) of the level below.

Thus, when congestion occurs, a "high" priority flow will have the
ability to send 8 times as much data as a "very-low" priority flow if
both have data to send. This prioritization is independent of the
media type. The details of which packet to send first are
implementation defined.

For example: If there is a high priority audio flow sending 100 byte
packets, and a low priority video flow sending 1000 byte packets, and
outgoing capacity exists for sending >5000 payload bytes, it would be
appropriate to send 4000 bytes (40 packets) of audio and 1000 bytes
(one packet) of video as the result of a single pass of sending
decisions.

This has two significant issues I can see:

   - It only allows ratios of 1:2:4:8. This is not granular enough to be
   very useful. Especially when balancing transmission capacity between audio
   tracks and video tracks; in practice it's common for video tracks to use
   much more than 8 times the bitrate of audio tracks.
   - A single enum is used for both relative bitrate and relative QoS
   priority. But in my experience it's common for an application to want a
   flow that uses *less* bitrate to have a *higher* QoS priority. This may
   even be the *more* common situation.

So, why do we have one enum for both things? I'd propose defining the
priority as something that only impacts the priority in network queues
(QoS-level priority and SCTP priority, as previously discussed), and use
something else to control the relative bitrate allocation.

For example, a floating point value attached to each RTCEncodingParameters
 (say, relativeCapacity), where an encoding with twice the relativeCapacity of
another encoding will be given twice the transmission capacity. If
implementations can handle ratios of 1:2:4:8, I don't see any reason why
they shouldn't be able to handle arbitrary ratios.

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 16 Oct 2017 14:10:03 +0200

Hi Taylor,

I think the text you quoted is directly related to your transmission 
queuing. And as no one was willing to write up a specific queuing 
algorithm that would be more flexible, the discussion in the WG at the 
time this was written ended up in a per prioritization level round robin 
queue basically, where each lower priority is given one transmission 
slot in the upper queue. Thus giving roughly halving, but it could be 
less actually depending on how many processes are part of a particular 
level.

I also think you scheme for relative capacity is likely good, but appear 
to be missing two important parameters. One for minimal bit-rate, and 
one for maximum. At least for audio sources, there is a limit to scaling 
these both upwards and downwards. The upper limit when reached should be 
redistributed, or otherwise it might be wasted. The minimal is an 
interesting one. Because one have to decided if this means taking more 
from the lower prioritise and how they relate to other. The classical 
example of one audio plus one video comes to mind, where one would like 
to maintain the audio in user interactive applications despite taking 
more from the video. To the point where one have to turn off video to 
maintain audio.

So, I think you have to work on the proposal a bit more.

Cheers

Magnus

Den 2017-10-11 kl. 19:29, skrev Taylor Brandstetter:
> Service - DSCP and Multiplexing". The "local prioritization" section 
> gives this guidance:
>
>   * It only allows ratios of 1:2:4:8. This is not granular enough to
>     be very useful. Especially when balancing transmission capacity
>     between audio tracks and video tracks; in practice it's common for
>     video tracks to use much more than 8 times the bitrate of audio
>     tracks.
>   * A single enum is used for both relative bitrate and relative QoS
>     priority. But in my experience it's common for an application to
>     want a flow that uses /less/ bitrate to have a /higher/ QoS
>     priority. This may even be the /more/ common situation.
>
> (QoS-level priority and SCTP priority, as previously discussed), and 
> use something else to control the relative bitrate allocation.
>
> For example, a floating point value attached to each 
> |RTCEncodingParameters| (say, |relativeCapacity|), where an encoding 
> with twice the |relativeCapacity| of another encoding will be given 
> twice the transmission capacity. If implementations can handle ratios 
> of 1:2:4:8, I don't see any reason why they shouldn't be able to 
> handle arbitrary ratios.
>
>
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

-- 

Magnus Westerlund

----------------------------------------------------------------------
Media Technologies, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Taylor Brandstetter <[email-masked]>
Date: Mon, 16 Oct 2017 11:28:59 -0700

This is something that came up in the WebRTC virtual interim as well (but
for the simulcast use case, rather than audio vs. video). When bandwidth is
constrained, you don't necessarily want to send multiple simulcast layers,
each with a low bitrate. You want to start by sending only the lowest
quality encoding, until you have enough bits to send the next encoding, etc.

I think we could meet most everyone's use cases with a combination of:

   - Priority
   - Min bitrate
   - Max bitrate
   - "Relative" bitrate ratio

For example, suppose I set up simulcast layers like so:

encodings = [
{
  rid: "f",
  priority: "low",
  minBitrate:  600000,
  maxBitrate: 2500000,
  relativeBitrate: 4.0
},
{
  rid: "h",
  priority: "medium",
  scaleResolutionDownBy: 2.0,
  minBitrate: 150000,
  maxBitrate: 700000,
  relativeBitrate: 2.0
},
{
  rid: "q",
  priority: "high",
  scaleResolutionDownBy: 4.0,
  maxBitrate: 200000,
  relativeBitrate: 1.0
}
]

The algorithm would be "if you can't meet the minimums of every encoding,
prioritize the higher priority encodings." So if only 100kbps is available,
only the "q" (quarter-resolution) encoding would be used. If the available
bandwidth reaches, say, 300kbps, then it would be split between "h" and
"q", with a split of 200kbps/100kbps. And so on.

I believe this is pretty similar to how our hard-coded implementation
already works (I took some of these values directly from the code).

On Mon, Oct 16, 2017 at 5:10 AM, Magnus Westerlund <
[email-masked]> wrote:

> queuing. And as no one was willing to write up a specific queuing algorithm
> that would be more flexible, the discussion in the WG at the time this was
> written ended up in a per prioritization level round robin queue basically,
> where each lower priority is given one transmission slot in the upper
> queue. Thus giving roughly halving, but it could be less actually depending
> on how many processes are part of a particular level.
>
> to be missing two important parameters. One for minimal bit-rate, and one
> for maximum. At least for audio sources, there is a limit to scaling these
> both upwards and downwards. The upper limit when reached should be
> interesting one. Because one have to decided if this means taking more from
> the lower prioritise and how they relate to other. The classical example of
> one audio plus one video comes to mind, where one would like to maintain
> the audio in user interactive applications despite taking more from the
> video. To the point where one have to turn off video to maintain audio.
>
>    flow that uses *less* bitrate to have a *higher* QoS priority. This
>    may even be the *more* common situation.
>
>  (say, relativeCapacity), where an encoding with twice the
> relativeCapacity of another encoding will be given twice the transmission
> capacity. If implementations can handle ratios of 1:2:4:8, I don't see any
> reason why they shouldn't be able to handle arbitrary ratios.
>
>
> rtcweb mailing [email-masked]://www.ietf.org/mailman/listinfo/rtcweb
>
>
> --
>
> Magnus Westerlund
>
> Ericsson AB                 | Phone  +46 10 7148287 <+46%2010%20714%2082%2087>Torshamnsgatan 23 <https://maps.google.com/?q=Torshamnsgatan+23&entry=gmail&source=g>           | Mobile +46 73 0949079 <+46%2073%20094%2090%2079>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Oct 2017 16:45:44 +0200

Den 16. okt. 2017 20:28, skrev Taylor Brandstetter:
> This is something that came up in the WebRTC virtual interim as well
> (but for the simulcast use case, rather than audio vs. video). When
> bandwidth is constrained, you don't necessarily want to send multiple
> simulcast layers, each with a low bitrate. You want to start by sending
> only the lowest quality encoding, until you have enough bits to send the
> next encoding, etc.
> 
>   * Priority
>   * Min bitrate
>   * Max bitrate
>   * "Relative" bitrate ratio

My thinking is that this grows very complex.

I don't think we have much chance of getting this right for all use
cases. For instance, screenshare-type video varies enormously in
bandwidth (very low at static images, enormous at slide change) - if it
competes against audio, a "relative" bitrate ratio will not be useful at
all.

> The algorithm would be "if you can't meet the minimums of every
> encoding, prioritize the higher priority encodings." So if only 100kbps
> is available, only the "q" (quarter-resolution) encoding would be used.
> If the available bandwidth reaches, say, 300kbps, then it would be split
> between "h" and "q", with a split of 200kbps/100kbps. And so on.
> 
> already works (I took some of these values directly from the code
I worry about the complexity of getting this specified. Having code
helps, of course.

I like extension specs. Especially for trying out things where our
solutions may be perfect for some scenarios, but might not work in other
scenarios, and the base specs attempt to cover all cases.

Is there a way we can punt this?

> 
> 
> On Mon, Oct 16, 2017 at 5:10 AM, Magnus Westerlund
> <[email-masked] <mailto:[email-masked]>>
> wrote:
> 
>     Hi Taylor,
> 
>     algorithm that would be more flexible, the discussion in the WG at
>     the time this was written ended up in a per prioritization level
>     round robin queue basically, where each lower priority is given one
>     transmission slot in the upper queue. Thus giving roughly halving,
>     but it could be less actually depending on how many processes are
>     part of a particular level.
> 
>     I also think you scheme for relative capacity is likely good, but
>     appear to be missing two important parameters. One for minimal
>     bit-rate, and one for maximum. At least for audio sources, there is
>     a limit to scaling these both upwards and downwards. The upper limit
>     when reached should be redistributed, or otherwise it might be
>     wasted. The minimal is an interesting one. Because one have to
>     decided if this means taking more from the lower prioritise and how
>     they relate to other. The classical example of one audio plus one
>     video comes to mind, where one would like to maintain the audio in
>     user interactive applications despite taking more from the video. To
>     the point where one have to turn off video to maintain audio.
> 
>>     (Mainly copied from https://github.com/w3c/webrtc-pc/issues/1625
>>     <https://github.com/w3c/webrtc-pc/issues/1625>)
>>
>>     The sections below are "Local prioritization" and "Usage of
>>     Quality of Service - DSCP and Multiplexing". The "local
>>     prioritization" section gives this guidance:
>>
>>       * It only allows ratios of 1:2:4:8. This is not granular enough
>>         to be very useful. Especially when balancing transmission
>>         capacity between audio tracks and video tracks; in practice
>>         it's common for video tracks to use much more than 8 times the
>>         bitrate of audio tracks.
>>       * A single enum is used for both relative bitrate and relative
>>         QoS priority. But in my experience it's common for an
>>         application to want a flow that uses /less/ bitrate to have
>>         a /higher/ QoS priority. This may even be the /more/ common
>>         situation.
>>
>>     So, why do we have one enum for both things? I'd propose defining
>>     the priority as something that only impacts the priority in
>>     network queues (QoS-level priority and SCTP priority, as
>>     previously discussed), and use something else to control the
>>     relative bitrate allocation.
>>
>>     For example, a floating point value attached to
>>     each |RTCEncodingParameters| (say, |relativeCapacity|), where an
>>     encoding with twice the |relativeCapacity| of another encoding
>>     will be given twice the transmission capacity. If implementations
>>     can handle ratios of 1:2:4:8, I don't see any reason why they
>>     shouldn't be able to handle arbitrary ratios.
>>
>>
>>
>>     [email-masked] <mailto:[email-masked]>
>>     <https://www.ietf.org/mailman/listinfo/rtcweb>
> 
>     -- 
> 
>     Magnus Westerlund 
> 
>     Ericsson AB                 | Phone  +46 10 7148287 <tel:+46%2010%20714%2082%2087>
>     Torshamnsgatan 23
>     <https://maps.google.com/?q=Torshamnsgatan+23&entry=gmail&source=g>           | Mobile +46 73 0949079 <tel:+46%2073%20094%2090%2079>
>     SE-164 80 Stockholm, Sweden | mailto: [email-masked] <mailto:[email-masked]>

---

From: Taylor Brandstetter <[email-masked]>
Date: Tue, 17 Oct 2017 09:59:48 -0700

>
> For instance, screenshare-type video varies enormously in

I think that's where it's important to get the terminology right; for
example, rtcweb-transports uses the term "capacity", which implies that
it's possible to use fewer bits than the capacity. Maybe that's the best
way to think of this. We also can allow some implementation flexibility.

If we do punt this, I still think we should remove the part of
rtcweb-transports that talks about the priority causing codecs to be
configured with send rates at these 1:2:4:8 ratios:

   o  When the available bandwidth is known from the congestion control
>       algorithm, configure each codec and each data channel with a
>       target send rate that is appropriate to its share of the available
>       bandwidth.

We should make it clear that the priority only affects the delivery of
packets (pacing decisions, QoS, SCTP priority), and not how the packets are
generated upstream. Assuming others agree.

On Tue, Oct 17, 2017 at 7:45 AM, Harald Alvestrand <[email-masked]>
wrote:

> <tel:+46%2010%20714%2082%2087>
> >     Torshamnsgatan 23
> >     <https://maps.google.com/?q=Torshamnsgatan+23&entry=gmail&source=g>
>          | Mobile +46 73 0949079 <tel:+46%2073%20094%2090%2079>
> <mailto:[email-masked]>

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 17 Oct 2017 22:18:09 +0200

Den 17. okt. 2017 18:59, skrev Taylor Brandstetter:
> packets (pacing decisions, QoS, SCTP priority), and not how the packets
> are generated upstream. Assuming others agree.

The words about configuring the codecs were intended as advice - most
codecs are terrible at dealing with random lost packets (even with FEC
and NACK and all that), so if the pacer is going to throw their packets
away, the best thing to do is to not generate them.

(Hierarchical coding schemes with frames that have no references to them
work fine as long as you throw away complete frames, though. So
sometimes it works.)

This section is already headed with the words "Two example
implementation strategies are:", so it should already be clear that it's
advice, not normative language.

I'm happy to weaken the words even more, but wouldn't want to lose the
advice entirely.

---

From: Taylor Brandstetter <[email-masked]>
Date: Tue, 17 Oct 2017 15:10:01 -0700

I think the issue is that the example leads you to get the wrong idea about
what the priority is intended to effect. Does it affect encoding, or just
transport? Is the advice meant to say "give the codecs a 1:2 split of the
available bandwidth"? Or "give the codecs some split of the available
bandwidth determined elsewhere, then as packets are queued for
transmission, send them with a 1:2 weighted round-robin scheme or something
equivalent"? I interpreted it as the former. It may just be that I'm not
interpreting this as intended, and there really isn't an issue, other than
possibly needing some clarification.

---

From: Bernard Aboba <[email-masked]>
Date: Tue, 17 Oct 2017 16:23:06 -0700

In practice, many enterprise networks do their own policing/re-marking, so
browser-enforced bandwidth limitations may bear no relation to corpnet
policies.  For example, an enterprise may have its own policies on what
traffic should be marked "high priority" and how much bandwidth it should
be allocated.  As a result, the traffic marked by the browser with a
particular DSCP might be re-marked and different bandwidth limits might be
imposed. As a result, traffic that the browser would chose to drop due to
its own limits might have been allowed to pass through, and traffic the
browser would not drop might be dropped later.  In these situations, it
probably makes more sense for the browser just to mark traffic and let the
enterprise network re-mark and impose its own bandwidth limits as necessary.

On Tue, Oct 17, 2017 at 3:10 PM, Taylor Brandstetter <[email-masked]>
wrote:

> I think the issue is that the example leads you to get the wrong idea
> about what the priority is intended to effect. Does it affect encoding, or
> just transport? Is the advice meant to say "give the codecs a 1:2 split of
> the available bandwidth"? Or "give the codecs some split of the available

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 18 Oct 2017 06:32:54 +0200

Throwing away packets at sender is only supposed to happen at all after congestion is detected. The idea is that the sender can do a better job of deciding what to discard than the router can.

Den 18. oktober 2017 01:23:06 CEST, skrev Bernard Aboba <[email-masked]>:
>>> >     bandwidth (very low at static images, enormous at slide
>change) -
>>> if it
>>> >        o  When the available bandwidth is known from the
>congestion
>>> control

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Taylor Brandstetter <[email-masked]>
Date: Wed, 18 Oct 2017 08:53:48 -0700

Bernard, I think you're describing a situation where the flows would (or at
least should) use different congestion controllers, but this guidance is
for "[w]hen an WebRTC endpoint has packets to send on multiple streams that
are congestion-controlled under the same congestion control regime".

Also, a sidenote: I'm realizing now that we probably don't want to use
RTCPriorityType to indicate which simulcast stream should be allocated bits
first. That's still mixing up QoS with something that should be
independent. Could we use the order of the RTCRtpEncodingParameters list?
If their order in the list determines the order of the RIDs in
"a=simulcast", then this already indicates preference. JSEP doesn't
explicitly say how the order is determined, though.

On Tue, Oct 17, 2017 at 9:32 PM, Harald Alvestrand <[email-masked]>
wrote:

> Throwing away packets at sender is only supposed to happen at all after
> congestion is detected. The idea is that the sender can do a better job of
> deciding what to discard than the router can.
>
>
>> In practice, many enterprise networks do their own policing/re-marking,
>> so browser-enforced bandwidth limitations may bear no relation to corpnet
>> On Tue, Oct 17, 2017 at 3:10 PM, Taylor Brandstetter <[email-masked]
>> > wrote:
>>
>>> On Tue, Oct 17, 2017 at 1:18 PM, Harald Alvestrand <[email-masked]
>>> > wrote:
>>>