Subject: [rtcweb] -transports- discussion of DSCP

From: Ted Hardie <[email-masked]>
Date: Wed, 13 Aug 2014 09:10:27 -0700

For a while now the draft has said:

   For UDP, this specification assumes the ability to set the DSCP code
   point of the sockets opened on a per-packet basis, in order to
   achieve the prioritizations described in [I-D.ietf-tsvwg-rtcweb-qos]
   (see Section 4.1) when multiple media types are multiplexed.  It does
   not assume that the DSCP codepoints will be honored, and does assume
   that they may be zeroed or changed, since this is a local
   configuration issue.

   Platforms that do not give access to these interfaces will not be
   able to support a conforming WebRTC implementation.

If a platform does not have this capability or does not honor the request
to use it (since it may be limited to privileged users), it seems possible
that an alternate approach is for the WebRTC implementation to suspend
cross-media type multiplexing when it is not available.  There's a cost to
that, especially for IPv4, but that still seems to support a conforming use
of WebRTC.

Would the working group support describing that alternative as conforming?

My concern here is that access to the ability to set different DSCP
markings on a per packet basis for a flow may not be available generally on
all platforms.  Even if we want to make a forward looking push toward that
ability, naming what we think is the appropriate alternative may make sense.

regards,

Ted (as an individual)

---

From: Martin Thomson <[email-masked]>
Date: Wed, 13 Aug 2014 12:07:20 -0700

On 13 August 2014 09:10, Ted Hardie <[email-masked]> wrote:
> My concern here is that access to the ability to set different DSCP markings
> on a per packet basis for a flow may not be available generally on all
> platforms.  Even if we want to make a forward looking push toward that

Yeah, requiring per-packet DSCP seems overly enthusiastic.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 14 Aug 2014 10:18:45 +0200

> Yeah, requiring per-packet DSCP seems overly enthusiastic.

I know it's possible to do this on Linux (for UDP, of course). I think 
it still requires an extra sysctl before sending each packet, which is 
reasonably high overhead if your app is sysctl-limited. But I don't know 
of any case where you would be allowed to set per-flow DSCP markings and 
would not be allowed to set per-packet DSCP markings; all the reasons to 
not do marking seem to be reasons to not do marking.

I'd like to know if the APIs exist on other platforms (whether they're 
overridden by default policy or not).

The two alternatives if we don't get per-packet setting are:

- Deaggregate, so that per-port-flow classifiers can do their thing
- Don't ask the network to differentiate, keep precedence of flow a 
sender-stack thing only

I don't know which is the more (or less) distasteful / useful of those 
two paths.

>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Justin Uberti <[email-masked]>
Date: Thu, 14 Aug 2014 11:17:36 -0700

On Thu, Aug 14, 2014 at 1:18 AM, Harald Alvestrand <[email-masked]>
wrote:

>>> markings
> I know it's possible to do this on Linux (for UDP, of course). I think it
> still requires an extra sysctl before sending each packet, which is
> reasonably high overhead if your app is sysctl-limited. But I don't know of
> any case where you would be allowed to set per-flow DSCP markings and would
> not be allowed to set per-packet DSCP markings; all the reasons to not do
> marking seem to be reasons to not do marking.
>
> I don't know which is the more (or less) distasteful / useful of those two
> paths.

The other option, which I believe works on MacOS, is to setsockopt before
each sendto:

    setsockopt(socket_, IPPROTO_IP, IP_TOS, &dscp, sizeof(dscp));

---

From: Martin Thomson <[email-masked]>
Date: Thu, 14 Aug 2014 13:24:36 -0700

>     setsockopt(socket_, IPPROTO_IP, IP_TOS, &dscp, sizeof(dscp));

That is a solution to the problem, not a fallback position :)

I don't mind whether we require the fallback to be no markings, or we
specify a whole-flow marking that can be used (which is functionally
the same has having no markings, though much harder to define
consistent behaviour as things change), or whether we require
disaggregation.

---

From: Roman Shpount <[email-masked]>
Date: Thu, 14 Aug 2014 16:45:37 -0400

>     setsockopt(socket_, IPPROTO_IP, IP_TOS, &dscp, sizeof(dscp));
And on windows you can do the same by calling the following before sending
each UDP packet:
setsockopt(d_socket, IPPROTO_IP, IP_TOS, (char*)&tos, sizeof(tos));

I believe this will work on any BSD based network stack.

The problem with this approach is that it requires two system calls for
each sent packet. It also has some implications on threading, since without
setting TOS you can send data via the same socket without locking. Now,
because each send is two system calls, lock will be required to ensure
right TOS value, so the overhead can be even higher. There was a discussion
about updating Linux kernel to add support for setting IP_TOS via the
control structure in sendmsg, but I do not think this patch was ever
accepted.

It was also recommended that ICE packets are sent with the same TOS as main
data. Technically, you can end up with the situation where some TOS values
go through the network and some do not. If data is sent with a different
TOS value then ICE, ICE connectivity can succeed but sending the data will
fail. This means an ICE connectivity check for each TOS value, which might
produce different ICE candidates for different TOS values.

Taking all of this into account I would suggest that a single TOS value
should be used for each flow and data with different TOS values should be
dis-aggregated across different flows.
_____________
Roman Shpount

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 14 Aug 2014 23:52:23 +0200

On 14 Aug 2014, at 22:45, Roman Shpount <[email-masked]> wrote:

> On Thu, Aug 14, 2014 at 1:18 AM, Harald Alvestrand <[email-masked]> wrote:
> I know it's possible to do this on Linux (for UDP, of course). I think it still requires an extra sysctl before sending each packet, which is reasonably high overhead if your app is sysctl-limited. But I don't know of any case where you would be allowed to set per-flow DSCP markings and would not be allowed to set per-packet DSCP markings; all the reasons to not do marking seem to be reasons to not do marking.
> 
> I'd like to know if the APIs exist on other platforms (whether they're overridden by default policy or not).
> 
> - Don't ask the network to differentiate, keep precedence of flow a sender-stack thing only
> 
> I don't know which is the more (or less) distasteful / useful of those two paths.
> 
> The other option, which I believe works on MacOS, is to setsockopt before each sendto:
>     
> And on windows you can do the same by calling the following before sending each UDP packet:
> The problem with this approach is that it requires two system calls for each sent packet. It also has some implications on threading, since without setting TOS you can send data via the same socket without locking. Now, because each send is two system calls, lock will be required to ensure right TOS value, so the overhead can be even higher. There was a discussion about updating Linux kernel to add support for setting IP_TOS via the control structure in sendmsg, but I do not think this patch was ever accepted.
FreeBSD supports using a cmsg. So there is only one system call: sendmsg().

Best regards
Michael
> 
> It was also recommended that ICE packets are sent with the same TOS as main data. Technically, you can end up with the situation where some TOS values go through the network and some do not. If data is sent with a different TOS value then ICE, ICE connectivity can succeed but sending the data will fail. This means an ICE connectivity check for each TOS value, which might produce different ICE candidates for different TOS values.
> 
> Taking all of this into account I would suggest that a single TOS value should be used for each flow and data with different TOS values should be dis-aggregated across different flows.
> _____________
> Roman Shpount
> 
>

---

From: Roman Shpount <[email-masked]>
Date: Thu, 14 Aug 2014 18:15:07 -0400

_____________
Roman Shpount

On Thu, Aug 14, 2014 at 5:52 PM, Michael Tuexen <
[email-masked]> wrote:

> > It was also recommended that ICE packets are sent with the same TOS as
> main data. Technically, you can end up with the situation where some TOS
> values go through the network and some do not. If data is sent with a
> different TOS value then ICE, ICE connectivity can succeed but sending the
> data will fail. This means an ICE connectivity check for each TOS value,
> which might produce different ICE candidates for different TOS values.
> >

---

From: Roman Shpount <[email-masked]>
Date: Thu, 14 Aug 2014 18:16:44 -0400

> FreeBSD supports using a cmsg. So there is only one system call: sendmsg().
Do you know if setting TOS via cmsg in sendmsg works for OSX as well?

Thank You,
_____________
Roman Shpount

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 15 Aug 2014 00:30:43 +0200

On 15 Aug 2014, at 00:16, Roman Shpount <[email-masked]> wrote:

> On Thu, Aug 14, 2014 at 5:52 PM, Michael Tuexen <[email-masked]> wrote:
From reading the code: They don't provide a direct way to access it.
They have a proprietary cmsg type SO_TRAFFIC_CLASS, which their own values
priority levels which they map to DSCPs.

Best regards
Michael

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Fri, 15 Aug 2014 14:20:19 +0000

I thought SO_TRAFFIC_CLASS was only for IPv6. Does it also work for IPv4?

Also, I thought these cmsg types were only for the recvmsg direction (when
SO_RECV_TRAFFIC_CLASS is set). Do they also work in the sendmsg direction?
If so, do they override the traffic class only for the specific sendmsg or
all subsequent sends on the socket? This last question would also apply to
FreeBSD, so that answer would be useful even if you have no idea what OSX
does.

Mo

-----Original Message-----
From: Michael Tuexen <[email-masked]>
Date: Thursday, August 14, 2014 at 6:30 PM
To: Roman Shpount <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [rtcweb] -transports- discussion of DSCP

On 15 Aug 2014, at 00:16, Roman Shpount <[email-masked]> wrote:

> On Thu, Aug 14, 2014 at 5:52 PM, Michael Tuexen
><[email-masked]> wrote:

Best regards
Michael

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Fri, 15 Aug 2014 14:51:35 +0000

> setsockopt(socket_, IPPROTO_IP, IP_TOS, &dscp, sizeof(dscp));

This works if your UDP transmit queue is always empty. If it ever backs
up, it fails, because this option clears the transmit queue. So while the
extra syscall overhead is trivial, it is unreliable for general use.

Also note that DART is likely to recommend a UDP flow only multiplex
different drop precedences within the same class (e.g. AF41, AF42), but
not different classes (e.g. AF41, EF). So even if there was a reliable way
to set DSCP per packet on every OS, we must still deal with the higher
level issue of whether to unbundle or unmark in cases DART does not
recommend. Should -transports- give guidance on this? Or leave it up to
implementations?

Mo

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 17 Aug 2014 21:36:21 +0200

> I thought SO_TRAFFIC_CLASS was only for IPv6. Does it also work for IPv4?
I guess so.
http://fxr.watson.org/fxr/source/bsd/netinet/udp_usrreq.c?v=xnu-2050.18.24#L1330
should handle IPv4 and IPv6 sockets.
> SO_RECV_TRAFFIC_CLASS is set). Do they also work in the sendmsg direction?
Yepp. The above is in udp_output(), a routine called when sending UDP packets.
> all subsequent sends on the socket? This last question would also apply to
Only on that packet. The value is not stored at the socket layer / endpoint.
> FreeBSD, so that answer would be useful even if you have no idea what OSX
Same for FreeBSD.

At least on FreeBSD you have a socket option to set the default value for all
future packets. Using the cmsg overwrites the the default on a per-packet
base.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Sun, 17 Aug 2014 21:37:12 +0200

> extra syscall overhead is trivial, it is unreliable for general use.
Which OS has a UDP send queue?

Best regards
Michael

---

From: "Avasarala, Ranjit (NSN - IN/Bangalore)" <[email-masked]>
Date: Mon, 18 Aug 2014 08:02:42 +0000

DSCP can be set by using setsockopt() system call.

Regards
Ranjit

-----Original Message-----
From: rtcweb [mailto:[email-masked]] On Behalf Of ext Michael Tuexen
Sent: Friday, August 15, 2014 4:01 AM
To: Roman Shpount
Cc: [email-masked]
Subject: Re: [rtcweb] -transports- discussion of DSCP

Best regards
Michael

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 18 Aug 2014 10:25:36 +0200

> extra syscall overhead is trivial, it is unreliable for general use.

I'd like to see a demo program that shows there's an issue.

A natural implementation strategy would be to mark the DSCP when 
creating a buffer in sendmsg(), before enqueueing it; there would only 
be an issue if some implementation had a strategy of marking the DSCP 
with the socket's currently selected DSCP value at dequeueing time (and 
one could argue that such an implementation has a bug that needs fixing).

> recommend. Should -transports- give guidance on this? Or leave it up to

Either rtcweb-transports- says what to do, or it refers to something 
normative that says what to do. The only higher level target in the 
chain is -overview, and that's clearly the wrong place.

Since DART's target is informative, by Hobson's choice it seems that 
rtcweb-transports- is the place.

I see the implementation choices to be unbundling, marking at the lowest 
class (?) recommended, or not marking at all - I assume that BE is a 
different class than AF4*.