Subject: Re: [rtcweb] New Version Notification for draft-jesup-rtcweb-data-01.txt

From: Michael Thornburgh <[email-masked]>
Date: Wed, 2 Nov 2011 18:12:53 -0700

comments on this draft:

section 3.2: reliable streams would be fairly simple to layer on top of any reliable and in-order datagram delivery service, not just SCTP's.

section 4.1: DCCP is not a good choice when any kind of reliability is desired. i've seen people on this list suggest that some kind of reliability/retransmission scheme can be layered on top of DCCP, and while that's certainly true, it's very un-optimal. DCCP already uses its own sequence numbers and acknowledgements to detect loss events and trigger congestion response, but that information is semantically hidden from higher-level users of DCCP. additional application-layer sequencing AND acknowledgement information must be implemented to 1) recover the original transmission order at the receiver (including waiting for gaps to be repaired, if in-order delivery is desired); and 2) duplicate application-layer loss detection at the sender (driven by the application-layer acks) must be performed to re-determine which segments were lost so they can be retransmitted. that sucks because, under the covers, DCCP already knew exactly which segments were lost and was in the best position to trigger (fast-) retransmission if appropriate.

general comments on SCTP:

Matthew and i examined SCTP when we were designing MFP (the predecessor of RTMFP). in fact, if you look at the MFP protocol spec in the Internet Archive, you might notice a suspicious similarity to SCTP's chunk scheme.  :)

we rejected SCTP as inappropriate for real-time communication and our other use cases for several reasons. some of these have been at least partially addressed since our initial dismissal of SCTP in 2004 (in particular, DTLS in SCTP addresses the limitations of using TLS in SCTP [RFC3436], which at the time was the only way to "secure" SCTP messages; that mechanism meant that all secure messages/streams had to be fully reliable and in-order, which was unacceptable for real-time communication).

issues with SCTP that remain, and which i believe make it undesirable for WebRTC, include:

  o as you point out in section 5.3, running SCTP over DTLS, rather than DTLS over SCTP, is desired so that all SCTP fields, not just the user data, are secured. however, this configuration is currently not defined by IETF and would have to be specified.

  o DTLS and SCTP handshakes must be performed serially (no matter which order they happen in), which increases the number of round-trips necessary to establish communication.

  o (most important for real-time communication): each user data fragment/chunk is assigned an SCTP Transmission Sequence Number (TSN) at the time of first transmission. that means even if your SCTP implementation supported stream prioritization somehow, the priority decision is only made at first transmission time. since there's just one TSN space and the Gap Ack structure only talks about TSNs, it's undesirable for gaps to persist (else the Gap Ack structure will continue to grow as more losses naturally happen). therefore it's desirable to repair gaps as quickly as possible. this may inappropriately increase the priority of a low priority fragment/chunk/stream during periods of congestion, which is exactly when priority matters (this is a "priority inversion").

  o (second most important for real-time communication): there's only one receive window advertisement for all of the streams, rather than one receive window per stream. this means there's no per-stream flow control. so if you're receiving (for example) a bulk file transfer and real-time player position updates and text chat messages, and you need to suspend the file transfer stream for a time, that means you must also suspend the player position updates and text chat messages. unless you spin up an entirely separate SCTP for each flow control domain, which is lame and defeats the purpose of stream multiplexing.

  o SCTP specifies the maximum number of streams in each direction at association startup. web applications may not know the number of streams needed up front; in fact, the number of streams needed in any real-world non-SS7 data application is very likely to evolve over the lifetime of that application, naturally increasing and decreasing.

  o the semantics of unordered messages are confusing and not a good map to WebRTC. they are semantically equivalent to an entirely separate stream loosely associated to the ordered stream of the same number. there's no way to tell (without application layer sequencing) if an unordered message has been lost/abandoned by the sender. at the protocol level, TSNs can be used to recover the queuing order of received unordered messages, but the TSNs are semantically disconnected from the SCTP user. recovering the original queuing order over a short reorder/reassembly window period is desirable in some real-time applications.

  o an SCTP receiver should be able to choose to receive stream messages in originally-queued order or as-received-on-the-network order on a per-stream basis, and be able to recover the original queuing order to whatever extent desired (potentially limited by real-time constraints) when receiving in network order. SCTP's unordered message semantics are designed for "out-of-band" messages, and are not a good fit for general "real-time" data. transmission order should be determined by the sender, reception order should be determined by the receiver.

  o the stream sequence number being only 16 bits limits the maximum message rate through high delay networks where message gaps can be reliably detected at the receiver, when the sender uses limited-reliability, to 32767 messages/RTT. that sounds large, but could be easily reached even today in moderately high bandwidth*delay paths if messages are small.

specifying and implementing SCTP over DTLS over ICE over UDP is probably nearly as much work as specifying and implementing a new network protocol that actually solves the problems cleanly and holistically. having done that before a few times, i'm in favor of the clean holistic approach.  :)

-mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Randell Jesup
Sent: Monday, October 31, 2011 8:18 PM

A new version of I-D, draft-jesup-rtcweb-data-01.txt has been successfully
submitted by Randell Jesup and posted to the IETF repository.

[...]

http://www.ietf.org/id/draft-jesup-rtcweb-data-01.txt

[...]

---

From: Randell Jesup <[email-masked]>
Date: Thu, 03 Nov 2011 00:51:59 -0400

> section 3.2: reliable streams would be fairly simple to layer on top of any reliable and in-order datagram delivery service, not just SCTP's.

Yes.

> section 4.1: DCCP is not a good choice when any kind of reliability is desired.

I  agree; it was added in there by one of the authors to cover the 
options available.

> we rejected SCTP as inappropriate for real-time communication and our other use cases for several reasons. some of these have been at least partially addressed since our initial dismissal of SCTP i…

Ok, that sounds like SCTP/DTLS (or at least DTLS/SCTP) is usable given 
your comments above.

>    o as you point out in section 5.3, running SCTP over DTLS, rather than DTLS over SCTP, is desired so that all SCTP fields, not just the user data, are secured. however, this configuration is cur…

Ok - though conceptually SCTP should be able to run over anything 
presenting a UDP-like interface.   There are some fallouts, like no 
multi-homing.  Can some others here (Magnus?) with better understanding 
of the exact RFC requirements state if it would be acceptable to 
reference a SCTP-over-DTLS draft, which should be fairly straightforward 
to create and move forward?

>    o DTLS and SCTP handshakes must be performed serially (no matter which order they happen in), which increases the number of round-trips necessary to establish communication.

True, though since we're not sending media data over the SCTP link I 
don't think this is a significant problem; almost any 
reliable-datagram-over-DTLS would have a similar issue I think.

>    o (most important for real-time communication): each user data fragment/chunk is assigned an SCTP Transmission Sequence Number (TSN) at the time of first transmission. that means even if your SC…

Priority inversion is a bad thing; though since it would only apply to 
the data streams the impact is limited.  I'd be interested in hearing 
Michael Tuexen's/etc discussion of this, and how big an impact it would be.

>    o (second most important for real-time communication): there's only one receive window advertisement for all of the streams, rather than one receive window per stream. this means there's no per-…

That's more concerning.  Correct me if I've misunderstood: if I stop 
reading from the file-transfer stream, that will block the other 
streams?  One could layer something on top of SCTP to provide per-stream 
queues I imagine, and ensure in the rtcweb code that the main receive 
window doesn't stall, but the problem with that would be providing 
reliability if the file-transfer stream is reliable - if it's reliable, 
we'd have to in theory buffer a large amount of data *or* block other 
channels *or* put a flow-control layer on top of the reliable channels, 
which hurts the overall complexity and advantage of using existing 
code.  Perhaps I do misunderstand this issue (I hope so actually).

Or we state that the JS app must service all channels expeditiously or 
real-time channels may be blocked, which is probably the way we'd 
resolve it.  Definitely sub-optimal and an annoyance for the JS app 
writer in some cases, but probably not a major problem.

>    o SCTP specifies the maximum number of streams in each direction at association startup. web applications may not know the number of streams needed up front; in fact, the number of streams neede…

I would assume we would define a maximum number of data streams, and 
specify that number when opening the association.  Not optimal, but 
workable.

>    o the semantics of unordered messages are confusing and not a good map to WebRTC. they are semantically equivalent to an entirely separate stream loosely associated to the ordered stream of the…

Thus far we haven't required that unordered delivery be exposed to the 
JS app, though that capability would be handy for some uses.  I have no 
problem with the application being required to include its own sequence 
number in the data if it cares and if that helps with this objection.

>    o an SCTP receiver should be able to choose to receive stream messages in originally-queued order or as-received-on-the-network order on a per-stream basis, and be able to recover the original q…

Agreed; but the spec (and code) for SCTP exists, even if sub-optimal, 
and timeframe is a major constraint.

>    o the stream sequence number being only 16 bits limits the maximum message rate through high delay networks where message gaps can be reliably detected at the receiver, when the sender uses limi…

Worst-case RTT is probably on the order of 2s (~2 satellite links each 
direction).  Note that for most use-cases this would be an intolerably 
long delay, but not for all.  16K messages/second may be reachable in 
theory, but given the likely bandwidth of said long-latency links I 
don't think this is a restriction that would bother me at all (even with 
high bandwidth I don't think this would bother me).

> specifying and implementing SCTP over DTLS over ICE over UDP is probably nearly as much work as specifying and implementing a new network protocol that actually solves the problems cleanly and holi…

Before I answer that: Thanks greatly for the detailed critique!

In response I have to say: put a protocol proposal on the table or 
gather a group to do so, and do so ASAP, because if we don't get an 
alternative, it *will* be either SCTP/DTLS/(ICE/UDP) or 
pseudo-TCP/DTLS/(ICE/UDP), and I don't know what we'd do for the 
unreliable congestion-controlled channels (maybe DCCP).

If you do put forward a proposal, do you really believe it can be done, 
standardized, agreed to and implemented in the timeframe required?  
Specifying SCTP/DTLS seems relatively straightforward compared to 
standardizing a new protocol...  The timeframe for this is quite 
constrained already; one of the strongest arguments against SCTP would 
also be timeframe, except that SCTP gives us congestion-controlled 
unreliable data, which pseudo-TCP-over-DTLS doesn't - we'd have to use 
yet another unspecified mechanism for congestion control of the 
unreliable data channels.

I think the hottest item you've flagged from my perspective is the 
receive-window blocking, followed by wanting to understand the possible 
priority-inversions.  Michael?  Randy?

-- 
Randell Jesup
[email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 3 Nov 2011 14:09:09 +0100

>> o as you point out in section 5.3, running SCTP over DTLS, rather
>> than DTLS over SCTP, is desired so that all SCTP fields, not just
>> the user data, are secured. however, this configuration is
>> currently not defined by IETF and would have to be specified.
> 
> 
> multi-homing.  Can some others here (Magnus?) with better
> understanding of the exact RFC requirements state if it would be
> acceptable to reference a SCTP-over-DTLS draft, which should be
> fairly straightforward to create and move forward?

It is fine for this WG to request publication of a document with
normative references to internet drafts. What really happens is two
things. First we WG chairs + AD needs to determine if it is reasonable
to request publication without the normative reference being ready. That
depends on how clean the separation is. Secondly even if IESG approves
our document, the RFC can't be published until that draft also is
approved and ready to be published. This is fairly common and if you
look at the RFC-editors home page they have this clusters for documents
that await publication due to the normative reference not being ready.

When it comes to the actual usage of either of IP/UDP/SCTP/DTLS and
IP/UDP/DTLS/SCTP we have the same two dependencies on other WGs.

First we need a SCTP over foo document, which is TSVWG. The SCTP over
UDP is already an WG document and making some progress. For SCTP over
DTLS there would be need for a new document, or possibly get that
defined in the same as SCTP over UDP directly.

Secondly if one uses SDP one needs a definition of how one expresses
this on the media description line. That includes the "proto" identifier
itself and any additional information, like direction for establishing
the connection which would be similar to what COMEDIA defines for TCP
(RFC 4145).

>From a time perspective IP/UDP/SCTP/DTLS is the quicker road as the SCTP
over UDP is already in progress and DTLS over SCTP is defined.

So what exist here is really:

https://datatracker.ietf.org/doc/draft-ietf-mmusic-sctp-sdp/
https://datatracker.ietf.org/doc/rfc6083/ (DTLS over SCTP)
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-sctp-udp-encaps/

> unreliable congestion-controlled channels (maybe DCCP).

Personally view:

I think the choice will be between either

DTLS/SCTP/UDP or SCTP/DTLS/UDP

and

TCP/UDP + DCCP/UDP

> 
> If you do put forward a proposal, do you really believe it can be
> done, standardized, agreed to and implemented in the timeframe
> required? Specifying SCTP/DTLS seems relatively straightforward
> compared to standardizing a new protocol...  The timeframe for this
> is quite constrained already; one of the strongest arguments against
> SCTP would also be timeframe, except that SCTP gives us
> congestion-controlled unreliable data, which pseudo-TCP-over-DTLS
> doesn't - we'd have to use yet another unspecified mechanism for
> congestion control of the unreliable data channels.
> 

I personally don't think think this can be done in short time frames
within the IETF. The reason is that IETF will have to do the following
to get this going and in the end produce an RFC:

1. Write a Charter proposal for a WG in TSV.
2. Have a BOF on the need for creating a new transport protocol
3. Take in all the additional views from other proponents of transport
protocol evolution. I think that means combing Structure Streams with
SCTP + DCCP to build a new swiss army knife of a protocol
4. Agree on the new charter
5. Send it to IESG for approval
6. Have the IETF last call and deal with the comments.
7. Finally having a WG
8. Do a requirements phase
9. Design the new protocol in a committee
10. Have the new protocol receive wide review from OPS, Sec etc.
11. Improve spec by having N meeting cycles go by.
12. WG last call
13. IETF last call
14. IESG approval
15. RFC editor
16. RFC publication

Just the required process steps in the above is on the order of 3
months. My guess is that even if interest in this holds it is 3-4 years
before IETF can conclude on a new transport protocol.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Randell Jesup <[email-masked]>
Date: Thu, 03 Nov 2011 11:41:17 -0400

> the connection which would be similar to what COMEDIA defines for TCP

We probably need to do these in any case (SCTP or not).

> over UDP is already in progress and DTLS over SCTP is defined.

Right, but that also means we'd need to define reliable stream-like 
layers on top of DTLS-over-SCTP, deal with large message fragmentation, 
etc.  Perhaps someone with more DTLS-over-SCTP knowledge (Michael, 
Randy) could detail what the impact would be.  (I'll also start 
reviewing the DTLS-over-SCTP draft.)

> DTLS/SCTP/UDP or SCTP/DTLS/UDP

I agree that this is the fundamental choice, barring some surprise. 
There is one other option, though I would vote against it and I doubt 
anyone really wants it: drop internal data streams and require use of 
websockets for data (and that means no unreliable streams, which 
severely crimps real-time data use).

Sometimes it seems like it would be easier to rubber-stamp (document if 
you prefer) an existing protocol design done outside the standards 
process than to develop one inside of it, which is a problem, but not 
one we'll solve here.

Realize that if we don't find a solution that works within the standards 
process, the vendors in question will be forced to implement something 
that gives the desired functionality in the timeframe required, and then 
it would turn into a case of a defacto standard in need of 
documentation.  Obviously I think all here would prefer to avoid that 
scenario.

> to get this going and in the end produce an RFC:

[snip 16 steps]

> before IETF can conclude on a new transport protocol.

Even getting to the point where we have a complete enough draft within 
the IETF is probably a minimum of 2 years (witness OPUS/codec).

-- 
Randell Jesup
[email-masked]

---

From: Ralph Giles <[email-masked]>
Date: Thu, 3 Nov 2011 17:09:57 -0700

Oops, forgot to cc the list.

 -r

---------- Forwarded message ----------
From: Ralph Giles <[email-masked]>
Date: 3 November 2011 17:09
Subject: Re: [rtcweb] New Version Notification for
draft-jesup-rtcweb-data-01.txt
To: Randell Jesup <[email-masked]>

> submitted by Randell Jesup and posted to the IETF repository.

Some feedback on the draft, more editorial than technical.

Please see the attached patch for a number of formatting, spelling,
grammar and wording improvements. I don't think any of these change
meaning, but if you want a more granular report, you can pull my edit
history from http://thaumas.net/~giles/git/draft-foo.git. Or ask and I
can send separate patches.

Note that there are two 'Req. 10's. I didn't fix that because
renumbering makes it harder to discuss the draft, that they should be
updated before publishing another.

More content-oriented comments:

In Req. 1, the wording is a little unclear. Is it the media streams,
data streams, or both which MUST be able to change at any time?

In Req. 7 and 10 (the first one), I worry that referring specifically
to javascript may end up dated and/or too browser-centric. Can we just
refer to application or webapp? There are places where the distinction
between the user-agent and the content which is requesting the call is
important, but I'm not sure javascript's ubiquity as a vehicle for the
later is relevant.

For example:

[Req. 7] Data streams MUST provide message fragmentation support such
that IP-layer fragmentation does not occur no matter how large a
message or buffer the application passes to the Browser for
transmission.

[Req. 10] The data stream packet format/encoding MUST be such that it
is impossible for a malicious applications to generate a crafted
message which would be interpreted as another native protocol over UDP
- such as UPnP, RTP, SNMP, STUN, etc.

In the "User Space vs Kernel implementation" section, you talk about
deployability, but not about the relative difficulty of implementing
SCTP-on-DTLS vs DTLS-on-SCTP with current kernel stacks. At least a
reference to the discussion lower down (section 5.2) would be helpful
here, I think.

In turn, that section, "User Space vs Kernel implementation," rather
assumes the solution. That might be fine, but instead of saying it
must be possible to implement SCTP and DTLS in user space, you could
rather say, "It MUST be possible to implement the data protocol
without increasing the footprint for system-level calls, i.e. using
UDP sockets."

Since browser vendors support a small handful of platforms, but that's
a poorly defined notion, I suggest this actually be a SHOULD. E.g. do
we know if other browsers currently have sufficient access to UDP to
implement even RTP? We can get the meaning across with less rigid
language.

Thanks for putting the draft together,
 -r

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 5 Nov 2011 22:10:18 +0100

On Nov 3, 2011, at 4:41 PM, Randell Jesup wrote:

> Right, but that also means we'd need to define reliable stream-like layers on top of DTLS-over-SCTP, deal with large message fragmentation, etc.  Perhaps someone with more DTLS-over-SCTP knowledge (Michael, Randy) could detail what the impact would be.  (I'll also start reviewing the DTLS-over-SCTP draft.)
If the upper layer knows that you use stream n in "stream line mode", you can just
ignore the message framing. Pretty simple. No changes needed. Runs over SCTP or DTLS.

If you need large messages with message boundaries, it can be added to SCTP.
It should be not too hard. Randy and myself will look into it... However, 
to allow multiplexing with having more than one message being sent concurrently,
a new chunk is necessary.

Just one note: DTLS-over-SCTP is an RFC and you can find a patch for OpenSSL at
http://sctp.fh-muenster.de/dtls-patches.html

> I agree that this is the fundamental choice, barring some surprise. There is one other option, though I would vote against it and I doubt anyone really wants it: drop internal data streams and require use of websockets for data (and that means no unreliable streams, which severely crimps real-time data use).
> 
> 
> Sometimes it seems like it would be easier to rubber-stamp (document if you prefer) an existing protocol design done outside the standards process than to develop one inside of it, which is a problem, but not one we'll solve here.
> 
> Realize that if we don't find a solution that works within the standards process, the vendors in question will be forced to implement something that gives the desired functionality in the timeframe required, and then it would turn into a case of a defacto standard in need of documentation.  Obviously I think all here would prefer to avoid that scenario.
> 
> Even getting to the point where we have a complete enough draft within the IETF is probably a minimum of 2 years (witness OPUS/codec).
> 
> 
> -- 
> Randell Jesup
> [email-masked]
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Ralph Giles <[email-masked]>
Date: Sun, 6 Nov 2011 17:51:48 -0800

> a new chunk is necessary.

I humbly suggest something derived from RFC 3533 if this turns out to
be necessary.

 -r

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 11 Nov 2011 14:15:37 +0100

Hi Michael,

see my comments in-line.

Best regards
Michael
On Nov 3, 2011, at 2:12 AM, Michael Thornburgh wrote:

> section 4.1: DCCP is not a good choice when any kind of reliability is desired. i've seen people on this list suggest that some kind of reliability/retransmission scheme can be layered on top of DCCP, and while that's certainly true, it's very un-optimal. DCCP already uses its own sequence numbers and acknowledgements to detect loss events and trigger congestion response, but that information is semantically hidden from higher-level users of DCCP. additional application-layer sequencing AND acknowledgement information must be implemented to 1) recover the original transmission order at the receiver (including waiting for gaps to be repaired, if in-order delivery is desired); and 2) duplicate application-layer loss detection at the sender (driven by the application-layer acks) must be performed to re-determine which segments were lost so they can be retransmitted. that sucks because, under the covers, DCCP already knew exactly which segments were lost and was in the best posit
> ion to trigger (fast-) retransmission if appropriate.
> 
That is correct. SCTP adds one RTT to whatever DTLS is requiring.
So when using SCTP/DTLS/UDP, DTLS needs 3 RTTs (including the initial
RTT required for the Cookie exchange).
When using DTLS/SCTP/UDP DTLS needs 2 RTTs (since there is no need for
the DTLS Cookie). I'm not sure how many RTTs ICE takes.
>  o (most important for real-time communication): each user data fragment/chunk is assigned an SCTP Transmission Sequence Number (TSN) at the time of first transmission. that means even if your SCTP…
What might help is that messages have priorities and the sender is allowed to abandon messages
with low priorities in case of congestion (timer based retransmissions). Something which is supported by PR-SCTP.
>  o (second most important for real-time communication): there's only one receive window advertisement for all of the streams, rather than one receive window per stream. this means there's no per-st…
You can use priorities tied to streams at the sender side.
>  o SCTP specifies the maximum number of streams in each direction at association startup. web applications may not know the number of streams needed up front; in fact, the number of streams needed…
You don't need a lot of resources for the receive side of a stream. So you could either negotiate a number
large enough or use the extension
http://tools.ietf.org/html/draft-ietf-tsvwg-sctp-strrst-12
which is currently in IETF LC. It enables you to add streams on the fly.
>  o the semantics of unordered messages are confusing and not a good map to WebRTC. they are semantically equivalent to an entirely separate stream loosely associated to the ordered stream of the sa…
The concept of PR-SCTP is that the receiver can handle this. Since the sender decided that the sequencing
is not important, why should be receiver care?
>  o an SCTP receiver should be able to choose to receive stream messages in originally-queued order or as-received-on-the-network order on a per-stream basis, and be able to recover the original que…
It is a sender side thing. If the sender requires in-sequence delivery, you want the receiver to ignore
this requirement? If there is not sequencing constraint, the sender should not send it ordered.
>  o the stream sequence number being only 16 bits limits the maximum message rate through high delay networks where message gaps can be reliably detected at the receiver, when the sender uses limite…
I don't understand this limitation. The is a 32-bit sequence number space (TSNs) which limits you 2**31 - 1
DATA chunks being in flight (as indicated by the last sentence of Section 6.1 of RFC 4960),
but the 16-bit per stream sequence numbering (SSNs) does not have this restriction.
The receiver will recover the sequencing based on SSN and TSN. At least this is 
> 
> specifying and implementing SCTP over DTLS over ICE over UDP is probably nearly as much work as specifying and implementing a new network protocol that actually solves the problems cleanly and holistically. having done that
It would be good to have a clean understanding of the problems. I agree with that.
> before a few times, i'm in favor of the clean holistic approach.  :)
> 
> -mike
> 
>

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 11 Nov 2011 06:51:23 -0800

On Fri, Nov 11, 2011 at 5:15 AM, Michael Tuexen
<[email-masked]> wrote:
> the DTLS Cookie).

There's no need for the DTLS cookie in either case, actually, since ICE provides
the appropriate proof of return routability.

Also, DTLS should probably be used with False Start here, so you can reduce
to one RTT.

-Ekr

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 11 Nov 2011 18:01:57 +0100

> the appropriate proof of return routability.
I don't know much about ICE....
So assume that the endpoint willing to accept DTLS connections.
Can't I just send an ClientHello via plain UDP to the endpoint (assuming that I can reach it)?

Best regards
Michael

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 11 Nov 2011 09:14:21 -0800

> Can't I just send an ClientHello via plain UDP to the endpoint (assuming that I can reach it)?

When ICE is involved, there's no real concept of "an endpoint willing to
accept DTLS connections". At a high level, the way that ICE works is
that the communicating parties establish a session out of band and
then use the ICE handshake to bind one or valid 5-tuple flows to the
the session. Packets that arrive at one endpoint that aren't part of one
of these flows can more or less be discarded. (At least at this stage).

Since the purpose of the DTLS cookie is to prevent blind resource
consumption DoS attacks, and ICE inherently that you're not going
to be doing handshakes with random remote IP addresses, I don't
think there's much of an issue here.

-Ekr

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 11 Nov 2011 18:31:26 +0100

> of these flows can more or less be discarded. (At least at this stage).
OK, you you only pass those packets to DTLS, which belong to the 5 tuple
which you have already communicated with, and discard the others, it is OK.
So SCTP needs one RTT, DTLS two.

Thanks for the clarification.

Best regards
Michael

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 11 Nov 2011 09:33:53 -0800

> So SCTP needs one RTT, DTLS two.

Possibly < two for DTLS, because of resumption and False Start. So, somewhere
between 1 and 2.

Best,
-Ekr

---

From: Michael Tüxen <[email-masked]>
Date: Fri, 11 Nov 2011 19:17:50 +0100

> Possibly < two for DTLS, because of resumption and False Start. So, somewhere
Correct.

Best regards
Michael

---

From: Michael Thornburgh <[email-masked]>
Date: Fri, 11 Nov 2011 11:52:32 -0800

hi Michael. my comments also inline.

> From: Michael Tuexen [mailto:[email-masked]] 
> Sent: Friday, November 11, 2011 5:16 AM
> Subject: Re: [rtcweb] New Version Notification for draft-jesup-rtcweb-data-01.txt
> 
> Hi Michael,
> 
> see my comments in-line.
> 
> Best regards
> Michael
[...]
> with low priorities in case of congestion (timer based retransmissions). Something which is supported by PR-SCTP.

sure. but the problem here is the low priority stream is likely to be one that needs full reliability (like a bulk data transfer that you want to use "background" bandwidth but still eventually get delivered). in that case you don't want the sender to abandon the low priority message, and that's what leads to the priority inversion.

> You can use priorities tied to streams at the sender side.

priority doesn't solve the problem at all. the issue is that if the receiver needs to stop accepting data on one stream, the only indication of reverse pressure to the sender is the association-wide receive window advertisement. the only in-protocol flow control is association-wide. the only solution is to not have the receiver suspend reading any stream and to use application-layer flow control to signal the sender to stop sending data on one stream or another.

> which is currently in IETF LC. It enables you to add streams on the fly.

ok. i'm glad someone thought of that.

> is not important, why should be receiver care?

my main point was that, specifically in the case of PR-SCTP, since unordered messages don't have a stream sequence number there's no way to tell at the receiver if one (or more) has been abandoned. gaps in the ordered message space can be detected.

> this requirement? If there is not sequencing constraint, the sender should not send it ordered.

it's natural to use unordered messages for real-time data, since they can be delivered to the user as soon as they arrive even if there are gaps. however, in exactly these real-time scenarios, it's still often useful to know the original queuing order of the messages, and to allow the receiver to partially recover the original queuing order in, for example, a jitter/reorder buffer. this can be accomplished with application-layer sequence numbers, but it seems silly to have to duplicate existing protocol-level functionality (see my point about flow control above).

> The receiver will recover the sequencing based on SSN and TSN. At least this is 

this is only a problem with PR-SCTP. if stream sequence numbers can be abandoned, then there's ambiguity in the stream sequence number space when you receive a FORWARD TSN chunk if there were more than 32768 stream sequence numbers in flight.

[...]

-mike

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 11 Nov 2011 22:14:49 +0100

> sure. but the problem here is the low priority stream is likely to be one that needs full reliability (like a bulk data transfer that you want to use "background" bandwidth but still eventually get…
So you want to send the bulk transfer with low priority but full reliability?
And have some other messages, for example with high priority, but only limited reliability?
You could do this. The only point: You need something to drop to improve things.
Or do you want everything with full reliability?
> priority doesn't solve the problem at all. the issue is that if the receiver needs to stop accepting data on one stream, the only indication of reverse pressure to the sender is the association-wid…
We have looked into this. Providing per stream flow control is hard to do as an extension
to SCTP without introducing deadlocks. Looking at the SCTP socket API, there is a socket
per association, so you can't read per stream. If you want signaling from the receiver,
it must be done at the application layer. You can't use the flow control mechanism for that.
> my main point was that, specifically in the case of PR-SCTP, since unordered messages don't have a stream sequence number there's no way to tell at the receiver if one (or more) has been abandoned.…
Conceptual question: Why does unordered  data need some ordering information.
It sounds to me that you don't have unordered stuff in mind, but ordered unreliable stuff.
> it's natural to use unordered messages for real-time data, since they can be delivered to the user as soon as they arrive even if there are gaps. however, in exactly these real-time scenarios, it's…
Hmm. Looking at the service you want doesn't seem be unordered. For me this looks like
an ordered unreliable service, taking some timing requirements into account.
So the problem is not, that SCTP does not provide a SSN for unordered DATA chunks,
but that you use unordered messages instead of ordered and that currently you can't
tell SCTP to deliver messages if they stayed for some time in the receive buffer.
However, this is a receiver side only thing which could be done. This would be
a local receiver side only modification supporting partial reliability. It could be
done. It is more an API issue.

However, I would assume that this applies mostly to multi media stuff. Isn't RTP
the protocol being considered for that?
> this is only a problem with PR-SCTP. if stream sequence numbers can be abandoned, then there's ambiguity in the stream sequence number space when you receive a FORWARD TSN chunk if there were more…
Hmm. Interesting point. I think you don't need to limit the number of outstanding user messages per
stream to 32768. The crucial point is that you can only FORWARD-TSN the 64K lowest TSN per stream
in a single FORWARD-TSN chunk. So you might need multiple FORWARD-TSN chunks if you need to FORWARD-TSN
a lot of messages per stream. But it doesn't limit the throughput, I think.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 11 Nov 2011 19:14:27 -0500

> So you want to send the bulk transfer with low priority but full reliability?

Yes, this would be quite common for things like background file 
transfers and data synchronization.

> You could do this. The only point: You need something to drop to improve things.

Yes, the background reliable flow *should* slow down, but it must remain 
reliable.  It would probably be ok if the background flow slowed down, 
and the high-priority flow also slowed down.

Note that we're talking about interactions with congestion-control here, 
and the algorithm can be modified (and we're talking about integrated 
congestion control between media and data anyways).

> Or do you want everything with full reliability?

No.

> it must be done at the application layer. You can't use the flow control mechanism for that.

We can probably state that the app must deal with getting data for all 
streams; it can't decide not to service one stream of data, or implement 
application-level flow control.

> It sounds to me that you don't have unordered stuff in mind, but ordered unreliable stuff.

Because of state:  If you have datagrams A, B, C, and it's delivered B, 
A, C, you may throw away A *if* the data in A was already provided or 
updated in B, but not throw it away if B and A are independent.  You 
need to know the original ordering to know that.  Games will do stuff 
like this.

Also, with datagrams you may need to know about losses when in-order 
(gaps), to change how you process data (simple example: PR-SCTP for 
lossy audio; on a missing packet you synthesize a replacement instead of 
ignoring the loss.)

Yes, these can be done with application sequence numbers (so long as 
out-of-order delivery is allowed).  It's moderately inefficient to do so 
if the sequence data is available in the protocol.

> an ordered unreliable service, taking some timing requirements into account.

A jitter buffer is in fact inherently an in-order partial-reliability 
service, true - but with different characteristics from SCTP 
partial-reliability (adaptive to the jitter level, perhaps to appearance 
of spikes or bulk-delay changes, concealment, whether it's isochronous 
in output, etc).  As mentioned before, there are other uses of true 
out-of-order delivery.

> done. It is more an API issue.

Right: After reading a datagram, do GetLastSerialNum() or some such.

> the protocol being considered for that?

Yes, and *we* don't plan to operate RTP jitter buffers through SCTP.  
But an application built on webrtc might, and certainly they may do the 
B, A, C stuff I mentioned above (and react in their app to whether there 
was any loss).

-- 
Randell Jesup
[email-masked]

---

From: Michael Tüxen <[email-masked]>
Date: Sat, 12 Nov 2011 22:00:03 +0100

On Nov 12, 2011, at 1:14 AM, Randell Jesup wrote:

> Yes, this would be quite common for things like background file transfers and data synchronization.
> 
> Yes, the background reliable flow *should* slow down, but it must remain reliable.  It would probably be ok if the background flow slowed down, and the high-priority flow also slowed down.
> 
> Note that we're talking about interactions with congestion-control here, and the algorithm can be modified (and we're talking about integrated congestion control between media and data anyways).
OK, I see.
> We can probably state that the app must deal with getting data for all streams; it can't decide not to service one stream of data, or implement application-level flow control.
That is a good idea.
> Because of state:  If you have datagrams A, B, C, and it's delivered B, A, C, you may throw away A *if* the data in A was already provided or updated in B, but not throw it away if B and A are independent.  You need to know the original ordering to know that.  Games will do stuff like this.
This means that there are sequencing constraints... So it is not unordered. I think
the service is ordered, unreliable what is used...
> 
> Also, with datagrams you may need to know about losses when in-order (gaps), to change how you process data (simple example: PR-SCTP for lossy audio; on a missing packet you synthesize a replacement instead of ignoring the loss.)
> 
> Yes, these can be done with application sequence numbers (so long as out-of-order delivery is allowed).  It's moderately inefficient to do so if the sequence data is available in the protocol.
> 
> A jitter buffer is in fact inherently an in-order partial-reliability service, true - but with different characteristics from SCTP partial-reliability (adaptive to the jitter level, perhaps to appearance of spikes or bulk-delay changes, concealment, whether it's isochronous in output, etc).  As mentioned before, there are other uses of true out-of-order delivery.
I think the point is:
The service used us unreliable, ordered. And the application want to do the
ordering by itself. So it should do so. Which includes providing the necessary
information for it.
> Yes, and *we* don't plan to operate RTP jitter buffers through SCTP.  But an application built on webrtc might, and certainly they may do the B, A, C stuff I mentioned above (and react in their app to whether there was any loss).
It can do whatever it wants using unordered, but it has to provide the information
needed for it by itself. It might be a sequence number, some timing information,...

Best regards
Michael

---

From: Randell Jesup <[email-masked]>
Date: Sat, 12 Nov 2011 20:49:48 -0500

> the service is ordered, unreliable what is used...

To be utterly clear: the game example I gave requires unordered, 
unreliable (though unordered reliable and unordered partial-reliability 
would work too).  It also needs to know the original sequence to know if 
there are gaps or out-of-order deliveries.  If it doesn't get the 
original sequence information from the stack (i.e. know if a packet has 
a gap in delivery, or a packet is out-of-order (and ideally where in the 
sequence it belonged), then it must add it's own message serial numbers 
to the application data.

Any sequence constraints as described might apply to parts of the 
message, not only to the message as a whole.  How to handle this is 
entirely in the application domain.

> ordering by itself. So it should do so. Which includes providing the necessary

Sure it can provide the information itself.  It is a waste of bits since 
the stack knows the information, but it's only a minor waste (16 or 32 
bits per datagram).  We certainly can deal with that if the stack 
doesn't supply it.

-- 
Randell Jesup
[email-masked]

---

From: Wolfgang Beck <[email-masked]>
Date: Mon, 14 Nov 2011 17:35:10 +0800

Is rtcweb-data protocol only usable if both browsers use the same JS
client? If not, how is the content of the data stream -- the 'codec'
-- negotiated between the different JS clients? ROAP will not work as
it is the JS clients which writes to the data stream. The ROAP
component of the browser doesn't know what's in there.

rtcweb-data will only work across different servers if you have
standardized the datastream's content.

However, if we dropped the server-to-server interworking requirement,
rtcweb-data would be a good base for exciting applications.

Wolfgang Beck

---

From: Justin Uberti <[email-masked]>
Date: Mon, 14 Nov 2011 05:42:36 -0500

On Mon, Nov 14, 2011 at 4:35 AM, Wolfgang Beck <
[email-masked]> wrote:

> client?

No.

> If not, how is the content of the data stream -- the 'codec'

The format of the data stream will be standardized.

> standardized the datastream's content.

Agree.

> rtcweb-data would be a good base for exciting applications.

I agree the data channel will result in exciting applications, regardless
of s2s interworking.

---

From: Wolfgang Beck <[email-masked]>
Date: Mon, 14 Nov 2011 19:20:14 +0800

On Mon, Nov 14, 2011 at 6:42 PM, Justin Uberti <[email-masked]> wrote:
> On Mon, Nov 14, 2011 at 4:35 AM, Wolfgang Beck
> <[email-masked]> wrote:

Now it gets interesting: who is responsible for the rtcweb-data content?

if it is the browser:
- every new application will have to be standardized.
- for every new application, the browser has to be updated.
***whether we do server-to-server or not***

It basically means every rtcweb-data application needs to get the
approval of Firefox, Google, and Microsoft.

If it is the JS client:
- the JS client has to handle Offer/Answer. The browser doesn't know
enough about the content to do this.

Wolfgang Beck

---

From: Justin Uberti <[email-masked]>
Date: Mon, 14 Nov 2011 07:09:57 -0500

Not sure what you mean. The wire protocol is standardized, but applications
can send whatever they want over it.
On Nov 14, 2011 7:20 PM, "Wolfgang Beck" <[email-masked]>

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 14 Nov 2011 12:29:37 +0000

> ***whether we do server-to-server or not***

No only the psuedo-transport layer is being standardized - i.e., SCTP/DTLS/UDP.  What goes on top of that (i.e., what the JS actually communicates) would be whatever the JS wants and thus yes of course it would only truly "work" if the same JS was used on both peers or someone went and defined some specific application protocols for it.  That might actually happen - like if someone went and defined an IM protocol to run on the data-channel they could create a spec for it for other web-apps to be able to interoperate.  (or a JS library doing something over the data channel could become so popular as to be a defacto spec for one)

-hadriel

---

From: Wolfgang Beck <[email-masked]>
Date: Mon, 14 Nov 2011 14:10:41 +0100

On Mon, Nov 14, 2011 at 1:29 PM, Hadriel Kaplan <[email-masked]> wrote:
>
> No only the psuedo-transport layer is being standardized - i.e., SCTP/DTLS/UDP.

That's what I hoped to hear.

> What goes on top of that (i.e., what the JS actually communicates) would be whatever the JS wants and thus yes of
> course it would only truly "work" if the same JS was used on both peers or someone went and defined some specific
> application protocols for it.  That might actually happen - like if someone went and defined an IM protocol to run on the
> data-channel they could create a spec for it for other web-apps to be able to interoperate.  (or a JS library doing
> something over the data channel could become so popular as to be a defacto spec for one)

In this case, the different JS clients will have to negotiate this
protocol. In the current ROAP model, this is done by the browser, not
the JS client. It will probably get ugly if I want to negotiate a
combined rtcweb-data and A/V session.

Wolfgang Beck

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 14 Nov 2011 23:29:53 +0800

On 11/14/2011 07:20 PM, Wolfgang Beck wrote:
> On Mon, Nov 14, 2011 at 6:42 PM, Justin Uberti<[email-masked]>  wrote:
I believe the discussion above bears very little relationship to what 
has been proposed.
As I read the proposal:

- Data channel does not touch the servers.
- Of course two JS apps can only have sensible communication if they 
agree on the data format. Being the same application is one way to 
agree; it's not the only one.
- The data format (content of data) is not negotiated in offer/answer. 
Just the fact that there is a data channel is negotiated.
- Standardizing a data format may be useful. But it is not required. 
Bilateral agreements work too.

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 14 Nov 2011 07:37:26 -0800

> has been proposed.  As I read the proposal:

[BA] Yes, that's how I read it as well.  The data channel port/IP address is
determined by offer/answer, so how could it "touch the servers"?

---

From: Wolfgang Beck <[email-masked]>
Date: Tue, 15 Nov 2011 09:02:04 +0800

On Mon, Nov 14, 2011 at 11:29 PM, Harald Alvestrand
<[email-masked]> wrote:
> I believe the discussion above bears very little relationship to what has
> been proposed.
> As I read the proposal:
>
I did not read it otherwise. However, the content of the data stream
could be viewed as a 'codec'
which has to be agreed upon, probably with parameters. And to do this
'codec' negotiation, the server-to-server protocol comes into play.

Wolfgang Beck

---

From: Saúl Ibarra Corretgé <[email-masked]>
Date: Tue, 15 Nov 2011 09:54:05 +0100

Hi,

> 'codec' negotiation, the server-to-server protocol comes into play.

Not sure if a codec is abstract enough. I could see the data channel as a candidate to do IM on the media plane, in the same fashion as MSRP (RFC 4975), but this would of course require a gateway. Is this scenario valid according to the data channel's mission or should it be pursued in another way?

Thanks,

--
Saúl Ibarra Corretgé
AG Projects