Subject: [core] draft-ietf-core-coap-09: Separate response in a lossy context

From: Jeroen Hoebeke <[email-masked]>
Date: Mon, 26 Mar 2012 17:19:00 +0200

Dear working group,

During the plug test event, we noticed different behavior of implementations in case of separated responses in a lossy context.

Case 1:
- client sends CON request
- ACK from server is lost
- separate CON response from server to client
- ACK from client to server

Some implementations do not consider the CON response as an implicit acknowledgement and retransmit the CON request, causing unnecessary traffic. In section 4.1, the draft says: 
>  
> In particular, a CoAP request message may have elicited a
>    separate response, in which case it is clear to the requester that
>    only the ACK was lost and a retransmission of the request would serve
>    no purpose. 

This is only an implementation note. It would be better to state explicitly in section 4.1 of the draft: "The reception of a separate confirmable response before the acknowledgement should be considered as an implicit acknowledgement and should not result in a retransmission of the request in case the acknowledgment never arrives."

Case 2:
- client sends CON request
- server sends ACK
- server sends separate CON response
- ACK from client to server is lost
- server retransmits the CON response
In several implementations, the reception of the ACK results in the removal of the context of the transaction (which makes sense for constrained devices). When the server retransmits the response, this results in a RST message. 

It would be useful to describe this behavior in section 4.1 (or section 4.4.4 as one of the cases where a RST is sent). For a CON request for which the ACK is lost, this behavior is well explained in section 4.1, but for the CON response for which the ACK is lost this is missing.
 
"The reception of a retransmitted confirmable response due to the loss of the client's acknowledgment, may result in RST message in case the client removed the state information after sending the acknowledgment."

Kind regards,
Jeroen Hoebeke

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 26 Mar 2012 18:07:40 +0200

I feel that this discussion is really interesting and needed.

In my opinion there are lots of situations that should be openly discussed
when separate responses are involved.

Case 1: I am working on it, and I have some doubts that when you have this
optimization in place the protocol is consistent. Will get back soon on
this.

Case 2: Are we sure that server behaviour MUST never change when it
receives a RST to the CON instead of an ACK?

Probably in some cases you may want to use a separate response to be sure
that the client has received it, but if we standardize that clients can
lose their state this will not be possible anymore.

Best,
Angelo

On Mon, Mar 26, 2012 at 17:19, Jeroen Hoebeke <[email-masked]
> wrote:

> Dear working group,
>
> During the plug test event, we noticed different behavior of
> implementations in case of separated responses in a lossy context.
>
> Case 1:
> Some implementations do not consider the CON response as an implicit
> acknowledgement and retransmit the CON request, causing unnecessary
> traffic. In section 4.1, the draft says:
> >
> This is only an implementation note. It would be better to state
> explicitly in section 4.1 of the draft: "The reception of a separate
> confirmable response before the acknowledgement should be considered as an
> implicit acknowledgement and should not result in a retransmission of the
> request in case the acknowledgment never arrives."
>
> Case 2:
> In several implementations, the reception of the ACK results in the
> removal of the context of the transaction (which makes sense for
> constrained devices). When the server retransmits the response, this
> results in a RST message.
>
> It would be useful to describe this behavior in section 4.1 (or section
> 4.4.4 as one of the cases where a RST is sent). For a CON request for which
> the ACK is lost, this behavior is well explained in section 4.1, but for
> the CON response for which the ACK is lost this is missing.
>
> "The reception of a retransmitted confirmable response due to the loss of
> the client's acknowledgment, may result in RST message in case the client
> removed the state information after sending the acknowledgment."
>
> Kind regards,
> Jeroen Hoebeke
>
>
>
>
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 26 Mar 2012 18:14:11 +0200

On Mar 26, 2012, at 18:07, Angelo P. Castellani wrote:

> Probably in some cases you may want to use a separate response to be sure that the client has received it, but if we standardize that clients can lose their state this will not be possible anymore.

That is not part of the REST semantics.
If we have a good use case for this, we could decide that we want to make it work.
Currently, as you note, there is no support "certified delivery" for responses, just as in HTTP.

Grüße, Carsten

---

From: Guido Moritz <[email-masked]>
Date: Mon, 26 Mar 2012 18:17:28 +0200

I am not a fan of ASCII-Art ;), but maybe we can have a state-machine depicting the different states for client and server to make sure about consistency in transitions etc? The TCP RFC is also using states to make sure to cover all cases that can occur.

Guido

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Angelo P. Castellani
Gesendet: Montag, 26. März 2012 18:08
An: Jeroen Hoebeke
Cc: core WG
Betreff: Re: [core] draft-ietf-core-coap-09: Separate response in a lossy context

In my opinion there are lots of situations that should be openly discussed when separate responses are involved.

Case 1: I am working on it, and I have some doubts that when you have this optimization in place the protocol is consistent. Will get back soon on this.

Case 2: Are we sure that server behaviour MUST never change when it receives a RST to the CON instead of an ACK?

Best,
Angelo

On Mon, Mar 26, 2012 at 17:19, Jeroen Hoebeke <[email-masked]> wrote:

Dear working group,

Case 1:

Case 2:

Kind regards,
Jeroen Hoebeke

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Mon, 26 Mar 2012 18:20:12 +0200

On Mon, Mar 26, 2012 at 18:17, Guido Moritz <[email-masked]>wrote:

> I am not a fan of ASCII-Art ;), but maybe we can have a state-machine
> depicting the different states for client and server to make sure about
> consistency in transitions etc? The TCP RFC is also using states to make
> sure to cover all cases that can occur.****
>
>
I think that this document could be really useful, and also including in
that document some interesting cases (especially where separate responses
are involved).

Implementers may benefit from having such a document, and also having
critical examples that they should handle in their implementation.

Best,
Angelo

---

From: Salvatore Loreto <[email-masked]>
Date: Mon, 26 Mar 2012 18:35:40 +0200

if we want include a state-machine depicting the different states for 
client and server (and I think it would be a great idea)
the best place to have it is in the core-coap main spec, perhaps as an 
Appendix

/Sal

On 3/26/12 6:20 PM, Angelo P. Castellani wrote:
> On Mon, Mar 26, 2012 at 18:17, Guido Moritz 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     I am not a fan of ASCII-Art ;), but maybe we can have a
>     state-machine depicting the different states for client and server
>     to make sure about consistency in transitions etc? The TCP RFC is
>     also using states to make sure to cover all cases that can occur.
>
>
> I think that this document could be really useful, and also including 
> in that document some interesting cases (especially where separate 
> responses are involved).
>

---

From: Salvatore Loreto <[email-masked]>
Date: Mon, 26 Mar 2012 18:40:05 +0200

> Currently, as you note, there is no support "certified delivery" for responses, just as in HTTP.
I don't get the HTTP comparison here;
what do you mean saying that HTTP does not support "certified delivery" ?
you mean that if the TCP connection brakes (somehow) then the server has 
no way to send back an answer... or what

/Sal

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 26 Mar 2012 19:02:14 +0200

> you mean that if the TCP connection brakes (somehow) then the server has no way to send back an answer... or what

In both protocols, there is no way for the server to find out whether the application actually processed the response successfully.
(I.e., this is not a three-way ACK.)
In the HTTP case, the server does not see whether the last bytes of its response reached the client's computer (this could be fixed by changing the TCP socket interface, but then there is no way to find out whether these then reached the client, and whether the client finished the processing e.g. before a crash).

Grüße, Carsten

---

From: Thomas Fossati <[email-masked]>
Date: Mon, 26 Mar 2012 19:08:29 +0200

On Mar 26, 2012, at 6:35 PM, Salvatore Loreto wrote:
> if we want include a state-machine depicting the different states for client and server (and I think it would be a great idea) 
> the best place to have it is in the core-coap main spec, perhaps as an Appendix 

I think this is a very good idea (BTW I have made my two FSMs pics before implementing evcoap that I can happily share if you want).

Anyway, if we want to provide such information, every I-D that introduces new state (Observe and atomic Block the twos that come immediately to my mind) have to specify their own "appendix" FSM to the core FSM.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 26 Mar 2012 19:11:36 +0200

On Mar 26, 2012, at 19:08, Thomas Fossati wrote:

> I have made my two FSMs pics before implementing evcoap that I can happily share if you want

Yes, please.

Grüße, Carsten

---

From: Guido Moritz <[email-masked]>
Date: Mon, 26 Mar 2012 19:15:44 +0200

> Anyway, if we want to provide such information, every I-D that introduces
new
> state (Observe and atomic Block the twos that come immediately to my mind)
> have to specify their own "appendix" FSM to the core FSM.
Block may benefit from this as well, indeed. But having a combined FSM for
all the funky features of CoAP may blow up the figure way too much. Maybe
first start with the easiest ones which have been identified during the ETSI
plugtest, i.e./e.g. separated response in lossy networks.

Guido

---

From: Jeroen Hoebeke <[email-masked]>
Date: Mon, 26 Mar 2012 21:27:28 +0200

> plugtest, i.e./e.g. separated response in lossy networks.

State machines depicting client and server states is a good idea I definitely support, since it will help developers (in a similar how the examples in Appendix B can help them). Apart for state machines in the appendix, we should not forget to phrase all possible/allowed effects of packet losses in the draft text itself, thereby referring to the two suggestions I made in my earlier e-mail. Looking forward to any improvement or verification of these:

"The reception of a separate confirmable response before the acknowledgement should be considered as an implicit acknowledgement and should not result in a retransmission of the request in case the acknowledgment never arrives."

Kind regards,
Jeroen

---

From: Thomas Fossati <[email-masked]>
Date: Tue, 27 Mar 2012 00:59:54 +0200

>> I have made my two FSMs pics before implementing evcoap that I can happily share if you want

http://koanlogic.com/coap/evcoap-server-fsm.pdf
http://koanlogic.com/coap/evcoap-client-fsm.pdf

---

From: Salvatore Loreto <[email-masked]>
Date: Tue, 27 Mar 2012 09:00:13 +0200

> In the HTTP case, the server does not see whether the last bytes of its response reached the client's computer (this could be fixed by changing the TCP socket interface, but then there is no way to…
sure,
but if you use TCP you are reasonable sure that the OS Stack does it 
works properly
so unless the server does not close the socket right after having sent 
the answer it works
and the REST semantic is safe...
not sure this is the case here in CoAP

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Tue, 27 Mar 2012 11:32:48 +0200

> http://koanlogic.com/coap/evcoap-client-fsm.pdf

Nice drawing Thomas.

I think that your graph is more focused on the overall server/client
software implementation, and I found it quite complex to read from a
protocol perspective (are some states even missing?).

Following the suggestion of Guido, I tried to sketch the FSM of CoAP itself
from a protocol perspective point-of-view, mainly to better understand all
the possible transitions.

If you mind to have a look, you can find it here:
http://www.dei.unipd.it/~castellani/coap-fsm.pdf
(it is really a draft version right now, please be patient if you find any
error)

Best,
Angelo

p.s.: I still have to add the RST messages to the graph, I will update this
later.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Mar 2012 13:31:38 +0200

> http://www.dei.unipd.it/~castellani/coap-fsm.pdf

Can we try to separate the message and request/response layers?

Grüße, Carsten

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Tue, 27 Mar 2012 13:53:54 +0200

> Can we try to separate the message and request/response layers?

Having the Token Option in the request/response layer leads to a problem in
the separation.

In this state machine graph I made the assumption that proper session
matching is applied before applying state transitions. In my point of view
"proper session matching" means checking (loc-host, loc-port, rem-host,
rem-port, token).

Legenda: "loc: Local", "rem: Remote", "host: IP Host", "port: UDP Port",
"token: CoAP Token Option"

If I remove the Token Option from the session matching condition (i.e., I
check [loc-host, loc-port, rem-host, rem-port]), I can't devise any real
state since I can still receive any message in any state, and transition to
any other state depending upon the Token contained in it... Suggestions?

Best,
Angelo

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Mar 2012 14:00:59 +0200

An ACK-0 (the one you get for a request that causes a separate response, as well as the one that asks the separate response) does not have a token.
I believe we still have a proper separation between the retransmission/duplicate detection mechanics (based on MID) and the request/response layer.
Can you point out where you need the token to do the matching here?

I don't know what a "session" is, by the way.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Mar 2012 14:02:04 +0200

> I don't know what a "session" is, by the way.

Sorry, this one was about Angelo's last message.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 27 Mar 2012 14:11:22 +0200

On Mar 27, 2012, at 09:00, Salvatore Loreto wrote:

> but if you use TCP you are reasonable sure that the OS Stack does it works properly
> so unless the server does not close the socket right after having sent the answer it works

No, the app may have crashed.
It may have become impatient and have closed the socket before the data actually got delivered.
It may be buggy and simply not read the offered data.
Etc.

> not sure this is the case here in CoAP

Exactly the same, I'd say.

(But, yes, having TCP implemented professionally in the kernel has its advantages that application developers don't always keep in mind.)

Grüße, Carsten

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Tue, 27 Mar 2012 16:52:40 +0200

I propose an example to better clarify my point:
           C                S
           |                |
id=0x38 |  | GET /a         | /|\
        |  | Token: "a"     |  | id=0x21
=======\|/=|================|==|=======
           | CON MID=0x1234 |
           |--------------->|
           |                |
id=0x39 |  | GET /b         | /|\
        |  | Token: "b"     |  | id=0x22
=======\|/=|================|==|=======
           | CON MID=0x1235 |
           |--------------->|
           |                | response is delayed
           |                | message layer sends
           | ACK MID=0x1235 | an empty ack
           |<---------------|
           |                |
id=0x? /|\ | 2.05 "B"       |  |
38 or 39|  | Token: "b"     |  | id=0x22
========|==|================|=\|/=======
           | CON MID=0xfefe |
           |<---------------|

Assume that the message-layer has no knowledge of the content of each
message, but its knowledge is limited to an identifier univocally
identifying each different conversation (previously called session).

How does the message-layer know to which identifier should the response in
the example be mapped to?

Best,
Angelo

On Tue, Mar 27, 2012 at 14:02, Carsten Bormann <[email-masked]> wrote:

> > An ACK-0 (the one you get for a request that causes a separate response,
> as well as the one that asks the separate response) does not have a token.
> > I believe we still have a proper separation between the
> retransmission/duplicate detection mechanics (based on MID) and the
> request/response layer.

---

From: "Angelo P. Castellani" <[email-masked]>
Date: Wed, 28 Mar 2012 16:08:06 +0200

> Can we try to separate the message and request/response layers?

I have built a FSM with the message-layer splitted from the
request/response-layer. I will validate this by implementation.

If you want to have a look at that you can find it here:
http://www.dei.unipd.it/~castellani/coap_m_fsm.pdf
http://www.dei.unipd.it/~castellani/coap_rr_fsm.pdf

Comments are welcome.

(it is a draft drawing, please be patient).

Best,
Angelo

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Mar 2012 17:51:51 +0200

> http://www.dei.unipd.it/~castellani/coap_m_fsm.pdf

I like that much better.

The RETX ACK is missing its reason (i.e., having got another copy of the CON).

It would also probably be useful to indicate that this is a state per MID, and that MIDs attach to the *sender* of the NON/CON.

TX-CON allocates a MID.

NONs received get dedupped, too (except see below), and their MIDs time out the same way.

In my implementation there is also a reply from the RR layer upon a received CON or NON that the RR layer (or its client) didn't care about duplicate detection ("stateless").  This MID then goes right back to closed.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 28 Mar 2012 17:59:06 +0200

> http://www.dei.unipd.it/~castellani/coap_rr_fsm.pdf

Again, much better.

The states are per token.
Note that the client (the one who sent the request) selects the token, and the token again attaches to the endpoint pair.

Block (with M=1 responses) and observe create somewhat more complex state charts.

A somewhat novel event in my implementation is "active token was reused":

When a client starts using the same token for a request that the server thought still was active (e.g., for an observation relationship or for a pending response), this activity is effectively canceled.

Grüße, Carsten