Subject: Re: [Webpush] Early TSV-ART review of draft-ietf-webpush-protocol-09

From: Brian Raymor <[email-masked]>
Date: Tue, 4 Oct 2016 02:33:13 +0000

Magnus - Thanks for the review. I'll have a pull request available tomorrow to address most of your feedback. In the interim, I've shared a few comments and clarifications below.

On September 27, 2016 3:28 AM, Magnus Westerlund <[email-masked]> wrote:

<snip>

> 3. Section 7.2:

>    To limit the number of stored push messages, the push service MAY
>    either expire messages prior to their advertised Time-To-Live or
>    reduce their advertised Time-To-Live.

> Do I understand this correctly, that theses options are push service 
> side actions that is not notified at that point to the application 
> server? Instead it will have to note that the message was early expired 
> if it subscribes to delivery receipts?

This text should be clarified. There are two cases. First, the push service can only reduce the requested TTL in its response to a request from an application server to deliver a push message:

https://tools.ietf.org/html/draft-ietf-webpush-protocol-10#section-5.2

   A push service MAY retain a push message for a shorter duration than
   requested.  It indicates this by returning a TTL header field in its
   response with the actual TTL.  This TTL value MUST be less than or
   equal to the value provided by the application server.

Second, if an application server subscribes to delivery receipts it will receive a 410 if the push service does not then honor that TTL:

https://tools.ietf.org/html/draft-ietf-webpush-protocol-10#section-6.2

   The push service MAY cease to retry delivery of the message prior to
   its advertised expiration due to scenarios such as an unresponsive
   user agent or operational constraints.  If the application has
   requested a delivery receipt, then the push service MUST return a 410
   (Gone) response to the application server monitoring the receipt
   subscription.

If the application server does not subscribe to delivery receipts, then it's a fire-and-forget scenario.

> 4. Dealing with overload situations

Could webpush implementers review Magnus's questions below and consider whether there is further general guidance that can be offered based on their early experiences with the protocol?

> Reviewing Section 7.1 and other parts I find the discussion of how 
> overload situations of various kinds are dealt with missing some cases. 
> So the general handling of subscriptions are covered in Section 7.1 and 
> with a mitigation of redirecting to another server to handle the new 
> subscription.

> What I lack discussion of are how any form of push back are handled when 
> first the deliver of push service to UA link is overloaded. Is the 
> assumption here that as the push service can store messages the delivery 
> will catch up eventually, or the message expires? 

Your assumption in this case is basically correct. The push service could also use the acknowledgements from the user agent as an indicator of overload and reduce the rate.

> How does one handle a 
> 0-RTT messages when one has a queue of messages to deliver, despite 
> having a UA to Push service connection?

> The second case is how the push service server can push back on 
> accepting new message when it is overloaded. To me it appears that the 
> load on a push service can be very dynamic. Thus overload can really be 
> periods. Thus, what push back to application servers is intended here? 
> Just, do a 503 response on the request from the application server?

We decided on a 429. See https://www.ietf.org/mail-archive/web/webpush/current/msg00341.html - although it's also possible for the push service to terminate a subscription. 

https://tools.ietf.org/html/draft-ietf-webpush-protocol-10#section-8.4

   A push service MAY return a 429 (Too Many Requests) status code
   [RFC6585] when an application server has exceeded its rate limit for
   push message delivery to a push resource.  The push service SHOULD
   also include a Retry-After header [RFC7231] to indicate how long the
   application server is requested to wait before it makes another
   request to the push resource.

   A push service or user agent MAY also terminate subscriptions
   (Section 7.3) that receive too many push messages.

> I do note that RFC 7231 does not appear to have any guidance one how one 
> sets the Retry-After header to spread the load of the retries. This is a 
> known issue. And in this context with automated or external event 
> triggered message creations the push back mechanism needs to be robust 
> and reduce the issues, not make them worse.

> I would recommend that some recommendations are actually included here 
> for handling overload from application server message submissions.

> 5. Life of subscription in relation to transports

> I find myself a bit perplexed of if there are any relation between the 
> subscription and the relation to an existing transport connection and 
> outstanding request, i.e. the availability to receive messages over 
> push. I would guess, that there are no strict need for terminating the 
> subscription even if there are no receiver for an extended time. 

Especially for mobile and/or battery powered clients, the expectation is that
the user agents will be dormant for extended periods of time.

> However, from an application perspective there could exists reason for 
> why a subscription would not be terminated as long as the UA is 
> connected, while any longer duration of no connections could motivate 
> the subscription termination.

> I personally would like a bit more clarification on this, but seeing how 
> these issues are usually handled around HTTP, I would guess the answer, 
> it will be implementation specific? Still there appear to be assumptions 
> around this, why it wouldn't matter that much, but that is only given 
> that one follows these assumptions.

I think that your assumption is correct. The behavior is specific to the push service.
Implementers have requested the ability for a push service to terminate a subscription
at will based on its internal policies.

Again, I'd ask that the webpush implementers share any general guidance or observations for this case.

<snip>

---

From: Costin Manolache <[email-masked]>
Date: Wed, 05 Oct 2016 01:58:13 +0000

On Tue, Sep 27, 2016 at 3:28 AM Magnus Westerlund <
[email-masked]> wrote:

> Hi,
>
> This review is an early, i.e. pre IETF Last Call TSV-ART review. The
> TSV-ART reviews document that has potential transport implications or
> transport related subjects. Please treat it as an IETF Last call comment
> if you don't want to handled it during the AD review.
>
> Document: draft-ietf-webpush-protocol-09
> Title: Generic Event Delivery Using HTTP Push
> Reviewer: M. Westerlund
> Review Date: Sept 27, 2016
>
> Below are a number of comments based on my review. The one with
> transport related subject is the overload handling in comment 4.
>
> 1. Section 3:
>
> So the Security Considerations do require use of HTTP over TLS. However,
> I do wonder if there would be a point to move that requirement up into
> the relevant connection sections, like 3. Especially as when one reads
> the confidentiality requirement in Section 4 for the message one wonders
> a bit why it is not explicitly required in section 3.
>
>
> 2. Section 5.2:
>
> TTL = 1*DIGIT
>
> Shouldn't the upper range for allowed values be specified here. At least
> to ensure that one doesn't get interoperability issues.
>
> 3. Section 7.2:
>

I agree it needs more clarifications - expiring prior to TTL is the same
with
reducing the TTL, so at least that part seems redundant.

In GCM, this would happen only if the sender has too many messages for
a device. I assume some implementation of push may not want to load
the list of saved messages or counters before saving - there are replication
delays and cost savings - so it might make sense to let the pushservice drop
messages without telling the app server. In GCM there is a special message
that is sent to the UA if the pushservice dropped (collapsed) messages, so
UA can do a full sync.

> will catch up eventually, or the message expires? How does one handle a

In most cases the UA to push service link is lightly used - payload size is
limited, as is number of messages per device ( in particular for mobile ).
Yes, I think the assumption is the push service will store - and keep
attempting
to deliver.

There was a separate thread on how to control the flow - since push
promise doesn't have a response - the focus was on delivery receipts, where
overload is far more likely. For UA, the message ack can control the flow.

> Just, do a 503 response on the request from the application server?

Yes, so far that's what GCM is doing. Expectation is that app server will
retry with
backoff.

> and reduce the issues, not make them worse.

Yes, another problem is that a push service can be global, and overload can
happen
only on a shard ( i.e. affect groups of users - but not the entire service
) - in particular
in case of DOS, bugs or an app server running load tests on too few devices.

> that one follows these assumptions.

In general, subscriptions are expected to last for a longer interval to
reduce
load ( devices calling subscribe, then sending the subscriptions to app
servers,
who need to store it ). On the other side, it's common for devices or
browsers to
be reset (clear data,etc) or stop being used.

The fact a device connects is an important signal - it is reasonable to keep
a subscription active while device keeps connection, but expire it after a
certain
time ( and also clear up any stored messages ). Adding some
numbers would be good - but I don't know what would be reasonable, it's
a trade-off between storage on server and traffic/battery on device.

>
> 6. Unclarities which requests that require HTTP/2.
>
> The specification is explicit that some request can be performed over
> HTTP/1.x. However, it is not particular clear which requests that
> require HTTP/2. I assume all the GET requests that will hang to enable
> PUSH promises needs to be HTTP/2? Maybe this should be clarified?
>

+1

Costin

>
> Cheers
>
> Magnus Westerlund
>
> ----------------------------------------------------------------------
> Services, Media and Network features, Ericsson Research EAB/TXM
> ----------------------------------------------------------------------
> Ericsson AB                 | Phone  +46 10 7148287
> <+46%2010%20714%2082%2087>
> Färögatan 6                 | Mobile +46 73 0949079
> <+46%2073%20094%2090%2079>
> SE-164 80 Stockholm, Sweden | mailto: [email-masked]
> ----------------------------------------------------------------------
>
> _______________________________________________
> Webpush mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/webpush
>

---

From: Brian Raymor <[email-masked]>
Date: Wed, 5 Oct 2016 03:10:23 +0000

I've created an initial pull request for review - https://github.com/webpush-wg/webpush-protocol/pull/131 - while we discuss whether there's additional guidance that can be shared for "4. Dealing with overload situations".

-----Original Message-----
From: Brian Raymor [mailto:[email-masked]] 
Sent: Monday, October 3, 2016 7:33 PM
To: Magnus Westerlund <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: RE: Early TSV-ART review of draft-ietf-webpush-protocol-09

<snip>

<snip>

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 5 Oct 2016 15:08:21 +0200

Hi,

I have looked at the pull request which looks good, with one very minor 
comment that you should check out on github.

Cheers

Magnus

Den 2016-10-05 kl. 05:10, skrev Brian Raymor:
> I've created an initial pull request for review -
> https://github.com/webpush-wg/webpush-protocol/pull/131 - while we
> discuss whether there's additional guidance that can be shared for
> "4. Dealing with overload situations".

>
>
> -----Original Message----- From: Brian Raymor
> [mailto:[email-masked]] Sent: Monday, October 3, 2016
> 7:33 PM To: Magnus Westerlund <[email-masked]>;
> [email-masked]; [email-masked];
> [email-masked]; [email-masked] Subject: RE: Early TSV-ART
> review of draft-ietf-webpush-protocol-09
>
> Magnus - Thanks for the review. I'll have a pull request available
> tomorrow to address most of your feedback. In the interim, I've
> shared a few comments and clarifications below.
>
>> Do I understand this correctly, that theses options are push
>> service side actions that is not notified at that point to the
>> application server? Instead it will have to note that the message
>> was early expired if it subscribes to delivery receipts?
>
> This text should be clarified. There are two cases. First, the push
> service can only reduce the requested TTL in its response to a
> request from an application server to deliver a push message:
>
>  A push service MAY retain a push message for a shorter duration
> than requested.  It indicates this by returning a TTL header field in
> its response with the actual TTL.  This TTL value MUST be less than
> or equal to the value provided by the application server.
>
> Second, if an application server subscribes to delivery receipts it
> will receive a 410 if the push service does not then honor that TTL:
>
>  The push service MAY cease to retry delivery of the message prior
> to its advertised expiration due to scenarios such as an
> unresponsive user agent or operational constraints.  If the
> application has requested a delivery receipt, then the push service
> MUST return a 410 (Gone) response to the application server
> monitoring the receipt subscription.
>
> Could webpush implementers review Magnus's questions below and
> consider whether there is further general guidance that can be
> offered based on their early experiences with the protocol?
>
>>  overload situations of various kinds are dealt with missing some
>> cases. So the general handling of subscriptions are covered in
>> Section 7.1 and with a mitigation of redirecting to another server
>> to handle the new subscription.
>
>> What I lack discussion of are how any form of push back are handled
>> when first the deliver of push service to UA link is overloaded. Is
>> the assumption here that as the push service can store messages the
>> delivery will catch up eventually, or the message expires?
>
> Your assumption in this case is basically correct. The push service
> could also use the acknowledgements from the user agent as an
> indicator of overload and reduce the rate.
>
>> How does one handle a 0-RTT messages when one has a queue of
>> messages to deliver, despite having a UA to Push service
>> connection?
>
>> accepting new message when it is overloaded. To me it appears that
>> the load on a push service can be very dynamic. Thus overload can
>> really be periods. Thus, what push back to application servers is
>> intended here? Just, do a 503 response on the request from the
>> application server?
>
> We decided on a 429. See
> https://www.ietf.org/mail-archive/web/webpush/current/msg00341.html -
> although it's also possible for the push service to terminate a
> subscription.
>
>> I do note that RFC 7231 does not appear to have any guidance one
>> how one sets the Retry-After header to spread the load of the
>> retries. This is a known issue. And in this context with automated
>> or external event triggered message creations the push back
>> mechanism needs to be robust and reduce the issues, not make them
>> worse.
>
>> I would recommend that some recommendations are actually included
>> here for handling overload from application server message
>> submissions.
>
>> I find myself a bit perplexed of if there are any relation between
>> the subscription and the relation to an existing transport
>> connection and outstanding request, i.e. the availability to
>> receive messages over push. I would guess, that there are no strict
>> need for terminating the subscription even if there are no receiver
>> for an extended time.
>
> Especially for mobile and/or battery powered clients, the expectation
> is that the user agents will be dormant for extended periods of
> time.
>
>> However, from an application perspective there could exists reason
>> for why a subscription would not be terminated as long as the UA is
>>  connected, while any longer duration of no connections could
>> motivate the subscription termination.
>
>> I personally would like a bit more clarification on this, but
>> seeing how these issues are usually handled around HTTP, I would
>> guess the answer, it will be implementation specific? Still there
>> appear to be assumptions around this, why it wouldn't matter that
>> much, but that is only given that one follows these assumptions.
>
> I think that your assumption is correct. The behavior is specific to
> the push service. Implementers have requested the ability for a push
> service to terminate a subscription at will based on its internal
> policies.
>

-- 

Magnus Westerlund

---

From: Alissa Cooper <[email-masked]>
Date: Mon, 10 Oct 2016 09:21:51 -0400

The IESG will be reviewing this draft this week as it is scheduled for the telechat on Thursday, so please post the -11 version as soon as possible (including any text needed to address #4 below).

Thanks,
Alissa

> On Oct 5, 2016, at 9:08 AM, Magnus Westerlund <[email-masked]> wrote:
> 
> Hi,
> 
> I have looked at the pull request which looks good, with one very minor comment that you should check out on github.
> 
> Cheers
> 
> Magnus
> 
>

---

From: Brian Raymor <[email-masked]>
Date: Mon, 10 Oct 2016 16:49:55 +0000

webpush-11 was published. 

At this time, no further clarifications were added for #4.

-----Original Message-----
From: Alissa Cooper [mailto:[email-masked]] 
Sent: Monday, October 10, 2016 6:22 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: Early TSV-ART review of draft-ietf-webpush-protocol-09

Thanks,
Alissa