Subject: [Webpush] Message expiration (TTL) and Negative Acknowledgements

From: Brian Raymor <[email-masked]>
Date: Tue, 13 Oct 2015 22:10:07 +0000

At IETF 93, we agreed to limit negative acknowledgements to message delivery failures and not message expiration:

https://github.com/webpush-wg/webpush-protocol/issues/17

The thought was that the application server can maintain its own clock for the message expiration based on the TTL exchange where the application server requests a TTL and the push server is allowed to "advertise" a shorter TTL:

Section 6:

   A push service MAY choose to retain a push message for a shorter
   duration than that requested.  It indicates this by including a TTL
   header field in the response that includes the actual TTL.  This TTL
   value MUST be less than or equal to the value provided by the
   application server.

I was finishing up my editorial changes and noticed one potential exception. In Section 7.2, we have some operations guidance:

   Push services might need to limit the size and number of stored push
   messages to avoid overloading.  In addition to using the 413 (Payload
   Too Large) status code for too large push messages, a push service
   MAY expire push messages prior to any advertised expiration time.  A
   push service can reduce the impact [of] push message retention by reducing
   the time-to-live of push messages.

This would be a case where the application server should still receive a negative acknowledgement for expiration because it has no awareness that the push server could not honor its advertised TTL.  It's closer to the failure case where a push server ceases to retry delivery of a message prior to its expiration perhaps due to an unresponsive user agent.

Comments?

...Brian

---

From: Martin Thomson <[email-masked]>
Date: Tue, 13 Oct 2015 15:14:18 -0700

> This would be a case where the application server should still receive a negative acknowledgement for expiration because it has no awareness that the push server could not honor its advertised TTL.…

This doesn't need to be reported any differently from the case where
the push service tried 10 times and gave up.

---

From: Brian Raymor <[email-masked]>
Date: Tue, 13 Oct 2015 22:32:01 +0000

On October 13 2015 at 3:14 PM, Martin Thomson <[email-masked]> wrote:

> > This would be a case where the application server should still receive a
> negative acknowledgement for expiration because it has no awareness that
> the push server could not honor its advertised TTL.  It's closer to the failure
> case where a push server ceases to retry delivery of a message prior to its
> expiration perhaps due to an unresponsive user agent.
> 

Then, we're in agreement. When a receipt is requested, the same 5xx status code is returned for either giving up prior to the original expiration or lowering the advertised message expiration and failing to deliver.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 13 Oct 2015 16:10:08 -0700

> Then, we're in agreement. When a receipt is requested, the same 5xx status code is returned for either giving up prior to the original expiration or lowering the advertised message expiration and f…

WFM.  Now, if only we could decide on what that 5xx might be (or was
your intent to follow-up with a mail on that subject.)

---

From: Brian Raymor <[email-masked]>
Date: Tue, 13 Oct 2015 23:39:15 +0000

> your intent to follow-up with a mail on that subject.)

Next on my list ...

I've created a pull request - https://github.com/webpush-wg/webpush-protocol/pull/48 - to address:

https://github.com/webpush-wg/webpush-protocol/issues/17
https://github.com/webpush-wg/webpush-protocol/issues/18

I've separated the 5XX status code into a new issue - https://github.com/webpush-wg/webpush-protocol/issues/49

...Brian

---

From: JR Conlin <[email-masked]>
Date: Wed, 14 Oct 2015 14:09:16 -0700

I'm not sure if this relates, but there are a few implementation status
codes that might be useful. We've seen the following sorts of situations
arise and it would be good to have developers aware of them.

301 & 307: The resource has either moved permanently or temporarily.
(This may occur if a service in maintenance or if a service was shut
down and traffic needs to be reallocated.)  In the case of a 301
response, all future traffic should be permanently routed to the new host.

429: This will allow the server to apply "back pressure" to the
application service in cases where there may be too many incoming
messages in a short period of time. This response would carry a
RETRY-AFTER response header indicating the number of seconds that should
elapse before the remote server should retry transmission of the message.

On 10/13/2015 4:10 PM, Martin Thomson wrote:
> _______________________________________________
> Webpush mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/webpush

---

From: Martin Thomson <[email-masked]>
Date: Wed, 14 Oct 2015 14:37:50 -0700

> elapse before the remote server should retry transmission of the message.

How is that different from 503?

---

From: jr conlin <[email-masked]>
Date: Wed, 14 Oct 2015 15:02:46 -0700

> How is that different from 503?

Traditionally (as if that's a thing for the web) 503 tends to be used by
intermediary services like load balancers to indicate that "something's
amiss with the box I normally talk to". Since it's a 500 class error, it
indicates an error state possibly from outside the server. 429 was
identified as part of https://tools.ietf.org/html/rfc6585 to be used
more as a rate limiter and carries with it the "Retry-After" response
header indicating how long the requester should wait before retry. It's
a 400 class error, which means it should come from the responding server.

Either would work, of course, but it may be that programmers can take
advantage of pre-existing handlers in libraries to do the back-off for
them, rather than have to code the response themselves.

---

From: Benjamin Bangert <[email-masked]>
Date: Thu, 15 Oct 2015 09:07:55 -0700

So, nacking in the case that we have to clear a client's notifications (too
many) feels like it can cause a really nasty infinite loop error.

Here's the situation I'm seeing:
1. User hits limit on the most queued messages the Push Service wishes to
hold at once for a single client.
2. On the next incoming notification, Push Service clears out the client's
notification queue to make space for new notifications. (I believe GCM
already will drop a bunch of notifications at once when this happens)
3. We send out NACK's for all the dropped notifications
4. App-Servers get nack's and decide to resend notifications.
5. Go to step 1.

What steps are being taken with NACK's to ensure an app-server doesn't do
something like this? Is there going to be some status code or something to
indicate that these messages shouldn't be retried for some period of time?

Or is it going to be up to a Push Service to try and detect these loops,
and when they occur, the Push Service is just going to have to fully expire
all the clients subscriptions to stop the loop?

---

From: Martin Thomson <[email-masked]>
Date: Thu, 15 Oct 2015 10:10:51 -0700

On 15 October 2015 at 09:07, Benjamin Bangert <[email-masked]> wrote:
> Or is it going to be up to a Push Service to try and detect these loops, and
> when they occur, the Push Service is just going to have to fully expire all
> the clients subscriptions to stop the loop?

What steps do you think are appropriate.  I'd have to think some more
on it, but I think that the only reasonable option is to have the push
service do something.  We could recommend the use of Retry-After on
nacks.

Other than that, I guess that this is something the push service needs
to handle.

---

From: Benjamin Bangert <[email-masked]>
Date: Fri, 16 Oct 2015 14:20:29 -0700

I'd prefer to not have to send NACK's in the event that the Push Service
has to clear them out for a user that stored too many. But that's possibly
not an option?

For the NACK's, speaking of server constraints, how many receipts is a Push
Service expected to buffer for an App-Server? If an App-Server sends 500k
notifications in, and happens to not be connected to the receipt server at
the time.... how many of those is the Push Service supposed to hold onto
until the app-server starts pulling them?

If the stream of NACK's is several magnitudes higher than an App-Server can
process them, at what point can a Push Service start dropping NACK's on the
floor?

On Thu, Oct 15, 2015 at 10:10 AM, Martin Thomson <[email-masked]>
wrote:

> and

---

From: Martin Thomson <[email-masked]>
Date: Fri, 16 Oct 2015 15:02:20 -0700

> until the app-server starts pulling them?

The number is bounded by the number of messages the app server has
successfully requested delivery for.  If you have 500k outstanding
messages from the same app server, then you are hold that much message
state; having some proportion of that translate into ack/nack state
seems like it should be less state.

Of course, if you are shedding load, then I see no reason not to be
fairly aggressive about dumping nacks.  That's probably worth noting
in the operational considerations.

> process them, at what point can a Push Service start dropping NACK's on the

Inability to deliver an ACK or NACK can be treated exactly like
failure to deliver a message, I'd imagine.  You can't be expected to
hold these forever.

---

From: Brian Raymor <[email-masked]>
Date: Fri, 16 Oct 2015 22:05:01 +0000

On Thu, October 16, 2015 at 2:30 PM, Benjamin Bangert <[email-masked]>  wrote:

> I'd prefer to not have to send NACK's in the event that the Push Service has to clear them out for a user that stored too many.
> But that's possibly not an option?
> For the NACK's, speaking of server constraints, how many receipts is a Push Service expected to buffer for an App-Server? 

That would depend on the scalability of the push service.

The push service may advertise a lower TTL when accepting a push message from an application server. We've agreed that no acknowledgements are returned once the advertised TTL has expired. A push service could always return TTL with a value of 0 which then does not guarantee that a push receipt will be returned.

JR also proposed that we include a reference to 429 with Retry-After to limit the publish rate of an application server, so a push service is not storing 500K notifications in a batch from an application server that is requesting push receipts.

> If the stream of NACK's is several magnitudes higher than an App-Server can process them,
> at what point can a Push Service start dropping NACK's on the floor?

The push service can expire a receipt subscription at will. Otherwise, the push service needs to wait until the advertised TTL expires.