Subject: [Webpush] Receipt subscription follow-up

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 10:02:08 +1000

Not wanting to steal, Brian's thunder here, but this is the change we
discussed during the meeting:

  https://github.com/webpush-wg/webpush-protocol/pull/84

It's good stuff, but there's a small wrinkle I thought that I'd raise:
https://github.com/webpush-wg/webpush-protocol/issues/85

In short, Brian's proposed changes to receipts are good, but we need
to be clear about how an application server might choose to send a
receipt subscription obtained from one push in subsequent pushes.

I propose using origin (scheme+host+port).  If they are the same, then
a receipt subscription obtained from one push can be used with any
other push to the same origin.  The server is required to understand
these subscriptions (or not raise any errors if it can't).

---

From: Costin Manolache <[email-masked]>
Date: Tue, 26 Apr 2016 04:47:30 +0000

Seems reasonable - but it should be optional if an App Server is using
VAPID,
since the push service can simply use the VAPID public key, and it
simplifies the
AS logic.

Even if a PS doesn't require VAPID - it can still use a derivative of the
public key
in the receipt URL.

Costin

On Mon, Apr 25, 2016 at 5:02 PM Martin Thomson <[email-masked]>
wrote:

> _______________________________________________
> Webpush mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/webpush
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 15:03:51 +1000

> since the push service can simply use the VAPID public key, and it

I was talking primarily about what the application server has to do to
decide what to send.

If this were solely rate limiting, I'd agree.  The choice to send 429
can definitely use the token.  The problem is if there are multiple
application server instances and the push service decides to send the
same subscription back to two different instances, then things might
break.  The inclusion of the receipt subscription in the request is
permission to merge receipts.  The push service shouldn't merge
receipt subscriptions without permission.

---

From: Costin Manolache <[email-masked]>
Date: Tue, 26 Apr 2016 05:18:25 +0000

You mean an AS having multiple instances would use a different
receipt set for each instance ? That may be problematic for many
reasons - like load balancing, queuing, AS server restarts.

I would expect most PS will want to be able to load balance across
multiple connections from AS, and deliver receipts locally, i.e. if a
device is in EU to use AS connections in EU (if the AS has connections
 in different regions). And if an AS instance restarts, receipts would
go to the remaining.

Costin

---

From: Costin Manolache <[email-masked]>
Date: Tue, 26 Apr 2016 05:22:48 +0000

Reading more closely - I think it is fine, the PS is free to return a
different receipt URI, regardless
of what is included.
So a PS requiring VAPID can just return the same URL, based on VAPID.

Costin

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 15:23:03 +1000

On 26 April 2016 at 15:18, Costin Manolache <[email-masked]> wrote:
> receipt set for each instance ?

More likely one per data center/region.  An application "server" might
want to ensure that receipts go to the DC that generated the push
message.

> That may be problematic for many

Acknowledged.  Most of those just fall into the "stuff we deal with
when writing software at scale" bucket though.

---

From: Costin Manolache <[email-masked]>
Date: Tue, 26 Apr 2016 05:26:02 +0000

> want to ensure that receipts go to the DC that generated the push

The wording of the spec doesn't seem to ensure this.

I can see some benefits - but also costs :-)

Costin

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 15:26:25 +1000

> So a PS requiring VAPID can just return the same URL, based on VAPID.

That's not a rule we have.  I'm in two minds about whether we should
write it in.  If we want to have this happen, we should definitely say
that it's possible though.

I was imagining a case where you have VAPID tokens issued on a more
granular level than you have receipts being routed.  Imagine you have
one receipt subscription per region, but a single centralized
controller of the VAPID key so that all the different regions can be
counted by the PS as a single entity.  In that case, inferring from
VAPID might cause the application server some issues.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 15:28:40 +1000

> The wording of the spec doesn't seem to ensure this.

Yes.  I'm trying to get that clarity now.

> I can see some benefits - but also costs :-)

Yes.  One way of managing the costs is to have the PS reject a push if
the number of receipt subscriptions gets too high.  Brian has text for
that in his PR for that already.

---

From: Costin Manolache <[email-masked]>
Date: Tue, 26 Apr 2016 05:32:50 +0000

> that in his PR for that already.

Having one receipt subscription per region/DC may keep the number low
enough - but there is still the issue of locality. If you send a message in
US to a device in EU, should the PS be required to send the receipt back to
US, to the same DC, even if there is an AS server in EU, maybe less loaded
?

I agree it simplifies some logic on the AS side, but what happens when
a DC goes down  ?

Costin

---

From: Martin Thomson <[email-masked]>
Date: Tue, 26 Apr 2016 15:40:50 +1000

On 26 April 2016 at 15:32, Costin Manolache <[email-masked]> wrote:
> Having one receipt subscription per region/DC may keep the number low enough
> - but there is still the issue of locality. If you send a message in US to a
> device in EU, should the PS be required to send the receipt back to US, to
> the same DC, even if there is an AS server in EU, maybe less loaded ?

That's a choice the AS gets to make.  Maybe they have some regulatory
requirements that force a particular structure.  In any case, they can
always pretend that each region is entirely separate applications and
the PS will see them as multiple different clients.

> I agree it simplifies some logic on the AS side, but what happens when

I don't imagine that this is a free pass for having geographic
redundancy on the AS.  If receipts are supposed to go to EU and EU
disappears, I imagine that US will have to pick up the slack.  (More
likely that it will be between DCs in a region that this sort of
switch occurs, but you get the point.)

Maybe think of this less a simplification, but an optimization.

---

From: Brian Raymor <[email-masked]>
Date: Wed, 27 Apr 2016 00:29:31 +0000

> https://github.com/webpush-wg/webpush-protocol/issues/85

<snip>

I've adopted Martin's proposal in #85 and made further clarifications based on the initial review of the pull request.
If there are no further comments, I'll merge the pull request on Wednesday. I'd like to publish -05 before EOW.

Thanks,
...Brian

---

From: Costin Manolache <[email-masked]>
Date: Wed, 27 Apr 2016 04:12:01 +0000

+1

But I'm not sure we can support the mode that Martin described, so I think
we may
take advantage of "PS MAY return a new receipt subscription if it is unable
to reuse".
Buffering and flow control are complicated - the choice is between dropping
receipts
or sending them to a different DC. I haven't checked, but I think there is
no guarantee
on saving receipts if AS doesn't have enough ingress capacity.

One question for HTTP/2 experts: I assumed the receipt promises are subject
to
normal 'max concurrent streams' ( and this will be the main mechanism to do
flow control/balancing from PS to AS). However I see rfc7540:

...
" Note: The client never sends a frame with the END_STREAM flag for

      a server push."

How does the PS know that AS can accept more receipts, and how does it know
a receipt has been accepted by the AS ?

Costin

On Tue, Apr 26, 2016 at 5:29 PM Brian Raymor <[email-masked]>
wrote:

> I've adopted Martin's proposal in #85 and made further clarifications
> based on the initial review of the pull request.
> If there are no further comments, I'll merge the pull request on
> Wednesday. I'd like to publish -05 before EOW.
>
> Thanks,
> ...Brian
>
>
>
>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 27 Apr 2016 14:21:20 +1000

> a receipt has been accepted by the AS ?

There is no explicit h2-layer acknowledgment of a push.  That all
happens implicitly with TCP acks and maybe with flow control acks.  In
HTTP, a server usually doesn't get to learn if its response made it
until the client acts on the response in some way.

However, in this case, can we request that the application server
DELETE (just like the client uses DELETE to acknowledge)?

---

From: Costin Manolache <[email-masked]>
Date: Wed, 27 Apr 2016 14:35:53 +0000

+1 on adding a DELETE.  GCM does the same in XMPP receipts, with an ack.
One problem would be that we'll have an extra RT, since delete requires a
response frame - in XMPP it's a one-way ack.

I don't know if TCP flow control and TCP acks are enough.
How does the HTTP/2 flow control window apply to push promises ?

Another option would be to just use regular http and some json or plain
text to pull batches of receipts.

Costin

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 00:41:34 +1000

> How does the HTTP/2 flow control window apply to push promises ?

It might not have any effect given that flow control only affects
payloads and there isn't likely to be a payload.

I agree that the DELETE response is redundant, but it's not an
inordinate burden.

---

From: Costin Manolache <[email-masked]>
Date: Wed, 27 Apr 2016 14:50:33 +0000

> payloads and there isn't likely to be a payload.

That was my understanding as well - TCP window may help a bit, but there is
little feedback for load balancing, and quite tricky to use it right by AS.

> I agree that the DELETE response is redundant, but it's not an

Agreed - it does affect the throughput, but more connections can handle
this.

Costin

---

From: Brian Raymor <[email-masked]>
Date: Wed, 27 Apr 2016 17:22:04 +0000

On Tue, Apr 26, 2016 at 9:21 PM Martin Thomson <[email-masked]> wrote:
...
> DELETE (just like the client uses DELETE to acknowledge)?

To clarify, you're proposing that the application server delete the same push message resource that was deleted by the user agent to acknowledge receipt of the message? If so, something like:

If the push service receives the acknowledgement and the application
has requested a delivery receipt, the push service MUST return a 204
(No Content) response to the application server monitoring the
receipt subscription.

To ensure that a receipt is properly delivered to the application
server at least once, the application server MUST acknowledge receipt
of the response by performing a HTTP DELETE on the corresponding 
push message resource.

If the push service does not receive the acknowledgement within a
reasonable amount of time, then the receipt is considered to be not
yet delivered. The push service SHOULD continue to retry delivery of
the receipt until the advertised expiration of the corresponding push
message.

---

From: Costin Manolache <[email-masked]>
Date: Wed, 27 Apr 2016 19:42:20 +0000

The main purpose of the DELETE ( from my perspective ) is to prevent
PS flooding the AS, i.e. to provide a form of flow control. Secondary
reason is to provide more info for load balancing the receipts across
AS connections.

Having retry on the receipt would be expensive - I think it would
be better to not promise that, not sure about other PS but for us it
may be tricky. Retry if a AS connection goes down, or it doesn't ack
is ok - but not for the entire duration of the message TTL,
and without reliability guarantees, i.e. without database storage.

Costin

On Wed, Apr 27, 2016 at 10:22 AM Brian Raymor <[email-masked]>
wrote:

> To clarify, you're proposing that the application server delete the same
> push message resource that was deleted by the user agent to acknowledge
> receipt of the message? If so, something like:
>

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 10:11:50 +1000

> reason is to provide more info for load balancing the receipts across

Hmm, I just realized that h2 is pretty bad at this specific use case.
If all you send is headers (which is true in this case), then flow
control doesn't kick in.  Also, pushes aren't subject to the
concurrent stream limit because they don't require client interaction.

---

From: Brian Raymor <[email-masked]>
Date: Thu, 28 Apr 2016 00:29:42 +0000

> concurrent stream limit because they don't require client interaction.

PS can already rate control an AS:

A push service MAY return a 429 (Too Many Requests) status code
[RFC6585] when an application server has exceeded its rate limit for
push message delivery to a push resource. The push service SHOULD
also include a Retry-After header [RFC7231] to indicate how long the
application server is requested to wait before it makes another
request to the push resource.

We could include outstanding receipts in the guidance for this case?

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 10:34:31 +1000

> PS can already rate control an AS:

This was about the reverse, having rate control on the delivery of
receipts to the AS.  It's possible that a PS could overwhelm an AS
with these.

---

From: Brian Raymor <[email-masked]>
Date: Thu, 28 Apr 2016 00:51:40 +0000

> receipts to the AS.  It's possible that a PS could overwhelm an AS

I understand - my point is that the AS is creating its own problem. It's requesting
too many push messages with receipts and then failing to scale when receiving the
receipts from the PS. In this case, the AS needs to be encouraged by the PS to slow
down its rate of requests that are generating those receipts.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 28 Apr 2016 11:18:06 +1000

> down its rate of requests that are generating those receipts.

It's not that easy.  An AS could send messages out over several time
and still be flooded with receipts in a sudden spike.

---

From: Costin Manolache <[email-masked]>
Date: Thu, 28 Apr 2016 02:20:21 +0000

Yes, traffic is not uniform. For example if one network has an outage - a
lot of saved messages will
be delivered when the network recovers, resulting in a spike of receipts,
which may bring down
some AS servers - with cascading effect. And very little they can do about
it.

Also it's hard for PS to load balance without some feedback. Delete would
provide the feedback
and support flow control - at least if it is sent on the same h2
connection. If it's send on a different
connection - who knows where it'll be load balanced, and it gets quite
tricky to correlate.

One option would be to have multiple GET requests on the receipt resource -
each would get
a MAX_CONNECTION promises (receipts), after that the AS would need to make
another GET.  They can also have 2-3 concurrent GET requests if they want,
and manage
the flow by (outstanding GET * max_receipts_per_get).

May still lose some receipts (no confirmation), but ok if spec is clear
that receipts are best
effort.

Costin

---

From: Brian Raymor <[email-masked]>
Date: Thu, 28 Apr 2016 04:40:08 +0000

+Herve - to bring to his attention.

> One option would be to have multiple GET requests on the receipt resource - each would get 
> another GET.  They can also have 2-3 concurrent GET requests if they want, and manage

To track, I opened - https://github.com/webpush-wg/webpush-protocol/issues/87.

Martin and I discussed earlier this evening. I suggested a similar approach to what you're proposing - the ability for the AS to include a header field in its monitor request to indicate a maximum number of receipts. Martin then brought to my attention Herve's related work on Push-Policy:

https://www.ietf.org/proceedings/94/slides/slides-94-httpbis-2.pdf
https://tools.ietf.org/pdf/draft-ruellan-http-accept-push-policy-01.pdf

It seems more appropriate for HTTPbis to address policies for push promises rather than creating a special case in WebPush. Herve - what are your thoughts about including this scenario in your proposal?

...Brian

---

From: RUELLAN Herve <[email-masked]>
Date: Fri, 29 Apr 2016 11:21:44 +0000

> -----Original Message-----
> From: Webpush [mailto:[email-masked]] On Behalf Of Brian
> Raymor
> Sent: jeudi 28 avril 2016 06:40
> To: Costin Manolache <[email-masked]>; Martin Thomson
> <[email-masked]>; RUELLAN Herve
> <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Webpush] Receipt subscription follow-up
> 
> > One option would be to have multiple GET requests on the receipt
> > resource - each would get a MAX_CONNECTION promises (receipts), after
> > that the AS would need to make another GET.  They can also have 2-3
> > concurrent GET requests if they want, and manage the flow by
> (outstanding GET * max_receipts_per_get).
> 
> Martin and I discussed earlier this evening. I suggested a similar approach to
> what you're proposing - the ability for the AS to include a header field in its
> monitor request to indicate a maximum number of receipts. Martin then
> brought to my attention Herve's related work on Push-Policy:
> 
> It seems more appropriate for HTTPbis to address policies for push promises
> rather than creating a special case in WebPush. Herve - what are your
> thoughts about including this scenario in your proposal?

I think this is adding a new interesting dimension to my proposal. 
It can be integrated as a new Push-Policy. This is easy to do, but would have limitations: it would not be possible to combine this maximum number of resources to push with other Push-Policies such a Head (pushing only HEAD responses) or Fast-Load.
So a better solution would be to define a new Push-Policy for limiting the number of resources that can be pushed, and in addition to extend the Accept-Push-Policy and Push-Policy headers to be able to combine several policies.

Hervé