Subject: [Webpush] Analysis of performance impact of disaggregation

From: Martin Thomson <[email-masked]>
Date: Wed, 1 Apr 2015 16:18:26 -0700

One of the concerns raised about Microsoft's disaggregation proposal
was the performance impact it implies.  Since connections and the
state tied to them is volatile state, being able to efficiently repair
that state could be important.  Maybe some numbers might help here,
even if they are rough and based on some speculation.

For a server that chooses to operate as proposed and provide
independent subscriptions, this means that clients need to tell the
server about every subscription they are interested in after a
connection is restored.

If I'm right about this much, the choice over whether disaggregation
or aggregation is largely driven by costs at this point.  That means
understanding the costs.

## Costs for aggregation

An aggregating server needs a way to identify all undelivered messages
across all subscriptions for the registration.

I originally envisaged an architecture where this is done by
aggregating messages into a single persistent store that was common
across the registration.  This reduces the cost of session restoration
to connection establishment, plus one request, and then a database
lookup.

An aggregate on push design like this doesn't necessarily require a
server to explicitly maintain a list of subscriptions for a
registration, servers only need a way to identify the registration
based on the subscription URL.  I can go into more detail on how that
might work if people don't think this is possible.

Elio pointed out that their delivery architecture kept subscriptions
independent of each other and that aggregation would force them to
store additional information identifying active subscriptions.
That is, aggregate on pull.

Session restoration for an aggregate-on-pull architecture would entail
querying the registration store, then querying each of the listed
subscriptions independently.  Obviously, that imposes additional state
management overhead on the server, as well as having the cost and
delay of querying each subscription once the connection is online.
The only cost to the user agent is a potential delay in message
delivery after a connection break.

## Costs for disaggregation

A server without any aggregation doesn't need to maintain any
information about the correlation of subscriptions.  So the concerns
raised about aggregation are ameliorated.

The cost here is in the time that it takes to tell the server about
all of the different subscriptions you are interested in.  In HTTP/2,
that is largely driven by the size of the path component of the URL,
the rest can be aggressively compressed down to a handful of bytes
(let's say 16, including the frame header).  I'm assuming the absence
of bearer tokens for authentication, which might be used here at a
significantly increased cost in bytes.

The big latency cost comes in when the size of requests exceeds the
TCP CWND, which is probably still small at this point.  Assuming TLS
false start, that CWND is shared with the TLS CCS and Finished and the
HTTP/2 preface, which might amount to 100 bytes total.

The dominant factors here are therefore how many subscriptions and how
long is the URL for each.  Here's where the assumptions start to build
up.

I don't have good stats on this, but I'm going to suggest that you
might want to have 100 subscriptions active without performance
degradation (I expect the real number to be much lower than this in
practice, but outliers are where complaints originate).

URLs can be quite long, depending on what is packed into them.
Microsoft documentation
(https://msdn.microsoft.com/en-us/library/windows/apps/xaml/hh868252.aspx)
shows an example with a 159 octet path.

If that length is representative, 100 subscriptions will consume 17.5k
plus overheads.  Transmitting that data takes an insignificant amount
of time, but that amount exceeds the CWND for the second round trip of
TCP on common MSS values short of INIT_CWND 7.  If the CWND is
exceeded, then it takes an additional round trip to complete the
setup.  The example also includes a 62 byte bearer token sent by the
application server; I don't know if that is needed here or not for the
user agent.

I don't know what the current status is, but last time I checked
INIT_CWND 4 was the value that was in widespread use.  That means less
than 64 subscriptions.

If you wanted to be quite aggressive about reducing the size of the
path and are able to make some architectural changes, it should be
quite feasible to construct a subscription path that is only 45
characters in base64.  At that point, I think that we can discount the
CWND issue.

## Allowing choice

I think that I'd be disappointed if we allowed the push service to
pick here. That makes it harder to build a single client.  But it
might be OK given the need for some sort of prior relationship with
the push service for authentication purposes.  I don't think that this
is accepting the judgment of solomon, but I think we need to consider
the complexity costs of that option.