Subject: Re: [Webpush] Analysis of performance impact of disaggregation

From: Costin Manolache <[email-masked]>
Date: Tue, 12 May 2015 11:17:09 -0700

I missed this email - I want to clarify that the main cost (for us) is the
lookups we have to do at connect time. Aggregation means there is
one RT to the database, hitting a single "row".

Costin

On Wed, Apr 1, 2015 at 4:18 PM, Martin Thomson <[email-masked]>
wrote:

> One of the concerns raised about Microsoft's disaggregation proposal
> was the performance impact it implies.  Since connections and the
> state tied to them is volatile state, being able to efficiently repair
> that state could be important.  Maybe some numbers might help here,
> even if they are rough and based on some speculation.
>
> For a server that chooses to operate as proposed and provide
> independent subscriptions, this means that clients need to tell the
> server about every subscription they are interested in after a
> connection is restored.
>
> If I'm right about this much, the choice over whether disaggregation
> or aggregation is largely driven by costs at this point.  That means
> understanding the costs.
>
> ## Costs for aggregation
>
> An aggregating server needs a way to identify all undelivered messages
> across all subscriptions for the registration.
>
> I originally envisaged an architecture where this is done by
> aggregating messages into a single persistent store that was common
> across the registration.  This reduces the cost of session restoration
> to connection establishment, plus one request, and then a database
> lookup.
>
> An aggregate on push design like this doesn't necessarily require a
> server to explicitly maintain a list of subscriptions for a
> registration, servers only need a way to identify the registration
> based on the subscription URL.  I can go into more detail on how that
> might work if people don't think this is possible.
>
> Elio pointed out that their delivery architecture kept subscriptions
> independent of each other and that aggregation would force them to
> store additional information identifying active subscriptions.
> That is, aggregate on pull.
>
> Session restoration for an aggregate-on-pull architecture would entail
> querying the registration store, then querying each of the listed
> subscriptions independently.  Obviously, that imposes additional state
> management overhead on the server, as well as having the cost and
> delay of querying each subscription once the connection is online.
> The only cost to the user agent is a potential delay in message
> delivery after a connection break.
>
> ## Costs for disaggregation
>
> A server without any aggregation doesn't need to maintain any
> information about the correlation of subscriptions.  So the concerns
> raised about aggregation are ameliorated.
>
> The cost here is in the time that it takes to tell the server about
> all of the different subscriptions you are interested in.  In HTTP/2,
> that is largely driven by the size of the path component of the URL,
> the rest can be aggressively compressed down to a handful of bytes
> (let's say 16, including the frame header).  I'm assuming the absence
> of bearer tokens for authentication, which might be used here at a
> significantly increased cost in bytes.
>
> The big latency cost comes in when the size of requests exceeds the
> TCP CWND, which is probably still small at this point.  Assuming TLS
> false start, that CWND is shared with the TLS CCS and Finished and the
> HTTP/2 preface, which might amount to 100 bytes total.
>
> The dominant factors here are therefore how many subscriptions and how
> long is the URL for each.  Here's where the assumptions start to build
> up.
>
> I don't have good stats on this, but I'm going to suggest that you
> might want to have 100 subscriptions active without performance
> degradation (I expect the real number to be much lower than this in
> practice, but outliers are where complaints originate).
>
> URLs can be quite long, depending on what is packed into them.
> Microsoft documentation
> (https://msdn.microsoft.com/en-us/library/windows/apps/xaml/hh868252.aspx)
> shows an example with a 159 octet path.
>
> If that length is representative, 100 subscriptions will consume 17.5k
> plus overheads.  Transmitting that data takes an insignificant amount
> of time, but that amount exceeds the CWND for the second round trip of
> TCP on common MSS values short of INIT_CWND 7.  If the CWND is
> exceeded, then it takes an additional round trip to complete the
> setup.  The example also includes a 62 byte bearer token sent by the
> application server; I don't know if that is needed here or not for the
> user agent.
>
> I don't know what the current status is, but last time I checked
> INIT_CWND 4 was the value that was in widespread use.  That means less
> than 64 subscriptions.
>
> If you wanted to be quite aggressive about reducing the size of the
> path and are able to make some architectural changes, it should be
> quite feasible to construct a subscription path that is only 45
> characters in base64.  At that point, I think that we can discount the
> CWND issue.
>
> ## Allowing choice
>
> I think that I'd be disappointed if we allowed the push service to
> pick here. That makes it harder to build a single client.  But it
> might be OK given the need for some sort of prior relationship with
> the push service for authentication purposes.  I don't think that this
> is accepting the judgment of solomon, but I think we need to consider
> the complexity costs of that option.
>
> _______________________________________________
> Webpush mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/webpush
>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 12 May 2015 12:27:50 -0700

> one RT to the database, hitting a single "row".

I suspect that this is going to be the common mode: some sort of
aggregation here saves a lot of effort.  Do you have any data to
support the idea that connection-related costs are worth keeping down?
 For instance, the number of reconnects you see per hour as a
proportion of the user base, or anything like that.  I don't want to
ask if it might be proprietary though.