Subject: [TOOLS-DEVELOPMENT] Guidance on When to Upgrade Server Software

From: Russ Housley <[email-masked]>
Date: Wed, 11 Jul 2018 09:48:09 -0400

During the tools call yesterday, we talked about writing some guidance on this topic.  Please review and comment.

Russ

= = = = = = = = 

Guidance: When to Upgrade Server Software

There are many reasons to upgrade the software on the IETF servers,
including bug fixes, access to new features, performance improvement.
Yet, the software update needs to be done a at time that is going to
have minimal impact on the IETF community.  The impact might be the
result of a service outage while the upgrade take place, or it might
be the result of an unexpected problem with the newly installed
software.  This guidance is intended to avoid outages when the
IETF community is depending on the IETF servers the most.

Guidelines

  Do not make updates to any major server software the week before
  or during the IETF meeting.

  Do not make updates to the IETF Datatracker on Wednesday or
  Thursday of the IESG formal telechat.

  Do not make updates to the IETF Datatracker a few days prior to
  the I-D cut-off date for an IETF meeting.

Exceptions

  If something critical is broken, fix it.

  If a new feature will make things work more smoothly for the IESG or
  Secretariat, ask.  If they want it now, give it to them now.

Last update: 10 July 2018

---

From: Bob Hinden <[email-masked]>
Date: Wed, 11 Jul 2018 08:14:51 -0700

Looks good to me.

Sorry for missing yesterday’s call.

Bob

> On Jul 11, 2018, at 6:48 AM, Russ Housley <[email-masked]> wrote:
> 
> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Ben Campbell <[email-masked]>
Date: Wed, 11 Jul 2018 11:24:36 -0500

LGTM.

Ben.

> On Jul 11, 2018, at 8:48 AM, Russ Housley <[email-masked]> wrote:
>

---

From: Russ Housley <[email-masked]>
Date: Thu, 12 Jul 2018 09:14:13 -0400

I have not received any suggestions to change the text.  Where is the best place to post this?

Russ

---

From: Lou Berger <[email-masked]>
Date: Thu, 12 Jul 2018 09:33:36 -0400

Hi Russ,

Thanks for this. Two comments below

----------

> I have not received any suggestions to change the text.  Where is the best 

I suggest including the mail server and archive infrastructure at this time 
as well, given the typical male load during that time. This period Should 
probably be expanded to include the working group agenda upload deadlines 
as well.

>>  Secretariat, ask.  If they want it now, give it to them now.

How about s/want / deem it sufficiently critical to do

Thanks

Lou

---

From: Lou Berger <[email-masked]>
Date: Thu, 12 Jul 2018 12:24:08 -0400

PS sorry about the humorous voice to text typos...

---

From: Russ Housley <[email-masked]>
Date: Thu, 12 Jul 2018 18:47:08 -0400

Addressing Lou's comments.

Russ

= = = = = = = = = =

Guidelines

   Do not make updates to the IETF Datatracker a few days prior to
   the IETF meeting preliminary and final agenda publication dates.

   Do not make updates to the IETF Datatracker a few days prior to
   the working group agenda publication deadline for an IETF meeting.

Exceptions

   If a new feature will make things work more smoothly for the IESG or
   Secretariat, ask.  If they deem it sufficiently urgent, give it to
   them now.

Last update: 12 July 2018

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 13 Jul 2018 04:50:32 +0200

Comments inline:

> Addressing Lou's comments.

So far this makes some sense to me, but I'd actually limit this to
say (in order to provide stability in what the IESG see around a
telechat, not because the danger of downtime):

>    Do not make updates to the IESG-related parts of the IETF
>    Datatracker on Wednesday or Thursday of the IESG formal telechat.

And then follow up with:

     Do not make any updates to the IETF Datatracker within a few
     hours of the IESG formal telechat.

(as it could coincide with an AD energetically working through some
document issue).

>    the I-D cut-off date for an IETF meeting.

A few hours would make more sense.  It takes one single command to
roll back a release, and I think we've only had to roll back twice over
the 12 years we've made releases.  The downtime when making a release
are a matter of seconds.

>    the IETF meeting preliminary and final agenda publication dates.

I don't understand this at all.  We've routinely done this during the
last 20 meetings.  This feels like a completely gratuitous limitation.

>    the working group agenda publication deadline for an IETF meeting.

I have a hard time understanding this, too.  What incident (that hasn't
happened, ever) are we trying to prevent with this?

	Henrik

---

From: Russ Housley <[email-masked]>
Date: Fri, 13 Jul 2018 15:52:23 -0400

See below ...

[snip]

> (as it could coincide with an AD energetically working through some

Yes, stability is the concern.  I know that many ADs put their ballots into the datatracker one the day before the telechat or the morning before the telechat depending on their time zone.  As a result the ADs are using the datatracker heavily on these days.

I guess we should change the wording to accommodate a telechat that takes place on a day other than Thursday.

> the 12 years we've made releases.  The downtime when making a release

I realize the downtime to make a release is not the issue, but the time to realize that things are not working as expected and then rollback is the problem.  Again, the point is stability during a known period of heavy use.

> last 20 meetings.  This feels like a completely gratuitous limitation.

Again, the point is stability during a known period of heavy use.  We know the Secretariat is tinkering with the agenda to get it published.

> happened, ever) are we trying to prevent with this?

Again, the point is stability during a known period of heavy use.

Russ

---

From: Russ Housley <[email-masked]>
Date: Fri, 13 Jul 2018 18:42:34 -0400

I have been thinking about this some more.  I think we need to balance the desire for continuous incremental improvement and the recognition that the datatracker is now vital to so many aspects of IETF life.  Where is that balance point?

Russ

> On Jul 13, 2018, at 3:52 PM, Russ Housley <[email-masked]> wrote:
> 
> See below ...
> 
> 
> [snip]
> 
>>> 
>>> Guidelines
>>>

---

From: Robert Sparks <[email-masked]>
Date: Fri, 13 Jul 2018 20:49:51 -0500

I think we need to recognize that our test suite is relatively strong and our experience with the tracker to date gives us a good feel for what would be a low risk vs medium risk release. ( i posit that the days of a high risk release are _mostly_ behind us modulo big infrastructure or model changes). [*]

I was comfortable with the original draft, but if we want to share our thinking more (that us what this exercise primarily is about, I hope) we can write in more about using good judgement and being aware of time sensitive activities. Hard rules wont match reality (or may actually impede the greater good for the community). 

Perhaps a recasting that is more advisory vs prescriptive would serve future folks better?

Again, I’m not uncomfortable with the original formulation as I expect ‘doing the right thing’ to win out at any given point in time. 

RjS

[*] I think the mail infrastructure changes we ate going through and planning to go through are much more risky than any current datatracker improvements (consider mail disruption on a telechat morning) and we seem to be getting unleashing those reasonably right. 

RjS

Composed on my iPhone while on a tiny bouncy plane. 

> On Jul 13, 2018, at 5:42 PM, Russ Housley <[email-masked]> wrote:
>

---

From: Ben Campbell <[email-masked]>
Date: Sat, 14 Jul 2018 09:57:21 -0400

I am also okay with the original draft.

I think I’m agreeing with Robert, but I encourage people to avoid overthinking this. It doesn’t need to be perfect or even all that precise.

It’s really a principle-of-least-surprise matter. I think it’s good enough to point out the sort of events that we want to avoid disruption for, try not to disrupt them unless there is a really good reason, and otherwise offer a fair amount of discretion.

Ben.

> On Jul 13, 2018, at 9:49 PM, Robert Sparks <[email-masked]> wrote:
>

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 15 Jul 2018 08:54:50 -0400

> On 11 Jul 2018, at 11:14, Bob Hinden <[email-masked]> wrote:
> 

Me too.