Subject: [codesprints] Outage on the way

From: Glen Barney <[email-masked]>
Date: Mon, 11 Apr 2011 11:56:07 -0700

Dear IETF Leadership -

As you will hopefully remember, at IETF80 it was decided to proceed with a server upgrade and failover test, and the dates and details were announced by Russ on the IETF-ANNOUNCE list.

This is just a reminder that the process begins today, in approximately one hour.

We will bring the old server down into a standby mode (the server will be up but services like web, FTP, mail and rsync will be halted), do a final controlled synchronization of data from the old server in Fremont to the new server in Orlando, update the zone files, and then bring the new server online.

We will then operate for approximately two weeks out of Orlando, during which time we will correct any problems.  

During the latter half of the period we will prepare a new, identical (clone) server here in Fremont, and, on Monday April 25th we will fail back to Fremont using the same procedure.

Please remember - although I *hope* for no problems, I always *expect* that there will be problems.  We will be monitoring things from here, and the new server will have actual monitored coverage for 22 hours a day during the few days after each failover, just to make sure things work.  Please report problems in the usual manner (to IETF-ACTION).  If you feel the need, you can email me directly out of band at [email-masked].

Thank you for your patience as we work to test our failover facility, and implement these upgrades to improve the services we provide to the IETF.

Glen
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Glen Barney <[email-masked]>
Date: Mon, 11 Apr 2011 15:36:29 -0700

Dear IETF Leadership -

I'm very pleased to report that the failover process is completed, and everything at this stage appears to be working well.

We've tested the website, datatracker, ticketing system, mail system, mailman (you're seeing this message), and other systems, and we appear to be functioning well.

Of course, we will continue to monitor and correct problems as they're found.

In the meantime, expect some slowness over the next few hours as backlogged mail from around the world gets sent in and processed.

By tomorrow at this time, that flow should be caught up and all should be well.

As always, we will be monitoring, and any questions, please let us know!

Glen

---

From: Glen Barney <[email-masked]>
Date: Mon, 11 Apr 2011 17:33:02 -0700

Dear IETF Leadership -

Just a follow-up message regarding this failover.

As you will recall, this failover had a dual purpose of testing both the
Orlando facility, and also deploying us to an upgraded server, consisting
of new operating system and software versions.

I'm pleased to report that everything went exactly as planned.  We were
down for 2 hours and 3 minutes, and successfully executed the failover 
with no immediate issues apparent.  All systems appear to be working at
this stage, and things look good.

We are noting a bit of slowness at this stage, due to the heavy load of
traffic coming in which is resulting in high utilization levels at the
Orlando facility.  We will monitor this overnight, and increase our 
bandwidth allocation there if it becomes necessary to do so.

As I said previously, we will continue to monitor.  Our fail back is 
scheduled for Monday April 25th, and that should be equal in effect,
with even fewer bumps, since we will be failing back to an identical 
clone of the upgraded server we are now operating on.  Approximately
45 days thereafter, our new higher-bandwidth circuit will be installed
in Fremont, resulting in even further operating improvements.

Thank you all for your support, as always!

Now it's time for some birthday cake!  :-)

Glen

---

From: Bob Hinden <[email-masked]>
Date: Tue, 12 Apr 2011 08:43:59 -0700

Glen,

Glad to hear it went well!  

Thanks,
Bob

On Apr 11, 2011, at 5:33 PM, Glen Barney wrote:

> _______________________________________________
> IAOC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/iaoc

---

From: Glen Barney <[email-masked]>
Date: Tue, 12 Apr 2011 10:09:56 -0700

It did, thanks!  Amazingly well, actually!

Thanks for the support!

Glen