Subject: [Tools-implementation] Deploying trial instances of matrix and zulip - an initial plan

From: Robert Sparks <[email-masked]>
Date: Thu, 13 Aug 2020 09:57:09 -0500

Here is a barebones plan proposal. Please bash it.

To gain some experience (and shared community understanding) of matrix 
and zulip, we will run these services in a trial configuration for a 3 
month period that includes IETF109. We will tell the community to expect 
that these services will be taken completely down (they should expect 
all history, configuration, and other features to be lost) at the end of 
those 3 months, and if it is reasonable to provide longer lived 
instances of these services, new ones will be set up in a different 
location.

We (whole team) read the intro material about matrix and zulip and send 
email here if we see potential problems.

     https://matrix.org/

     https://zulipchat.com/

In particular, we need a quick, shared understanding of what logging is 
going to look like for each of these systems, particularly logging of 
1-1 conversations so we can inform the community what to expect. (Note 
that slack _does_ retain all conversations).

I will start asking the people who have indicated they _know_ and are 
running these servers (mhoye, tim april) for advice on initial setup, 
including how to set up bridges.

Glen deploys instances with as little extra configuration as is 
reasonable at

     matrix-trial1.ietf.org

     zulip-trial1.ietf.org

Note the 1.

Glen then sets up the bridges, in some configuration (hopefully a 
star-configuration from a service we're willing to keep) between slack, 
zulip, matrix, and if possible, jabber.

Proposed schedule:

* now-20Aug: Team has read intro material and raises flags by email

* now-20Aug: Robert consults with people who have experience running 
these servers

* Week of Aug 31- Sep 4: Glen deploys and sets up bridges. We announce 
availability to the community as soon as it's ready.

(Note that IETF 109 is 14Nov-20Nov +/- week before bleed).

* Week of Nov 30 - Dec 4: We review what we've learned and what the 
community says.

* Week of Dec 7 - Dec 11: trial1 instances are destroyed, and as 
warranted longer lived instances are set up.

Variants:

* If bridging fights back, we deploy silos the week of Aug 31 - Sep 4 
and work on bridging incrementally during the trial period.

* We include hosting jabber accounts in this as well.

---

From: Glen <[email-masked]>
Date: Thu, 13 Aug 2020 09:22:28 -0700

> Here is a barebones plan proposal. Please bash it.

/me pulls out huge stone-age-style club for bashing...  ;-)

> instances of these services, new ones will be set up in a different

This is good, and necessary, but I wondered when I read this if
emphasizing the "you only have 90 days" thing wouldn't be a little
de-motivating in terms of getting people to try it.  When we reach the
point of announcement, I'd recommend a heavy positive spin, in
whatever way it makes sense.  (Unless we *prefer* to stay on Jabber?
I've given up trying to understand what the community wants on any
given topic, and just fall back to John Levine's comment:  No matter
what we do, someone will be unhappy.)   But if we're going to go to
this effort, I'd think we'd *want* people to jump in en masse and
play, so, I don't know, might it make sense that instead of saying "To
gain shared experience" we say something like "As we investigate ways
of improving communication" or "moving beyond Jabber" or whatever.  I
don't know, it's just semantics, but it stuck in my head, so I'm
mentioning it.

> Glen deploys instances with as little extra configuration as is

My pleasure.

> Note the 1.

It took me a minute to figure out what "the 1" was.  I had flashbacks
to various movies, and looked for non-existent footnotes.  :-)
Numeral in subdomain noted.

> and work on bridging incrementally during the trial period.

Right.  IIRC you indicated that the bridges might provide the most
significant challenges, rather than just the standalone deployment of
the servers themselves.  I haven't looked at bridges myself.  But
maybe it makes sense to set up the standalone servers in a separate
phase, and then deal with the bridges thereafter.   Maybe something
like:

Week of Aug 24:  Glen deploys and sets up basic standalone
installations of Matrix Zulip
Week of Aug 24:  Robert/Team consults/investigates bridging methods
and challenges specifically
Week of Aug 31:  Glen deploys and sets up bridges

Or whatever.  Availability and loading is always fluid, so we may be
inadvertently delayed anyway, but the point is I'd rather start on
this sooner and "go with what we know" first, giving us more time for
challenges of any type (briding, schedules, availability, etc.) moving
forward.

That's all I had,
Glen

Glen

---

From: Robert Sparks <[email-masked]>
Date: Thu, 13 Aug 2020 14:02:37 -0500

I had a real-time chat with Henrik, where conversation uncovered another 
possible approach.

I could ask the people who know these things already to set up these 
trial instances themselves, for us, wherever they want (we could aim 
ietf domain names at them). They can take what they learn and recommend 
configurations for us for what we moved into a longer-term mode of support.

Is that something I should pursue?

RjS

---

From: Russ Housley <[email-masked]>
Date: Thu, 13 Aug 2020 15:17:50 -0400

I like the bottom-up approach.  I guess we will find out who really cares.

Russ

> On Aug 13, 2020, at 3:02 PM, Robert Sparks <[email-masked]> wrote:
> 
> I had a real-time chat with Henrik, where conversation uncovered another possible approach.
> 
> I could ask the people who know these things already to set up these trial instances themselves, for us, wherever they want (we could aim ietf domain names at them). They can take what they learn and recommend configurations for us for what we moved into a longer-term mode of support.
> 
> -- 
> Tools-implementation mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-implementation

---

From: Glen <[email-masked]>
Date: Thu, 13 Aug 2020 13:06:53 -0700

I am opposed to this approach, because it puts me in the same position
we're in with Codimd, and other items.

My opinion is that if the IETF is running something, the IETF should
run it.   The IETF spent over a decade arguing about and agreeing the
need to host mission critical tools on the actual IETF servers
operated by the secretariat.  They came up with a complex plan to move
all mission critical tools to the IETF servers.   We are still not
there.

If that is still the vision, then the IETF should not, IMO, "go
backwards" and start having random people elsewhere host things.

If the IETF's vision has changed, then of course I withdraw my
comments, but that would be news to me.

Glen

---

From: Robert Sparks <[email-masked]>
Date: Thu, 13 Aug 2020 15:31:13 -0500

Would your opposition be different if the domain was not ietf.org?

(c.f. ietf.slack.org).

---

From: Glen <[email-masked]>
Date: Thu, 13 Aug 2020 13:54:38 -0700

Mmmmm.  I don't know.

I'm thinking this over as I type this.  First, to be clear, I'm
seriously not trying to power grab here.  Nothing makes me happier
than having an *empty* "to-do" list.  Also  I am *attempting* to
sanitize this response to be neutral in tone and non-offensive, so
please forgive any language that I might miss.

That said, I think that what I AM trying to do is avoid any situation in which:

(a) Someone expects that the secretariat IT team can fix something,
which it turns out they cannot fix (just as an example of my thinking:
Something like CodiMD - or tools - I still get tickets into IETF
Action at times for tools.ietf.org issues - and I still sometimes get
nasty comments when I refer people to Henrik for that.)

-or-

(b) Someone asks (in essence), "We're paying for all these secretariat
things, why are they not running this?  Are they just not able to do
it?"  (which might not happen much in the real world, but in *this*
community I've already had that question asked a couple of times in
the past.)

Both of these are, I suppose, reputational things more than anything
else - but in this case, I'm not really worried about my own
already-destroyed reputation, I'm much more worried about AMS actually
doing the job that it is - by contract - supposed to be doing - which
is to operate the IT-related services of the IETF.  And both of those
concerns are real, and based on real situations that occur from time
to time.

In the case of your cited example, Slack, that's a third-party service
provided and hosted by a company.  I don't know anything about it, I
never even knew it had been set up until recently - and it's clear
both by the domain *and* by the nature of Slack that "the IETF" has
nothing to do with its operation or maintenance.  (I suppose the
*administration* of the Slack channel/zone should probably be a
secretariat thing, and it may already be, I don't know, that's not my
area - but the actual operation and hosting of the service is clearly
not an IETF thing, so clearly not my issue.)

But what we're talking about here is setting up and operating open
source software "as the IETF."  I think it's something we should do
in-house.  We're already bandwidth constrained (I have no idea when
our schedules will combine to let us migrate CodiMD to a
secretariat-managed location), and so repeating that situation by
having other people set up and host services we want elsewhere seems
counterproductive to me.  Even if we call it a test, it still triggers
both my concerns (a) and (b) above, and so I would still be
uncomfortable with it.

And I think that polluting the namespace further by putting it under
another domain like "ietftest.org" or whatever would be worse.

Now if, OTOH, this software were being set up and hosted by their
providers (i.e. "ietf.codimd.org" or whatever), then that might be a
different thing, and if that's what you mean (i.e. having each of
their commercial service arms set up and host the software on their
platforms) then I think I would have no opposition.

But just having "some person" "out there" "host it somewhere" (my
words only, but what I think we're talking about here) seems wrong to
me, and seems to be the opposite of all that we have been trying to
accomplish over the years.  Even as a test.  (Or maybe especially as a
test:  If one of these took off, it would be much more work to address
migration (and safety) of an existing implementation than it would to
set it up again fresh.)

So for example I have no opposition to GitHUB.   But if there was an
interest in running an IETF-managed GitLAB instance, well, I would
still feel like it's something I should take on, even if it means more
work for me.   And as a final note:  I have no opposition to public
cloud (if the IETF is paying) vs private cloud (which AMS provides
free as a part of its services), I don't care where it's deployed, I
care only about being able to do my job in the way that the community
expects.

Glen

---

From: Robert Sparks <[email-masked]>
Date: Mon, 17 Aug 2020 10:47:51 -0500

To focus on one tidbit: (trimming massively):

On 8/13/20 3:54 PM, Glen wrote:

<trim/>

> If one of these took off, it would be much more work to address
<trim/>
>>>>>>> We will tell the community to expect

I think it's important to say that there will be a point where there's a 
clean restart after we get some experience, and that we set the 
expectation that there _won't_ be migration.

I don't think that will cause people to not try things. It may even make 
them more comfortable exploring, especially in exploring corners.

RjS

---

From: Glen <[email-masked]>
Date: Mon, 17 Aug 2020 08:51:25 -0700

> them more comfortable exploring, especially in exploring corners.

Sounds good / +1

Glen

---

From: Robert Sparks <[email-masked]>
Date: Wed, 19 Aug 2020 10:29:36 -0500

I've asked if we can get help from some folks, but given the discussion 
with Glen, we should move forward with him doing installs.

There was some initial feedback from Richard and Rich in the slack 
#chat-tools channel recommending that we not pursue bridging. I've 
argued that letting the community experiment with it is necessary.

I also realized I made an assumption that we should call out explicitly: 
I assumed it would be _easy_ to configure each of these services to use 
the datatracker's identity service. After skimming the docs, I'm not 
sure that assumption will hold. I think we should _try_ to have that as 
part of the initial minimal configuration, but if it doesn't work out, 
we move forward with whatever native credentialing each service provides 
and try to unify later.

Here's a revised plan:

We deploy each for 3 months as below at <service>-trial1.ietf.org, 
minimally configured , without bridging. We then work on bridging, and 
if necessary, a unified login experience, incrementally. We'll strongly 
project that people should expect to lose any setup, joined channels, 
logs, whatever at the end of the 3 months, and will need to set things 
up again with what we've learned in mind.

Revised proposed schedule:

* now-27Aug: Team has read intro material and raises flags by email
* now-27Aug: Robert continues to try to get input from people 
experienced running these servers
* Week of Aug 31 - Sep 4: Glen deploys with minimal configuration. We 
announce availability to the community as soon as it's ready.
* First two weeks of Sep: We learn about bridging and credentials.
* Dec - digest post IETF 109 community input around what does and 
doesn't work
* Jan - take the trial instances down and start any production instances 
we deci
de to go with

---

From: Glen <[email-masked]>
Date: Wed, 19 Aug 2020 08:40:32 -0700

> with Glen, we should move forward with him doing installs.

Okay.

> we move forward with whatever native credentialing each service provides

Noted.

> up again with what we've learned in mind.

Okay.

> announce availability to the community as soon as it's ready.

No conflicts on my side at this time.

Glen