Subject: [Tools-implementation] Revisiting whether we should continue using Docker as we currently do.

From: Robert Sparks <[email-masked]>
Date: Wed, 9 Sep 2020 11:21:03 -0500

A few weeks ago, shortly after the yc.o team's work managed to crash the 
host they were working on through changing settings in a container, I 
proposed that we unroll the things we currently have in containers on 
production. At the time, Glen suggested we not do that. I'd like to ask 
the question again - I think we may have more to consider.

1. I remain uneasy about docker's implementation on OpenSuse. The 
container crash above, the issues we've run into with containers locking 
(and sometimes causing processes talking to them like apache) to hang 
are suspicious. That we've not been able to pin down what's really going 
on suggests to me the issue is in a place we can't really look, inside 
docker's interstitial networking or filesystem abstraction code perhaps.

2. Many of the containers we have (and in particular the one for the 
website) really need to be designed differently if they are going to 
remain deployed as containers. The amount of file-system mapping they do 
is not what the docker architects expect as a normal use-case. Mapping 
sockets in the way we do is also likely not something they focus on testing.

3. Docker is making Glen uncomfortable and the benefit for him 
(operationally) of the containerization is not proportional to the extra 
problems it is bringing.

So I again suggest that we unroll for the production deploys, at least 
for now. I think we can unroll everything at this point, but there might 
still be a hitch in unrolling the trac instances. Henrik - could you 
remind me what our thinking was with respect to those?

I do plan to keep up the pressure to have containerized versions of 
these services - this isn't a call to abandon Docker - but I suggest we 
need to change how we're currently using it.

RjS

---

From: Glen <[email-masked]>
Date: Wed, 9 Sep 2020 10:51:47 -0700

My full support and agreement here.  To clarify:  I suggested *not* 
doing it only because I didn't want to be the cause of heartburn or the 
loud voice whining.  :-)  But these arguments are persuasive, and I 
agree with everything said here, both in terms of present *and* future 
planning.

Glen

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 9 Sep 2020 19:55:58 +0200

Hi Robert,

> remind me what our thinking was with respect to those?

Since we're now running apache with a python 3.6 mod_wsgi, we cannot run
Trac under mod_wsgi until it's available for Python 3.  (There's progress,
but it's not there yet).  Which means moving it to nginx if we're not going
to use Docker.

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Wed, 9 Sep 2020 13:08:02 -0500

> but it's not there yet).  Which means moving it to nginx if we're not going
Glen mentioned possibly using mod_wsgi-express to run a separate apache 
instance that loaded the 2.7 module rather than the 3.6 module?

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 9 Sep 2020 20:28:56 +0200

> instance that loaded the 2.7 module rather than the 3.6 module?

I don't think you can run 2 Apache instances with different mod_wsgi modules
without containerizing one of them?

	Henrik

---

From: Russ Housley <[email-masked]>
Date: Wed, 9 Sep 2020 16:02:17 -0400

One of the reasons that we went to containers was to isolate some python2 from python3.  Will we just use virtual environment for that?

Russ

> On Sep 9, 2020, at 12:21 PM, Robert Sparks <[email-masked]> wrote:
> 
> A few weeks ago, shortly after the yc.o team's work managed to crash the host they were working on through changing settings in a container, I proposed that we unroll the things we currently have in containers on production. At the time, Glen suggested we not do that. I'd like to ask the question again - I think we may have more to consider.
> 
> 1. I remain uneasy about docker's implementation on OpenSuse. The container crash above, the issues we've run into with containers locking (and sometimes causing processes talking to them like apache) to hang are suspicious. That we've not been able to pin down what's really going on suggests to me the issue is in a place we can't really look, inside docker's interstitial networking or filesystem abstraction code perhaps.
> 
> 2. Many of the containers we have (and in particular the one for the website) really need to be designed differently if they are going to remain deployed as containers. The amount of file-system mapping they do is not what the docker architects expect as a normal use-case. Mapping sockets in the way we do is also likely not something they focus on testing.
> 
> 3. Docker is making Glen uncomfortable and the benefit for him (operationally) of the containerization is not proportional to the extra problems it is bringing.
> 
> So I again suggest that we unroll for the production deploys, at least for now. I think we can unroll everything at this point, but there might still be a hitch in unrolling the trac instances. Henrik - could you remind me what our thinking was with respect to those?
> 
> I do plan to keep up the pressure to have containerized versions of these services - this isn't a call to abandon Docker - but I suggest we need to change how we're currently using it.
> 
> RjS
> 
> 
> -- 
> Tools-implementation mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-implementation

---

From: Robert Sparks <[email-masked]>
Date: Wed, 9 Sep 2020 15:04:20 -0500

On 9/9/20 3:02 PM, Russ Housley wrote:
> One of the reasons that we went to containers was to isolate some 
> python2 from python3.  Will we just use virtual environment for that?

That's the essence of the question below about trac. Trac may need to 
remain in a container, or move to a separate vm, or something, though 
we're continuing looking closely at whether we can avoid either.

>
> Russ
>
>> On Sep 9, 2020, at 12:21 PM, Robert Sparks <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> A few weeks ago, shortly after the yc.o team's work managed to crash 
>> the host they were working on through changing settings in a 
>> container, I proposed that we unroll the things we currently have in 
>> containers on production. At the time, Glen suggested we not do that. 
>> I'd like to ask the question again - I think we may have more to 
>> consider.
>>
>> container crash above, the issues we've run into with containers 
>> locking (and sometimes causing processes talking to them like apache) 
>> to hang are suspicious. That we've not been able to pin down what's 
>> really going on suggests to me the issue is in a place we can't 
>> really look, inside docker's interstitial networking or filesystem 
>> abstraction code perhaps.
>>
>> remain deployed as containers. The amount of file-system mapping they 
>> do is not what the docker architects expect as a normal use-case. 
>> Mapping sockets in the way we do is also likely not something they 
>> focus on testing.
>>
>> (operationally) of the containerization is not proportional to the 
>> extra problems it is bringing.
>>
>> So I again suggest that we unroll for the production deploys, at 
>> least for now. I think we can unroll everything at this point, but 
>> there might still be a hitch in unrolling the trac instances. Henrik 
>> - could you remind me what our thinking was with respect to those?
>>
>> these services - this isn't a call to abandon Docker - but I suggest 
>> we need to change how we're currently using it.
>>
>> RjS
>>
>>
>> -- 
>> [email-masked] <mailto:[email-masked]>