Subject: [appsdir] IMPORTANT: tail-heavy process and early directorate reviews (fwd)

From: Claudio Allocchio <[email-masked]>
Date: Sat, 27 Jul 2013 15:31:45 +0200 (CEST)

Hi all,

as you can see from the message below, we are now ready to start the 
"early review" experiment which should allow us also more time for 
accomplishing reviews, and a better interaction with WGs at an earlier 
stage.

Please read carefully Jari's message, and also add your comments and 
suggestions!

In the mean time, our usual "during IETF LC" review process continues as 
normal business, while the early review process starts up (I'm to send the 
new list of drafts in LC in a few minutes).

all the best!

PS: As I'm on vacations (sending this from my house terrace on the 
seafront) I myself might be late until August 20th ;-)

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---------- Forwarded message ----------
Date: Fri, 26 Jul 2013 20:35:03 +0300
From: Jari Arkko <[email-masked]>
To: Barry Leiba <[email-masked]>,
     Pete Resnick <[email-masked]>,
     Stephen Farrell <[email-masked]>, Sean Turner <[email-masked]>,
     joel jaeggli <[email-masked]>, Benoit Claise <[email-masked]>,
     A. Jean Mahoney <[email-masked]>, Tero Kivinen <[email-masked]>,
     Claudio Allocchio <[email-masked]>
Subject: Re: tail-heavy process and early directorate reviews

Ok. So by now you should have seen that you've become a member of the dir-coord list (we changed the name). We've left TSV and OPS directorate coordinators out for now.

So here's the plan on how this will work. Please comment.

Internet Drafts sent for approval as RFCs are reviewed by individuals during the IETF Last Call, the Area Directors, IANA, as well as a number of volunteers in various directorates and review teams. The reviews from these teams has gained a significant role in ensuring that the IETF produces high-quality, understandable and implementable RFCs.

Yet, as discussed in http://www.ietf.org/blog/2013/05/balancing-the-process/ we have a general problem that quite a lot of the work around IETF documents happens at the end of the process. In particular, a number of the reviews during IETF last call point out issues that end up being raised by the IESG as comments when deciding for the final approval of the document. It is of course good that issues are caught, but raising them earlier would be better. And it would be better if the working groups - the intended focus point of work on a topic - would get to handle them, as opposed to raising these issues with the IESG. The IESG discussed these issues in its May 2013 retreat, and decided to experiment with three actions designed to move more work to the responsibility of the working groups:

(1) Perform some reviews that are now happening at IETF Last Call a bit earlier.
(2) Invite document shepherds on IESG telechats when there's a document that is likely to require discussion.
(3) When a document has a number of issues, hand over the process back to the working group, as opposed to the IESG tracking the issues. Among other things, this will ensure that changes are discussed in an open working group list and agreed through consensus.

Building quality and cross-area review to the process earlier is of course a big effort. For the first item above, we are launching an experiment to make a small change to current directorate review procedures to learn if we can move reviews a little bit earlier. If successful, this experiment will enable working groups to deal with issues before IETF Last Call and IESG review and empower the working groups to be in charge of the documents throughout their life cycle. We are also hoping that document quality will improve and number of issues discussed in the IESG will be lower.

While the number of reviews as such is not changed, some additional effort and care will however be required from the reviewers, directorate coordinators/secretaries, the working group chairs, and other participants. The experiment will show us whether this effort is reasonable and if there are any unexpected effects. The experiment is performed on a voluntary basis by each directorate. As early review requests come in, the directorates can throttle workload by either processing the requests or reverting to the existing procedures. Initially, the Gen-ART, Security Directorate, and Applications Area Directorate are included in this effort. Other directorates may be added later. The experiment will be reviewed after six months.

Care must be taken to avoid a number of possible drawbacks. There may be problematic documents that would require much re-review and effort. Similarly, working groups often perform a number of working group last calls on a document, and it would be undesirable to engage the reviewer before the document was really ready to be sent forward. And when reviewers send comments, it is important that the group listens to the comments in the right way, like you would for a review from a security expert or a general networking expert. E-mail practices around sending and responding to comments have to be carefully managed, as the outside reviewers are typically not list members.

For the working group chairs, you can request a review by sending mail to [email-masked], indicating the name of the document. You can ask for a review as soon as a working group last call has ended successfully and the document edits are done. Initially, it would make sense to ask for reviews on documents that you feel are likely the most stable ones. To avoid congesting the reviewer resources, ask for this service only for some documents, not as a wholesale service on every document you submit for publication.

You can ask for the review in parallel with ongoing AD reviews, filling out the shepherd questionnaire, etc. The hope is that reviews can come in in the same time frame as other tasks in this stage, and that you can take the comments into account and revise the document before finally sending it out for IETF Last Call. Reviews will be sent to you and potentially the working group mailing list. You may need to forward comments and/or approve new posters to the list. Discussion on the mailing list is encouraged, but please try to ensure that the reviewer is kept on the Cc line, as he or she may not be on the list itself. Treat the reviews with respect and keep it in mind that outside experts may have opinions that need to be taken into account, even if the working group had not considered those aspects before. Mediate and monitor the discussion actively. Try to keep the reviewer out of e-mail storms and work out solutions separately and then engage the reviewer again.

For the directorate coordinators/secretaries, please monitor the [email-masked]. If there is a request for a review, dispatch the task to a reviewer. Managing overload is your task. Please acknowledge requests and whether you can accommodate them. Directorates that today review documents twice, at IETF Last Call and then before entering the IESG telechat should continue this practice by doing the early review and then the IESG telechat review. Existing practices such as retaining the same reviewer for checking the same version is useful. Your review tools and practices may need adjustment to accommodate reviews happening at different times.

For the reviewers, take into account the context. Post your reviews to the working group chairs, authors, and Cc the working group mailing list if necessary. If your review team uses a template for these e-mails, some changes may be necessary in the template for the early review. Some of those changes have been discussed, e.g., in the Gen-ART list.

For everyone, please collect your experiences so that in six months we can evaluate how you liked the experience. And thank you for your participation in this effort!

Jari

---

From: Dave Crocker <[email-masked]>
Date: Sat, 27 Jul 2013 19:05:51 +0100

On 7/27/2013 2:31 PM, Claudio Allocchio wrote:
> should allow us also more time for accomplishing reviews

Not based on what I just went through, with the sequence shown below.

My review window started at 3 weeks and suddenly went down to 1.

This revised process needs to have predictable minimums for the 
reviewers.  1 week is unlikely to qualify.

To be clear, the problem is a lack of coordination between the IESG 
scheduling and the independent reviews.

d/

-------------------------------------------------

> Subject: Re: [appsdir] drafts for review
> Date: Thu, 4 Jul 2013 22:33:10 +0200 (CEST)
> From: Claudio Allocchio <[email-masked]>
> To: Dave Crocker <[email-masked]>
> CC: Claudio Allocchio <[email-masked]>
>
>
>> what's the due date?
>
>
> still not on IESG agenda... let's say July 24th? Thus we are safe... ?

-------------------------------------------------

> Subject: URGENT Re: [appsdir] drafts for review
> Date: Tue, 9 Jul 2013 14:17:11 +0200 (CEST)
> From: Claudio Allocchio <[email-masked]>
> To: Dave Crocker <[email-masked]>
> CC: Claudio Allocchio <[email-masked]>
>
>
> Dave,
>
> the IESG has put the draft on the next telechat! Which is on July 11th.
>
> Most of the ballots say "OK", thus if you can give it a quicker review and
> send it we can just confirm that.
>
> Sorry, they changed the IESG agenda yesterday...

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Barry Leiba <[email-masked]>
Date: Sat, 27 Jul 2013 17:44:40 -0400

> My review window started at 3 weeks and suddenly went down to 1.

Yes, that one sucked, and we're trying (separately) to fix that.  The
problem with the situation you're pointing out is how the appsdir
selection and due dates coincided with the IESG telechat schedule
(lack of coordination, as you say), and the fact that the appsdir
leaders thought that if a document in last call is not scheduled for a
telechat, then it wouldn't be on a telechat for some while yet (in
fact, the ADs will put a document on the very next telechat, if it
looks like there are no significant last call comments, and that can
happen with little warning).  Working on that.

> This revised process needs to have predictable minimums for the reviewers.
> 1 week is unlikely to qualify.

This is a different process, not a revised one.  In this one, WG
chairs can specifically ask for earlier reviews from directorates,
when they think their documents are ready for that -- rather than
waiting for IETF Last Call.  For documents that the WGCs select, this
can only help the scheduling, and documents following this path should
never have sudden changes of schedule.

Barry

---

From: Dave Crocker <[email-masked]>
Date: Sun, 28 Jul 2013 04:52:34 +0100

On 7/27/2013 10:44 PM, Barry Leiba wrote:
> waiting for IETF Last Call.

With thanks to RoseAnna Danna, oh.  Nevermind.

d/

---

From: Tobias Gondrom <[email-masked]>
Date: Sun, 28 Jul 2013 08:22:12 +0200

> never have sudden changes of schedule.

I agree with Dave.
Maybe a small comment:
how about: review time >= 3 weeks. (and no shortening of review dates...)
(normally a good review takes me about an afternoon depending on
document length. Btw. co-reviews take more time due to coordination
round-trips. To schedule a review into one of the following 20 days is
relatively easy. But usually my schedule is arranged and already packed
for the coming 7-10 days. That means, I can slot things in after that
but to fit a review into an already densely planned schedule of only the
following 7 days is really stretching it....)

Be upfront with WG chairs about the minimum timeline to help manage
expectations and planning. It might also help with triggering reviews
early.

Best regards, Tobias

> Barry
> _______________________________________________
> appsdir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/appsdir

---

From: Claudio Allocchio <[email-masked]>
Date: Sun, 28 Jul 2013 15:44:06 +0200 (CEST)

Hi all,

>> happen with little warning).  Working on that.

exactly. A draft of course will not be on a telechat BEFORE the LC is 
over, but as it is now, it may go on the IESH desk immediately after the 
LC ends. And sometimes the LC does not last very long. This is one of the 
problems we are trying to fix.

>>> 1 week is unlikely to qualify.

Correct. As below, 3 weeks is a fair time, but in order to achieve that, 
review items shall be listed earlier: LC end date has proven not to be a 
working parameter. For the "current" (not early) review process, we can go 
for drafts in "LC requested", but we need to have review request arrive 
from WGs earlier!

> following 7 days is really stretching it....)

This is not only "your" case, it is everybody's one indeed.

> expectations and planning. It might also help with triggering reviews

And I guess we need to remind periodically WG chairs about this early 
review process,... or even invent a way to make it "mandatory" (maybe not 
officially... but let's say they shall answer a question like "did the 
draft had and early review?" when they ask for IETF LC...

---

From: SM <[email-masked]>
Date: Sun, 28 Jul 2013 08:18:25 -0700

>scheduling and the independent reviews.

Yes.

>expectations and planning. It might also help with triggering reviews

It would help if WG chairs triggered reviews early.

At 06:44 28-07-2013, Claudio Allocchio wrote:
>And I guess we need to remind periodically WG chairs about this 
>early review process,... or even invent a way to make it "mandatory" 
>(maybe not officially... but let's say they shall answer a question 
>like "did the draft had and early review?" when they ask for IETF LC...

There is already a question in the write-up about whether, for 
example, a URI review is needed?  The answer is usually no.  APPSDIR 
operates in a semi-formal capacity, i.e. the reviews are not more 
special than other Last Call comments.  If the reviews are mandatory, 
even informally, some WG Chairs might line up at the microphone and 
ask questions.

Best regards,
-sm

---

From: Jari Arkko <[email-masked]>
Date: Mon, 29 Jul 2013 00:13:11 +0200

> expectations and planning. It might also help with triggering reviews

Good points. Thanks.

Jari