Subject: [iola-conversion-tool] Bug when Adding a document to the tracker

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Feb 2012 16:08:39 -0600

Ole, Henrik -

While poking at the bug Cindy found with the days-in-state for an added 
document I found some
problems with the edit form that really need to be addressed before we 
cut over.

The biggest is that the form didn't ask for the area the document should 
be associated with.
It occasionally happens that an AD from outside the area takes the 
shepherd role for a document, so
you can't infer this from which AD is assigned.

Less important, (but important none the less), the fields are initially 
populated very badly.

In the existing tracker, when I add a document, I get fields initially 
populated like this:

These all make sense as initial values - particularly the "None 
selected" and "None" fields.

In the betatracker, I get this (yes, different doc, b

Note the lack of area on that form, and the initial values - the only 
one that really makes sense is Publication Requested.

RjS

---

From: Robert Sparks <[email-masked]>
Date: Tue, 21 Feb 2012 16:19:11 -0600

The (yes, but a different doc, b
below is a typo - I had screenshots from different docs when I entered 
it, but decided
to go back and do the work to get screenshots from the same doc.
I didn't get back to delete that errant text - please disregard it.

---

From: Ole Laursen <[email-masked]>
Date: Wed, 22 Feb 2012 18:18:55 +0100

2012/2/21 Robert Sparks <[email-masked]>

> you can't infer this from which AD is assigned.

Right. At the moment, we don't store an area at all. Grepping for it, the
only place I managed to find using the attribute apart from the edit form
is the agenda document template where the proxy wrapper will infer the area
from the WG (if there's a WG) or else from the shepherding AD.

The search page looks for the area of the WG in the old schema if you do a
search for area, not the area attribute which is perhaps not surprising if
it's only set for documents in the IESG process.

So do you think this is a problem still?

> Less important, (but important none the less), the fields are initially

That's odd. Or rather, it's odd that how it worked before because AFAICT
from the documentation to Django, it shouldn't have. :) It could possibly
be because of the new Django version. In any case, I think I managed to fix
this now.

Ole

---

From: Robert Sparks <[email-masked]>
Date: Wed, 22 Feb 2012 13:00:48 -0600

That's really surprising, and I want to double-check that we haven't 
accidentally lost something.

Your search was on the production code?

I'm seeing idinternal.area_acronym get used in trunk at
idindex/views.py: def all_id2_entry(id):
  ....
./idindex/views.py-    if id.idinternal:
./idindex/views.py-        try:
./idindex/views.py:            area = id.idinternal.area_acronym

and in the search form
/idrfc/views_search.py:            
q_objs.append(Q(area_acronym=query['area']))

What are those replaced with in the new code?

RjS

On 2/22/12 11:18 AM, Ole Laursen wrote:
> 2012/2/21 Robert Sparks <[email-masked] 
> <mailto:[email-masked]>>
>
>     The biggest is that the form didn't ask for the area the document
>     should be associated with.
> Right. At the moment, we don't store an area at all. Grepping for it, 
> the only place I managed to find using the attribute apart from the 
> edit form is the agenda document template where the proxy wrapper will 
> infer the area from the WG (if there's a WG) or else from the 
> shepherding AD.
>
> The search page looks for the area of the WG in the old schema if you 
> do a search for area, not the area attribute which is perhaps not 
> surprising if it's only set for documents in the IESG process.
>
>     Less important, (but important none the less), the fields are
>     initially populated very badly.
>
>
> That's odd. Or rather, it's odd that how it worked before because 
> AFAICT from the documentation to Django, it shouldn't have. :) It 
> could possibly be because of the new Django version. In any case, I 
> think I managed to fix this now.
>
> Ole

---

From: Ole Laursen <[email-masked]>
Date: Wed, 22 Feb 2012 21:52:37 +0100

> accidentally lost something.

That's a good idea. I was going to say, "this is perhaps a surprise but".

> Your search was on the production code?

Yes.

> ./idindex/views.py:            area = id.idinternal.area_acronym

This code isn't actually in use as far as I can see. Most of the code
in idindex is dead.

> q_objs.append(Q(area_acronym=query['area']))

This is a bit tricky, because this part is actually going to query the
Rfc table (which is, by the way, not used much in the production
code). I was looking at the code for I-Ds which says:

    elif query['by'] == 'area':
        q_objs.append(Q(**{prefix+"group__ietfwg__areagroup__area":query['area']}))

> What are those replaced with in the new code?

For the query, basically: either doc.group is WG and doc.group.parent
is query area, or doc.group is individual submission and doc.ad area
is query area:

 docs = docs.filter(Q(group__type="wg", group__parent=query["area"]) |
                               Q(group__type="individ",
                                 ad__role__name="ad",
                                 ad__role__group=query["area"]))

Ole

---

From: Robert Sparks <[email-masked]>
Date: Wed, 22 Feb 2012 15:03:48 -0600

Inline

On 2/22/12 2:52 PM, Ole Laursen wrote:
> 2012/2/22 Robert Sparks<[email-masked]>:
Ok - I agree (the only path into it is wrapped with code that only runs
when not in production mode).
>          q_objs.append(Q(**{prefix+"group__ietfwg__areagroup__area":query['area']}))
Fair enough.

So, I'm going to send a summary of this to the IESG (copying this list) 
to spread that bit of information around.

> _______________________________________________
> iola-conversion-tool mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/iola-conversion-tool

---

From: Robert Sparks <[email-masked]>
Date: Wed, 22 Feb 2012 15:16:55 -0600

IESG :

Cindy and I stumbled over what we first thought was a bug, but it turned 
out to be a surprising
feature/side-effect of the conversion effort.

The data we've been collecting on the edit-info-form (what we have been 
using to add a document to the
tracker or to change it's IESG state/telechat date, etc.) about which 
area a document belongs to is not being
used by any part of the production datatracker other than that form.

So, it will not appear on the form in the new system (and the data 
collected in the old system is
not being migrated I believe).

That really surprised me, but after digging through the code and 
discussing it with Ole, I'm convinced it's
correct.

If we find a need to start collecting that data again in the future, it 
won't be hard to add.

RjS

> So do you think this is a problem still?
<snip/>

---

From: Russ Housley <[email-masked]>
Date: Wed, 22 Feb 2012 17:52:08 -0500

This should not be true.  A telechat agenda indicates which AD has the token, and it shows an Area.  I believe that when Jari was the token holder for -two-maturity-level, the area was still "(Gen)".

Russ

On Feb 22, 2012, at 4:16 PM, Robert Sparks wrote:

> IESG :
> 
> Cindy and I stumbled over what we first thought was a bug, but it turned out to be a surprising
> The data we've been collecting on the edit-info-form (what we have been using to add a document to the
> tracker or to change it's IESG state/telechat date, etc.) about which area a document belongs to is not being
> So, it will not appear on the form in the new system (and the data collected in the old system is 
> That really surprised me, but after digging through the code and discussing it with Ole, I'm convinced it's
> correct.
> 
> If we find a need to start collecting that data again in the future, it won't be hard to add.
> 
> RjS
> 
> 
>> The biggest is that the form didn't ask for the area the document should be associated with.
>> It occasionally happens that an AD from outside the area takes the shepherd role for a document, so
>> Right. At the moment, we don't store an area at all. Grepping for it, the only place I managed to find using the attribute apart from the edit form is the agenda document template where the proxy wrapper will infer the area from the WG (if there's a WG) or else from the shepherding AD.
>> 
>> The search page looks for the area of the WG in the old schema if you do a search for area, not the area attribute which is perhaps not surprising if it's only set for documents in the IESG process.
>>

---

From: Robert Sparks <[email-masked]>
Date: Wed, 22 Feb 2012 21:20:31 -0600

(Ole - question for you below)

Russ -

The existing code isn't displaying an area based on what was set on the 
edit-info form (and wasn't at the time that two-maturity-level was up 
afaict).

Looking in the production system at the next telechat:
<https://datatracker.ietf.org/iesg/agenda/>

I see things like this:

        3.2.1 New Items

draft-snell-atompub-tombstones 
<https://datatracker.ietf.org/doc/draft-snell-atompub-tombstones/> [txt] 
<http://www.ietf.org/id/draft-snell-atompub-tombstones-14.txt>
The Atom "deleted-entry" Element (Informational)
Token: Peter Saint-Andre (gen area)

I think you might have seen gen for Jari on -two-maturity-level as an 
artifact of whatever is causing gen to show
on that page.

In the trackerbeta, at <http://trackerbeta.ietf.org/iesg/agenda/>, the 
same item looks like this:

        3.2.1 New Items

draft-snell-atompub-tombstones 
<http://trackerbeta.ietf.org/doc/draft-snell-atompub-tombstones/> [txt] 
<http://www.ietf.org/id/draft-snell-atompub-tombstones-14.txt>
The Atom "deleted-entry" Element (Informational)
Token: Peter Saint-Andre (app area)

Ole - can you let us know why that shows as gen in the production system?

On 2/22/12 4:52 PM, Russ Housley wrote:
> This should not be true.  A telechat agenda indicates which AD has the 
> token, and it shows an Area.  I believe that when Jari was the token 
> holder for -two-maturity-level, the area was still "(Gen)".
>
> Russ
>
>
>>> Right. At the moment, we don't store an area at all. Grepping for 
>>> it, the only place I managed to find using the attribute apart from 
>>> the edit form is the agenda document template where the 
>>> proxy wrapper will infer the area from the WG (if there's a WG) or 
>>> else from the shepherding AD.
>>>
>>> The search page looks for the area of the WG in the old schema if 
>>> you do a search for area, not the area attribute which is perhaps 
>>> not surprising if it's only set for documents in the IESG process.
>>>

---

From: Ole Laursen <[email-masked]>
Date: Thu, 23 Feb 2012 15:20:04 +0100

2012/2/23 Robert Sparks <[email-masked]>

> 3.2.1 New Items   draft-snell-atompub-tombstones<https://datatracker.ietf.org/doc/draft-snell-atompub-tombstones/>
> [txt] <http://www.ietf.org/id/draft-snell-atompub-tombstones-14.txt>

Ah, hang on there. This is actually the single place I was referring to
before where the set area_acronym is used. So it says "gen" because "gen"
was set when it was added. You can't see this elsewhere, and it won't work
for searching either (actually, searching for gen area in the production
system does find this document but this is because it's an invididual
submission, and those are classified under gen for some reason).

I think there used to be a special handling of area in the edit form in
the Perl code base and the first port of it, that would put individual
submissions in gen and hide the area if it had already been assigned. So if
you're wondering why it's in gen and not in app, that may be the cause.

Ole

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 23 Feb 2012 16:16:25 +0100

Hi Ole,

> you're wondering why it's in gen and not in app, that may be the cause.

Ok, so if this is used by the IESG or some ADs during the IESG processing,
we should support it.  Do we have a straightforward way of associating a
non-WG document with an area independently of who the sponsoring AD is?

Since areas are groups, too, we should be able to have an association with
the area in the same way that wg documents are associated with a WG, I think.

Best regards,

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Thu, 23 Feb 2012 09:17:30 -0600

Well, we should circle back with Henrik on this then.

On 2/23/12 8:20 AM, Ole Laursen wrote:
> 2012/2/23 Robert Sparks <[email-masked] 
>     <https://datatracker.ietf.org/doc/draft-snell-atompub-tombstones/>
> Ah, hang on there. This is actually the single place I was referring 
> to before where the set area_acronym is used. So it says "gen" because 
> "gen" was set when it was added. You can't see this elsewhere, and it 
> won't work for searching either (actually, searching for gen area in 
> the production system does find this document but this is because it's 
> an invididual submission, and those are classified under gen for some 
> reason).
>
> I think there used to be a special handling of area in the edit form 
> in the Perl code base and the first port of it, that would put 
> individual submissions in gen and hide the area if it had already been 
> assigned. So if you're wondering why it's in gen and not in app, that 
> may be the cause.
>
>
> Ole

---

From: Russ Housley <[email-masked]>
Date: Thu, 23 Feb 2012 10:22:06 -0500

> non-WG document with an area independently of who the sponsoring AD is?

In cases like two-maturity-level, the AD is helping another area.  We need this info.

In addition, as Sean said on the IESG list, the RFC Editor can use this to determine which are ought to handle errata for the document.

Russ

---

From: Sean Turner <[email-masked]>
Date: Thu, 23 Feb 2012 12:46:21 +0700

Maybe we could use it to figure out which area errata should be assigned to?  Especially useful for ad sponsored drafts.

Spt

Sent from my iPhone

---

From: Ole Laursen <[email-masked]>
Date: Thu, 23 Feb 2012 21:20:38 +0100

> the area in the same way that wg documents are associated with a WG, I think.

Yes. It sounds like we need to keep track of the area for a few
individually submitted documents here that conceptually belong to an
area, in the same way as WG drafts belong to a WG? Is that correct?

If so, I suggest we just associate the document with the area instead
of the phony individual group and be done with it. I can add some UI
for that. Does that sound agreeable?

Ole

---

From: Robert Sparks <[email-masked]>
Date: Thu, 23 Feb 2012 14:33:22 -0600

On 2/23/12 2:20 PM, Ole Laursen wrote:
> 2012/2/23 Henrik Levkowetz<[email-masked]>:
I think this is mismodeling what we need a bit. Focusing on the 
individually submitted
nature of the document may not be right. To put stress on it:

How would you handle a document from a RAI working group that both 
Gonzalo and I needed
to recuse on. Some other AD (let say Peter for this example) would be 
the sponsor.
The doc should show from RAI and as a product of that RAI WG.

It seems natural to just have an area associated with a document at any 
time it is identified as IETF stream.

> for that. Does that sound agreeable?
Treating individual submissions as belonging to a psuedo-group has 
always disturbed me,
but I don't know if that first proposal is the right fix.

---

From: Russ Housley <[email-masked]>
Date: Thu, 23 Feb 2012 15:35:38 -0500

> for that. Does that sound agreeable?

I think that every IETF Stream document should have an associated Area.

Russ

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 23 Feb 2012 21:48:21 +0100

Hi Ole,

Yes.

> for that. Does that sound agreeable?

The document should be associated with an Area at the point where it is
accepted for sponsoring by an AD (but the area would not necessarily be
the area of the sponsoring AD) -- before that it doesn't have an
association with an area, if it's not a WG document, I believe.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 23 Feb 2012 21:59:41 +0100

Hi Robert, Ole,

On 2012-02-23 21:33 Robert Sparks said the following:
> individually submitted nature of the document may not be right. To
> put stress on it:
> 
> Gonzalo and I needed to recuse on. Some other AD (let say Peter for
> this example) would be the sponsor. The doc should show from RAI and
> as a product of that RAI WG.

Agreed, but that's in line with what I (am trying to, at least!) propose
above.  See more below.

> It seems natural to just have an area associated with a document at
> any time it is identified as IETF stream.

What I'm proposing is that this association can be either indirectly,
through an association with a WG, which would give the area of the WG;
or directly set to an area (which is independent of the area of any 
sponsoring AD).

> always disturbed me, but I don't know if that first proposal is the
> right fix.

The alternative is to not associate them with a group at all, until they
have been accepted by a WG, or a sponsoring AD.  I think that for the
purpose of code uniformity it's easier to associate them with a group
'Individual Contributions' (or whatever) than to code the equivalent of

  if doc.group == None:
     print "Individual"
  else:
     print doc.group

everywhere we need to show something for such a document.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 23 Feb 2012 22:06:10 +0100

Hi Russ,

> I think that every IETF Stream document should have an associated Area.

I think we're in agreement, and I think that's the result of the proposed
mechanism.

Best regards,

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Thu, 23 Feb 2012 15:53:59 -0600

Inline

> or directly set to an area (which is independent of the area of any

I'm OK with this proposal.
Just to make sure I'm following the thinking -

If I had a Document, I could ask it for its area. If it was an 
IETF-stream document,
it would have something to say, either because it could tell because 
what working
group it came from, or someone would have been required to tell it what 
its area
was when it became IETF stream. If it wasn't IETF stream, area would be 
None.

The only part that bothers me is knowing when to insist that someone 
provide the information,
but that should be reasonably containable.

> everywhere we need to show something for such a document.
Well, there's more to it than that or you would just bury that bit of 
complexity in
something like doc.displayDocumentSource()

But I have to apologize - I still have to be reminded that "Group" in 
the new
schema is modeling something quite a bit more than an IETF working group.

There are things like nomcoms, and external bodies, and a while bunch of 
other things
that are Groups, so having a Group out there that is the container for 
IETF stream
individual submission documents isn't bad.  What I've cringed at is that 
someone might
want to get/set the chairs of that group.

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 23 Feb 2012 23:08:24 +0100

Hi Robert,

Inline:

On 2012-02-23 22:53 Robert Sparks said the following:
> IETF-stream document, it would have something to say, either because 
> it could tell because what working group it came from, or someone 
> would have been required to tell it what its area was when it became 
> IETF stream. If it wasn't IETF stream, area would be None.

Ack.

> provide the information, but that should be reasonably containable.

Agreed.

> But I have to apologize - I still have to be reminded that "Group"
> in the new schema is modeling something quite a bit more than an IETF
> working group.

Ah.  Right.  Ok.

> There are things like nomcoms, and external bodies, and a while bunch
> of other things that are Groups, so having a Group out there that is
> the container for IETF stream individual submission documents isn't
> bad. What I've cringed at is that someone might want to get/set the
> chairs of that group.

<<grin>>  Yes, we wouldn't want that.  It would be possible to enforce
limitations on that by using an extra table which gives permissible
role names for each group type; not really hard to do.  But we might
alternatively expect the Secretariat to not do silly things :-)

Best regards,

	Henrik