Subject: [TOOLS-DEVELOPMENT] Observations from trying to improve the document authors captured for RFCs in the datatracker

From: Robert Sparks <[email-masked]>
Date: Tue, 9 May 2017 16:42:44 -0500

I spent more time than I planned seeing if we could improve the 
tracker's notion of who the authors of any given RFC were before we 
release the authorstats project. Here's a short dump of a few things I 
learned in the process before I lose track of them:

We currently model documentauthors using email addresses (the Email 
objects point to Person objects). This doesn't work when we don't have 
an email address for an author. We have begun discussing changing that 
to have it link to Person objects, with possibly an Email reference when 
it's available.

The RFC Editor's database has a few aspects that make it difficult to 
use as a source of authority (these are not requests for change, just 
observations on what's there)

- The columns holding author names and author addresses are each single 
text fields. Each row contains a comma separated string in each field.

- There are many RFCs where the author addresses in that database are 
incomplete (often because the author didn't supply an email address in 
that RFC). Sometimes, but not always, this is captured as a "none".

- There are several records where the order of names and the order of 
addresses don't line up (in cases like the previous bullet, the _number_ 
of names and addresses don't agree).

The RFC Editor's database (and the documents themselves) are very 
inconsistent on who to list as authors when a document comes out of the 
IAB or the IESG.

The datatracker currently does not recalculate authors when an RFC is 
published - what it has (until the migrations I just committed are 
applied to production) as the documentauthor set for any given RFC are 
what it had for the last revision of the Internet-Draft that led to it. 
New code needs to be created that updates authors when and RFC is published.

The datatracker doesn't know many of the email addresses that have been 
used in RFCs (until those migrations run again), or who they should be 
associated with. The migrations add about 850 addresses for People the 
tracker already knew, and addresses and Person records for another 220 
that the tracker didn't know yet.

The datatracker still has quite a number of duplicate Person records. As 
part of preparing this set of migrations we cleaned up 50ish cases of 
that. Some examples (already fixed) to show how sometimes this is easy, 
and sometimes it's hard : (format: [ (person name,person primary 
key),(person name, person primary key)] rfc_number )

[(<Person: Lisa Dusseault>, 108522L), (<Person: Lisa Dusseault>, 
22971L)] 4918
[(<Person: M. Rose>, 110964L), (<Person: Marshall Rose>, 2455L)] 3342
[(<Person: Margaret Cullen>, 119947L), (<Person: Margaret Wasserman>, 
100887L)] 7652
[(<Person: Mark Smith>, 120586L), (<Person: Mark Smith>, 6526L)] 1823

The draft parser has a hard job to do. There are many RFCs where authors 
have been "clever" or don't follow the one-author-per-address 
convention. Good examples of hard problems: RFC2016 and RFC2198. (In 
particular, things like "EMail: 
[email-masked]" defeat the parser.)

Applying what I just did to Internet-Drafts in general won't work as 
well as it did for drafts that have become RFCs. Factors contributing to 
that are:

- Things that become RFCs have some pressure to become more regular, 
hence parsable (even very old RFCs)

- The addresses used by folks in RFCs are relatively easy to map to 
People than the general case (fewer instances of 
inscrutable<random_string>@<easyemailproviderdujour> to deal with).

- Associating an address the tracker hasn't seen before with the right 
person will be particularly hard to automate. The probability of 
creating new Duplicate Person records will be very high.

Be sure to note that what we can do in the future to improve the quality 
of the data we capture as new Internet-Drafts are submitted is an 
entirely different problem/solution space to explore than trying to 
improve what we have for old I-Ds.

RjS