Subject: [TOOLS-DEVELOPMENT] Internet-draft repository and archive

From: Robert Sparks <[email-masked]>
Date: Mon, 16 Jun 2014 17:17:47 -0500

Based on last week's discussion, with Henrik and Alexa's help, I've 
pulled together a start of a better description of where we are, and 
verified what the current configuration is by looking carefully through 
the various configuration files.

For developers, the key information is at 
<http://trac.tools.ietf.org/tools/ietfdb/wiki/DraftRepositoryDirectories>

For the community:

The repository is exposed as follow
http://www.ietf.org/id/
ftp://ftp.ietf.org/internet-drafts/
rsync www.ietf.org::internet-drafts

The only archive that is exposed is the unified view described at the 
end of the page above. Specifically,
the different subdirectories described in the above wiki page are not 
exposed anywhere except through the unified view.

The archive is exposed over http at http://www.ietf.org/archive/id. 
Right now, the index is so large, it times out while building it, but 
fetches like http://www.ietf.org/archive/id/all_id2.txt work

The archive is not exposed at all by ftp or by rsync. If it were to be 
exposed, the full download would consume 7.1Gb of disk.

Actions left to take (unless otherwise stated, I will hold the token to 
be sure these get done):

* Henrik is addressing the issue in unknown-ids described on the wiki page
* The datatracker should be made to look into the unified archive for 
its txt links. The copy of rfcdiff running on ietfa should be configured 
to do the same (it may be already, I haven't looked at that yet).
* We should expose the archive over ftp and rsync. Care will have to be 
taken to make sure the symlinks are dereferenced before the files are 
exposed.
* Henrik has a better index and search page for the archive running at 
http://tools.ietf.org/id. That should be adopted for how we are serving 
http://www.ietf.org/id and http://www.ietf.org/archive/id to avoid the 
index building timeout problem.

Does anyone see anything else that needs to be done?

Scott - at one point I think you were getting copies of the split out 
directories in the archive somehow.
I don't think you can now without direct access to the filesystem on 
ietfa. Could you check please?
I think you were also who had reported we had a description page that 
was out of date - please point me to that?

I hope this makes things clearer for folks. Let me know where it's still 
confusing.

RjS

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 16 Jun 2014 18:26:52 -0400

On Jun 16, 2014, at 6:17 PM, Robert Sparks <[email-masked]> wrote:

> Scott - at one point I think you were getting copies of the split out directories in the archive somehow.
> I don't think you can now without direct access to the filesystem on ietfa. Could you check please?
> I think you were also who had reported we had a description page that was out of date - please point me to that?

I will test my grabber shell procedure tonight

the procedure uses ftp to get the high level directory listing from ftp.ietf.org/ietf-mail-archive
then iterates through each of the directories in that directory checking to see if I already have the 
mail file & grab it of not 

a quick look indicates the files are still there on ftp.ietf.org

am I missing something?

Scott

---

From: Robert Sparks <[email-masked]>
Date: Mon, 16 Jun 2014 17:29:36 -0500

On 6/16/14, 5:26 PM, Scott O. Bradner wrote:
>
> On Jun 16, 2014, at 6:17 PM, Robert Sparks <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> the procedure uses ftp to get the high level directory listing from 
> ftp.ietf.org/ <http://ftp.ietf.org/>ietf-mail-archive
> <http://ftp.ietf.org>
That's email, not drafts.

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 16 Jun 2014 18:37:44 -0400

I just checked by local archive 

I have a cron-driven routine that pulls down any IDs that I do not already have

the routine pulled down draft-ietf-appsawg-sieve-duplicate-07.txt which was announced 
as published today so it looks like things are still working

the routine uses ftp.ietf.org/internet-drafts 

Scott

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 16 Jun 2014 18:42:00 -0400

what ID population does ftp.ietf.org/internet-drafts have?

(current, all or what?)

tnx

Scott

---

From: Robert Sparks <[email-masked]>
Date: Mon, 16 Jun 2014 17:43:15 -0500

That is the repository - i.e. the currently active internet drafts.
(Please reskim the first message in this thread now that you've looked 
into things a little).

RjS

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 16 Jun 2014 18:55:00 -0400

now that I have the right context - this looks like a full list of actions

Scott

On Jun 16, 2014, at 6:17 PM, Robert Sparks <[email-masked]> wrote:

> Based on last week's discussion, with Henrik and Alexa's help, I've pulled together a start of a better description of where we are, and verified what the current configuration is by looking carefully through the various configuration files.
> 
> For developers, the key information is at <http://trac.tools.ietf.org/tools/ietfdb/wiki/DraftRepositoryDirectories>
> 
> For the community:
> 
> The only archive that is exposed is the unified view described at the end of the page above. Specifically,
> the different subdirectories described in the above wiki page are not exposed anywhere except through the unified view.
> 
> The archive is exposed over http at http://www.ietf.org/archive/id. Right now, the index is so large, it times out while building it, but fetches like http://www.ietf.org/archive/id/all_id2.txt work
> 
> The archive is not exposed at all by ftp or by rsync. If it were to be exposed, the full download would consume 7.1Gb of disk.
> 
> Actions left to take (unless otherwise stated, I will hold the token to be sure these get done):
> 
> * The datatracker should be made to look into the unified archive for its txt links. The copy of rfcdiff running on ietfa should be configured to do the same (it may be already, I haven't looked at that yet).
> * We should expose the archive over ftp and rsync. Care will have to be taken to make sure the symlinks are dereferenced before the files are exposed.
> * Henrik has a better index and search page for the archive running at http://tools.ietf.org/id. That should be adopted for how we are serving http://www.ietf.org/id and http://www.ietf.org/archive/id to avoid the index building timeout problem.
> 
> I hope this makes things clearer for folks. Let me know where it's still confusing.
> 
> RjS
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Robert Sparks <[email-masked]>
Date: Tue, 17 Jun 2014 14:15:25 -0500

This message is only about the datatracker proper. I will bring rfcdiff 
up in another thread.

The good news is that the datatracker code is already using the 
repository correctly almost everywhere.

Looking through the codebase, I find only two places were we point into 
tools.ietf.org for draft content.

The first I don't _think_ is actively used. It's in draft/util, when run 
manually to extract metadata from draft contents.

The second is for talking about expired drafts.

Please look at
http://datatracker.ietf.org/doc/draft-housley-evidence-extns/

It currently says:
This Internet-Draft is no longer active. Unofficial copies of old 
Internet-Drafts can be found here:
http://tools.ietf.org/id/draft-housley-evidence-extns

How should that be modified? Do we leave "Unofficial copies" and just 
change the URL to
http://www.ietf.org/archive/id/draft-housley-evidence-extns-01.txt
?

Or do we wait until we've ported the index and search functionality from 
tools so that could be just
http://www.ietf.org/archive/id/draft-housley-evidence-extns
?

Do we change the sentence that says "Unofficial copies of" to say 
"Copies of"?

Or do we show the contents of the draft from the archive (since we have it)
along with something to make it visually distinct?
(Possibly with a big [EXPIRED] marking)?

---

From: Tony Hansen <[email-masked]>
Date: Wed, 18 Jun 2014 12:47:28 -0400

On 6/17/14, 3:15 PM, Robert Sparks wrote:
> This message is only about the datatracker proper. I will bring 
> rfcdiff up in another thread.
>
> Looking through the codebase, I find only two places were we point 
> into tools.ietf.org for draft content.
>
> The first I don't _think_ is actively used. It's in draft/util, when 
> run manually to extract metadata from draft contents.
>
> Or do we wait until we've ported the index and search functionality 
> from tools so that could be just

The reference to tools.ietf.org/id/draft-housley-evidence-extns pulls up 
the most recent version of the draft, in this case the -01 version.

So pointing it at 
www.ietf.org/archive/id/draft-housley-evidence-extns-01.txt would be a 
direct replacement for the current functionality.

However, I note that if the link had used /html/ instead of /id/, it 
would have pulled up the html page with links to all of the versions. 
And I think that that is more useful tool that where it is currently 
pointing.

Consequently, I think it might be BETTER to wait for search 
functionality, allowing the datatracker to reference all of the 
documents in the archive with that name.

> Do we change the sentence that says "Unofficial copies of" to say 

Why is it plural? If it stays as a reference to the single document, I 
think it would be best written as:

"A copy of the old Internet-Draft can be found here:"

> (Possibly with a big [EXPIRED] marking)?

I hear that Watermarks R Us sells some nice background images like that. :-)

I think keeping the link is just fine instead of using a watermark. It 
emphasizes that the document is "gone" and you have to jump through 
additional hoops to get at it.

     Tony Hansen

---

From: Robert Sparks <[email-masked]>
Date: Tue, 24 Jun 2014 14:59:51 -0500

Unless someone speaks soon, I will push to have this implemented thus:

* the link will point to 
www.ietf.org/archive/id/draft-housley-evidence-extns-01.txt
* The text above the link will say
     "A copy of the old Internet-Draft can be found here:"

When we get the index/search tools ported from tools, we can update the 
page to take better advantage of it.

RjS

On 6/18/14, 11:47 AM, Tony Hansen wrote:
> The reference to tools.ietf.org/id/draft-housley-evidence-extns pulls 
> up the most recent version of the draft, in this case the -01 version.
>
> So pointing it at