Subject: [TOOLS-DEVELOPMENT] Backing up Github WG repositories

From: Robert Sparks <[email-masked]>
Date: Mon, 24 Feb 2020 17:13:13 -0600

All -

Here is a bare bones start at an SoW for building the backup 
infrastructure for github WG repositories as discussed in the git wg 
documents:

<https://github.com/rjsparks/github-backup-sow>

Please comment or suggest changes where things need to change. Don't 
worry about the normal SoW framing for now, but focus on whether this is 
asking for the right thing.

RjS

---

From: Russ Housley <[email-masked]>
Date: Tue, 25 Feb 2020 15:32:51 -0500

This looks fine to me.  I am not sure, but should item number 3 also allow restore to a local git repository?

Russ

> On Feb 24, 2020, at 6:13 PM, Robert Sparks <[email-masked]> wrote:
> 
> All -
> 
> Here is a bare bones start at an SoW for building the backup infrastructure for github WG repositories as discussed in the git wg documents:
> 
> Please comment or suggest changes where things need to change. Don't worry about the normal SoW framing for now, but focus on whether this is asking for the right thing.
> 
> RjS

---

From: "John R. Levine" <[email-masked]>
Date: 25 Feb 2020 15:42:59 -0500

> This looks fine to me.  I am not sure, but should item number 3 also allow restore to a local git repository?

I'd think it'd be a good idea to provide some idea of how many repos, how 
large they're likely to be, and how often they're backed up.  A lot of 
thongs that work fine on 100MB die painfully on multiple gigabytes.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Robert Sparks <[email-masked]>
Date: Wed, 26 Feb 2020 10:43:04 -0600

On 2/25/20 2:42 PM, John R. Levine wrote:
> I'd think it'd be a good idea to provide some idea of how many repos, 
> how large they're likely to be, and how often they're backed up.  A 
> lot of thongs that work fine on 100MB die painfully on multiple 
> gigabytes.

The frequency will start out as hourly, as specified in 
draft-ietf-git-github-wg-configuration.

At the moment, we can only thumbnail the expected initial number of 
repositories. We can put a theoretical upper bound on it using the total 
number of active WG/RG drafts.

Getting the size of what the backup of a repository will be is going to 
be something the project will have to discover. We could get some 
anecdotal points with help from one of the larger WG users perhaps. The 
difficulty will be in exercising the API (or things built on it) that 
gets the entire project (issues, pull-requests, etc), not just the git 
repository itself. Martin - have you tried that with any of the 
repositories you have they keys for?

RjS

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 26 Feb 2020 11:54:03 -0700

A few things to consider.

This should include labels in the things that are backed up.  Saving labels should be relatively trivial.

This should include comments on issues and pull requests.  A record of conversations can be critical to understanding how decisions were made.  This can get a little complicated.  I would be OK if comments on code (which can be on commits or on specific lines in pull requests) are not backed up.

Any restore operation will necessarily be limited.  Any tool that restores a repository will be unable to associate restored pull requests, issues, and comments with the original users who created them.  It should be possible to have the tool generate issues and pull requests with "From @foo at $datetime" annotations, but those will be attributed to the robot account that is used for this process.

In order to backup pull requests you will need to pull git commits from forks.  Attempting to save the code from a pull request in any other form would be far more complicated.  It would be good to specify this in a little more detail.  The approach I have seen work here is just to fetch the commits and apply branch labels to them.  It will not be possible for the original author of a pull request who created it from a fork to make additional changes, but they will be able to make a new pull request if they need to do anything.

A related question there is whether you want to save code only for open pull requests or whether you want to record where the pull request was merged.  Given that we will have git history, I don't think that we need to worry so much about old pull requests, aside from any commentary.

FWIW, I have some limited backup capability for pull requests and issues, which you can see here: https://quicwg.org/base-drafts/issues.html  With the 3k+ issues on that repo, it's very inefficient, but it does operate entirely offline.  It doesn't capture comments either.  I would like to have an alternative to this, because the method I use writes a very badly packed file to the repository, which is hugely inefficient.

---

From: Robert Sparks <[email-masked]>
Date: Wed, 26 Feb 2020 13:00:02 -0600

Hi Martin -

Have you tried using the organization migration api 
(<https://developer.github.com/v3/migrations/orgs/> or something like 
<https://github.com/josegonzalez/python-github-backup> against one or 
more of the repositories you have the owner role for? It would be good 
to know if either of those capture labels or comments.

RjS

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 26 Feb 2020 12:42:46 -0700

> to know if either of those capture labels or comments.

So I have used the v3 "REST" API, which does offer the ability to capture comments.  (I think that is what you mean by the first reference there; the organization migration option isn't really relevant here, by my understanding.)  I just didn't use that capability, mostly because the API already requires a massive amount of data.  There is a graphql API (https://developer.github.com/v4/) that is better in that regard, but I haven't had time to play with it.

The API has information about labels. I have used that successfully.

The python tool appears to capture labels and comments, but I haven't tried it.  I don't know what format it stores data in. That might be critical to determining its suitability for this project.

---

From: Robert Sparks <[email-masked]>
Date: Wed, 26 Feb 2020 15:32:55 -0600

On 2/26/20 1:42 PM, Martin Thomson wrote:
> So I have used the v3 "REST" API, which does offer the ability to capture comments.  (I think that is what you mean by the first reference there; the organization migration option isn't really relevant here, by my understanding.)

I think it is spot-on relevant. That is the current API github 
recommends for backing up organization repositories.

See 
<https://help.github.com/en/github/creating-cloning-and-archiving-repositories/backing-up-a-repository> 
where it says

> To download an archive of your repository, you can use the API for 
> user or organization migrations. For more information, see "Migrations 
> <https://developer.github.com/v3/migrations/>" in the GitHub Developer 
> documentation.

Now, it's clear from the warnings about the API changing without advance 
notice during the preview period that this is not yet stable, but it is 
what they aim people at right now.

Do you read their documentation differently?

>   I just didn't use that capability, mostly because the API already requires a massive amount of data.  There is a graphql API (https://developer.github.com/v4/) that is better in that regard, but I haven't had time to play with it.
>
So, we clearly specify that we have a requirement to capture labels and 
comments, and leave it to the bidders to propose tools that will do 
that, yes? We can make sure the language where we point to potential 
solutions doesn't imply they are fit, or preferred.

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 26 Feb 2020 16:56:35 -0700

On Wed, Feb 26, 2020, at 14:32, Robert Sparks wrote:
> > To download an archive of your repository, you can use the API for user or organization migrations. For more information, see "Migrations <https://developer.github.com/v3/migrations/>" in the GitHub Developer documentation. 
>
> Now, it's clear from the warnings about the API changing without 
> advance notice during the preview period that this is not yet stable, 
> but it is what they aim people at right now. 
> 

https://developer.github.com/v3/migrations/users/#download-a-user-migration-archive is a more targeted link.  I don't know about the format of the dump you get.

> So, we 
> clearly specify that we have a requirement to capture labels and 

Sounds good.  Thanks.

---

From: Robert Sparks <[email-masked]>
Date: Thu, 27 Feb 2020 14:55:31 -0600

> https://developer.github.com/v3/migrations/users/#download-a-user-migration-archive is a more targeted link.  I don't know about the format of the dump you get.

You have to do the steps before it on that page before there's anything 
to download.

Specifically: 
https://developer.github.com/v3/migrations/users/#start-a-user-migration 
will have had to have succeeded (it's where you get :migration_id, and 
https://developer.github.com/v3/migrations/users/#start-a-user-migration 
will have to have been checked to see that the migration has finished 
successfully.

And one more thing to consider: This whole API requires providing the 
credentials of an _owner_ of the repository being backed up. I suspect 
that means we'll have an identity that we ask WG draft repositories to 
add as an owner. Does that match your expectations?

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 28 Feb 2020 02:26:22 -0700

(Travel might delay more responses...)

> add as an owner. Does that match your expectations?

Now that I understand this more, that is just one of the problems: locking the repo every hour is likely to be too disruptive.  I don't particularly like the idea that we have this tool as an owner, but that's less of a problem.

---

From: Robert Sparks <[email-masked]>
Date: Fri, 28 Feb 2020 08:14:14 -0600

On 2/28/20 3:26 AM, Martin Thomson wrote:
> Now that I understand this more, that is just one of the problems: locking the repo every hour is likely to be too disruptive.

locking is an option, not a requirement, and it's turned off by default. 
I didn't see anything that indicated it would break things if you didn't 
lock the repository.

I think that option was there only to make it easier for people who were 
really _migrating_ a repository to a different place to avoid confusion 
or losing information during transition.

>    I don't particularly like the idea that we have this tool as an owner, but that's less of a problem.