Subject: [rtcweb] CNAME as a media stream identifier - what I don't think works

From: Harald Alvestrand <[email-masked]>
Date: Tue, 12 Jun 2012 12:14:08 +0200

OK, from the debate today it's clear that the picture is not clear in 
everyone's minds....

with the API that the WG currently has defined, the following is a legal 
set of calls.
So any spec needs to be able to describe the result - even if it is an 
error event at some point.
(Never mind if it is a *wise* set of calls.)

pc = new PeerConnection();

GetUserMedia(..., function(stream) {
      stream1 = stream;
      stream2 = new MediaStream(stream1.audioTracks, stream1.videoTracks);
}

..... stuff to start connecting ....

pc.addStream(stream1);
pc.addStream(stream2);

offer = pc.createOffer(....);

With MSID, the offer will look like this:

....
m=audio
a=ssrc:1234 msid=xyzzy a0
a=ssrc:1234 msid=xyww a0
...
m=video
a=ssrc:1256 msid=xyzzy v0
a=ssrc:1256 msid=xyww v0

and the expectation on the recipient side is that you will get 
pc.onaddstream called with:

1) stream with id = xyzzy, audio and video
2) stream with id = xyww, audio and video

Cullen, this is for you:
If a mediastream is identified by CNAME, what do you expect to have happen?

                         Harald

---

From: Anant Narayanan <[email-masked]>
Date: Tue, 12 Jun 2012 17:07:16 +0200

>       stream2 = new MediaStream(stream1.audioTracks, stream1.videoTracks);

indicates that the audio and video tracks are *copied* into stream2, and 
are not references. Consider, for instance, that if a resolution change 
was made to a track in stream2, the track from the same source in 
stream1 would remain unchanged.

> a=ssrc:1256 msid=xyzzy v0

to look like:

m=audio
a=ssrc:1234 cname:144a
a=ssrc:4321 cname:1e24
...
m=video
a=ssrc:1256 cname:144a
a=ssrc:6521 cname:1e24

> 2) stream with id = xyww, audio and video

would remain the same, except with the IDs substituted by CNAMEs.

-Anant

---

From: Harald Alvestrand <[email-masked]>
Date: Tue, 12 Jun 2012 17:26:48 +0200

On 06/12/2012 05:07 PM, Anant Narayanan wrote:
> indicates that the audio and video tracks are *copied* into stream2, 
> and are not references. 
I have concluded from the discussions this week that we're not of one 
mind on this, and that various items, including this, make more sense if 
we see them as references.

The actual language for the constructor is:

   "Addtracktostream'saudio track list."

There's nothing here about making a copy of /track/.

Remember that the original specification (a year ago) had chained tracks 
feeding into other tracks; we simplified this model to a model where a 
track always links a source to a sink, not to another track.

I think that our previous day's discussion has concluded that if we want 
to make a change to a track, and not have that change affect another 
track, we have to make a copy, and we have to make it explicit. For the 
(I think normal) case where we don't manipulate requirements on tracks, 
or are happy to have any changes affect all copies of the track, we 
don't need the copy.

> Consider, for instance, that if a resolution change was made to a 
> track in stream2, the track from the same source in stream1 would 
> remain unchanged.
>
> instead.

This markup (which is actually already standardized) says two things:

- the sender has two synchronization contexts, 144a and 1e24
- the recipient has no information about whether or not these are 
synchronized to the same clock

This also means that the packets of each stream are transmitted twice, 
of course.
> And, this:

That's a reasonable interpretation. Is it the one we want to make?

---

From: Stefan Hakansson LK <[email-masked]>
Date: Tue, 12 Jun 2012 21:55:43 +0200

> we see them as references.

They would have to be separate enough to allow individual enable/disable 
in any case.

---

From: Stefan Hakansson LK <[email-masked]>
Date: Tue, 12 Jun 2012 22:37:52 +0200

> stream1 would remain unchanged.

This is a bit unclear to me. What would a resolution change really mean 
in this case? To me the resolution (when a camera is used as source) 
would be the resolution delivered by the camera. Can it deliver two (or 
more) resolutions at the same time (perhaps also with different frame 
rates, aspect rations and so on)?

To me it makes most sense (if the application is going to explicitly ask 
for specific resolution, frame rate, aspec ration) to ask for it at 
getUserMedia time. If the app does not ask for this explicitly, at least 
the suitable resolution would be known by the browser whenever the video 
is played in a video element (from the size of it).

---

From: Stefan Hakansson LK <[email-masked]>
Date: Mon, 18 Jun 2012 09:21:07 +0200

On 06/12/2012 05:26 PM, Harald Alvestrand wrote:
> "Addtracktostream’saudio track list."
>

I prefer using the "msid=xyz a0" syntax for a couple of reasons:
* it does not overload the sync contexts (and lose some sync info in the 
process)
* it opens for optimization by sending a track only once even if 
included in more than one MediaStream

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Tue, 19 Jun 2012 21:20:44 +0000

On Jun 12, 2012, at 16:37 , Stefan Hakansson LK wrote:

> This is a bit unclear to me. What would a resolution change really mean in this case? To me the resolution (when a camera is used as source) would be the resolution delivered by the camera. Can it deliver two (or more) resolutions at the same time (perhaps also with different frame rates, aspect rations and so on)?
> 
> To me it makes most sense (if the application is going to explicitly ask for specific resolution, frame rate, aspec ration) to ask for it at getUserMedia time. If the app does not ask for this explicitly, at least the suitable resolution would be known by the browser whenever the video is played in a video element (from the size of it).
> 

I'm not sure but it looks like implementations are assuming that if getusermedia asks fro the same camera twice, the second time it will be busy. I was executing the second stream to be cloned from the first one. 

So lets say you had a 4k by 4k camera and wanted an HD stream for Alice and SD for bob. You would get the first stream with constrains set to HD, then clone that to second stream with constraints set to SD.

Sorry if two versions of this email arrived

---

From: Randell Jesup <[email-masked]>
Date: Wed, 20 Jun 2012 01:08:16 -0400

> I'm not sure but it looks like implementations are assuming that if getusermedia asks fro the same camera twice, the second time it will be busy. I was executing the second stream to be cloned from…

Right, and that's what's described up higher (new MediaStream()). I 
don't think getUserMedia() is *obligated* to return busy, but I would 
expect it to.

> So lets say you had a 4k by 4k camera and wanted an HD stream for Alice and SD for bob. You would get the first stream with constrains set to HD, then clone that to second stream with constraints s…

Ah, but here we disagree - there is no "clone with constraints", and 
even if there were you're implying a set of processing occur "somewhere" 
to make the cloned MediaStream.

MediaStream Processing could handle this (though I'd want to use a 
pre-defined scaler node, not do it in raw JS for perf and hardware-use 
reasons).

Cloning a 4k MediaStream and feeding them to the same (or different) 
PeerConnections would allow the PeerConnections to scale the video for 
encoding and transport.

-- 
Randell Jesup
[email-masked]

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 21 Jun 2012 08:47:19 -0400

On Jun 20, 2012, at 1:08 , Randell Jesup wrote:

> Right, and that's what's described up higher (new MediaStream()). I don't think getUserMedia() is *obligated* to return busy, but I would expect it to.
> 
>> 
> Ah, but here we disagree - there is no "clone with constraints", and even if there were you're implying a set of processing occur "somewhere" to make the cloned MediaStream.
> 
> MediaStream Processing could handle this (though I'd want to use a pre-defined scaler node, not do it in raw JS for perf and hardware-use reasons).
> 
> Cloning a 4k MediaStream and feeding them to the same (or different) PeerConnections would allow the PeerConnections to scale the video for encoding and transport.
> 
> 

Sure - that sounds like reasonable approach to me.