Subject: [rtcweb] rtcweb use cases for audio/rtp-midi

From: John Lazzaro <[email-masked]>
Date: Thu, 2 Jun 2011 11:56:46 -0700

Hi everyone,

Just a note to let everyone know I'm
monitoring this list, in case there is
interest in adding audio/rtp-midi to the
list of supported payload formats for rtcweb.

I can see a few reasons why audio/rtp-midi
support may be a good idea:

[1] There are now hundreds of iOS apps that
use rtp-midi via Apple's Network MIDI protocol,
to let iPads and iPhones serve as touch-screen
musical instrument and mixing console controllers.
One could imagine wanting platform-independent
versions of these applications in web browsers.

[2] Network musical performance sessions, as
described in the introduction section to RFC 4695
(soon to be obsoleted by RFC 6295).

[3] Both games and telephony have, over the years,
found it useful to have a simple General MIDI or
DLS synthesis engine around, to create low-bitrate
music in lieu of streaming audio files.  One could
imagine web applications benefitting from this as well,
and over-the-network MIDI control being a requirement.

[4] Platforms such as ChromeOS, which only have
a web browser, would benefit from having standardized
real-time MIDI control as a web standard, even if
network use is not of interest.  In this use case,
one would want to plug a MIDI keyboard into the
ChromeOS device USB port, and use the keyboard
to control a music synthesizer locally running as a
web app -- the sort of infrastructure that would need
to be around to make apps [1], [2], [3] above work best.

--

That all said, its not clear to me that it makes sense
for me to write an I-D describing this as a use case
for the first-generation rtcweb.  There are so many
challenges to reaching consensus and interoperability
for plain-vanilla audio-video teleconferencing, that I don't
think adding "nice-to-haves" like MIDI to the manifest
is prudent. So I was planning to just watch and wait.

But, if anyone out there thinks waiting is not a good
idea, feel free to speak up, and if there's sufficient
interest I'll write up an I-D on the topic.  Thanks,

-- 
John Lazzaro
http://www.cs.berkeley.edu/~lazzaro
john [dot] lazzaro [at] gmail [dot] com

---

From: Harald Alvestrand <[email-masked]>
Date: Sun, 05 Jun 2011 07:55:47 +0200

This (midi in browsers) seems to be closely related to what the W3C 
"audio" group is doing.
Are you in contact with them?

                        Harald

---

From: John Lazzaro <[email-masked]>
Date: Sun, 5 Jun 2011 09:21:07 -0700

On Sat, Jun 4, 2011 at 10:55 PM, Harald Alvestrand <[email-masked]> wrote:
> This (midi in browsers) seems to be closely related to what the W3C "audio"
> group is doing.

I've looked at their web pages a few times since
they began, but don't tracked it closely.  Here's
how it relates to audio/rtp-midi.

Basically, the session description parameters for
audio/rtp-midi lets a session define the renderer
(i.e. the way that MIDI should be turned into audio
output in the browser) in a extensible way.  And so,
one could do a short standards-track extension RFC
that describes  how to say "Here is Javascript that
embodies the rendering algorithm. Run it in the
browser using the W3C audio group tools for
sending audio samples to the client output".  Or
an equivalent for technologies like NaCl, for
situations where Javascript is too slow.

This approach is analogous to dispensing with
the native voice codec for telephony, and instead
send along the voice decoding algorithm in
Javascript to be run in the browser. With the
same advantages and disadvantages, to a
first order.

The alternative is to build MIDI renderers into the
browsers, in the same way we build native voice
codecs into browsers.  The RTP MIDI RFC describes
how to specify three standardized MIDI renderers,
offering increasing levels of programmability and
normative output: General MIDI, DLS2, and MPEG 4
Structured Audio.  Alternatively, the mechanism allows
for extensions to specify other renderers, be they
standardized or proprietary, via standards-track
extension RFCs.

Note that everything I discussed above covers the
[2], [3], and [4] use cases in my original post. The
[1] use case involves the user on the client manipulating
GUI elements (graphical depictions of piano keys, audio
volume faders, etc) that are sent as MIDI commands over
audio/rtp-midi to a device that the user wishes to control --
anything from a hardware music synthesizer, to a lighting
panel, to a PC running emulations of some hardware device.
In that case, there's no audio rendering specified.  Most of
the iOS apps that use RTP MIDI are of this type.