CODE: XT

The challenge:

The Flash Player has evolved through the ages to provide the most needed functionality. Through each version there have always remained a few common goals. What I have found is that:

Flash is small — from the player itself to the swf file format to the assets it is optimized to load, focus has been placed on small file sizes (this of course is not as apparent in many websites that are heavy in multimedia)

Flash supports standards — the player supports many web and multimedia formats standard in the industry, such as jpg, mp3 and xml

Flash is interactive — the players greatest strength is the dynamic behavior through ActionScript to allow user interactivity

MIDI, a music standard format that most computers support today, fits all of these categories (like a glove). In fact there’s an opensource project being developed to allow MIDI through Flash, though it requires an additional download and install to the user apart from the Flash Player itself (seen at osflash.org)

I’d like to take a poll. Do you think MIDI should be included in the Flash Player? Why or why not? I want both votes and opinions as I’ll organize the results and send them off to “Adobe, formerly known as Macromedia”. Please respond with some sort of opinion whether it’s pro or con. I’ll list the pros/cons I can think of below (you don’t have to read the rest of this email if you already have your opinion).

A little more on MIDI:
MIDI is a standard music format (some will argue that it’s the only standard) that represents pitches and instruments to be played as a song. It’s extremely small, being the vector of music, and has to be interpreted by a users soundcard. Almost all computers these days support standard MIDI, though it sounds synthesized (especially on the voice and string instruments). Some soundcards or additional software transform the common MIDI into amazing orchestrations, but most users don’t have this advanced playback.

MIDI pros:
can be generated dynamically and played through a sequencer to allow complete on-the-fly customization of sound.
very small in filesize
supported by almost all soundcards
numerous applications for the creation of MIDI songs (many are free)
it’s a standard that has been around for a long time (so there is a lot of support for it)
a small implementation (wouldn’t increase the Flash Player size by more than 50K)

MIDI cons:
most people will have a more synthesized sound
user experience isn’t guarenteed to be consistant (for those with higher quality soundcards)
as with all advancements, could make it really easy for developers to have annoying sounds playing on their sites

In short, if the Flash Player had a midi sequencer built in it would allow developers to create lightweight interactive music applications, such as this sheet music rendering application or music creation applications. It could also allow users to experience a website that contained sound effects or decent background music at very little bandwidth cost. Macromedia has always been good about listening to the developer community and will surely make efforts to build the features we need, provided we tell them.

In sending this challenge to the Flashcoders mailing list I received a great response. Here I hope to summarize their contributions acurately to their credit. (also see full conversation)

MIDI Sound

Making MIDI sound available in the Flash Player would give “major advantages over WAV and MP3 [with] tiny file size and the potential for allowing Flash to create music dynamically, even interactively.” It allows for flexibility in sound that can be readily available to the developer and “opens some new possibilities for using flash and actionscript in a creative way.” Regarding the industry “making MIDI available through Flash will open another market for Adobe. Right now graphic designers and programmers are using Flash, so having musicians on board would definately make Flash a universal tool.” It appears that there are many developers already creating solutions less desirable to fill a need. If Macromedia were to fill that need I am confident there would be a movement toward dynamic audio applications greater than that of design. “I believe adding MIDI to the Flash Player could have similar catalytic creative effect to when Atari first built a MIDI port into their computers — opening the floodgates to some of the best original sequencer software like Dr Ts. Commodore Amiga did the same thing when they included Video in gamut. Revolutionary in its day and still some great ideas from that machine have not reappeared.”

Once again, the drawback is “lots of bad tunes,” a “lack of quality control — somewhat akin to using system fonts with which there’s no guarantee of consistent text layout, only with MIDI the synthesizers in most sound cards are cheesy to say the least.” It is mentioned that “general MIDI banks embedded in the player” would bloat the size, but others say a MIDI solution would actually be very light weight and add little to the player. There is also a concern about device and driver compatability but it seems each OS abstracts those layers in their own way. A significant project already underway in cross platform MIDI/Flash solutions is Alexis Isaac’s Flash MIDI.

Devices

A broader use of MIDI is its compatability with many external devices. It could come as a simple “api for accessing midi devices on the user system, much like accessing a webcam, where you query for which devices are present, then get a handle to a device and start manipulating it.” “If there were low-level interfaces for it…you could attach devices that use midi…and use the midi commands to control Flash, and vice versa…. I don’t have any idea what kind of Flash content could put such controls to use, but the best way to find out would be to make it possible, then sit back and see what people do.” “In addition to keyboards, drum boxes, synths, FX devices and PCs, there is a valuable world of MIDI control surfaces…. MIDI in the Flash Player opens up a fascinating new world of using Flash to drive an exciting range of external media hardware…. Given the skill of programmers on this list I would not be surprized at what emerges, especially when used WITH…other MIDI-friendly hardware and software.”

There is the possibility of communicating with devices through Socket connections, or having an application-specific communication layer as an additional download/install. But “MIDI bandwidth demands can be very misleading. So routing via XML sockets…is great for some applications and hopeless real-time performance-wise for others.” It is possible, “but it would just be nice to minimise the number of pieces in the puzzle.”

Sound Buffer

Access to MIDI controls would be a great step for the Flash Player, but an api to control a sound buffer could prove the most powerful solution: “a more open, low level approach to sound in the player upon which MIDI and other implementations could be developed.” “A memory area that allows you to create sound waves. It is a sound buffer (which can be…inside a flash sandbox) which is no different than a buffer that holds strings or bitmaps…. One can have lots of sound buffers…that [hold] data in 16bit (or whatever) wave amplitudes.” “With a simple sound.setBuffer(binaryData) you could do a huge amount of interested things…. Sound synthesis, generation.” Using mathematics to create sounds instead of recordings “could be done with a minimal impact on filesize” in the deliverable. With “Sound capture…you could capture audio from mic and allow the user to edit it and process it. You could build annotation tools, voice messaging, musical applications etc…. Also if you can get a handle on the audio stream before it hits the audio device you could have live control over streamed audio, tone controls, reverb, delays, echo cancellation, noise reduction, etc.”

Some are concerned that by allowing access to an audio buffer there could be security vulnerabilities such as buffer overflows. These concerns are valid and can be read in full context, but are also preventable.

_______________________________________________

There has been such a large contribution just from the Flashcoders list that I’ve decided to post this publicly to allow everyone to add their voice. Please contribute to the ideas and names listed here as a petition for more audio capabilities for the Flash player.

This entry was posted on Monday, December 12th, 2005 at 5:41 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.