Named for a Tamil poetic form at about the time of Buddha's life, Ula~3D is the first creation of its kind, aimed at creating a complete animated 3D figure from sound in real time.
The first versions draw simple 'platonic solids' (spheres, cones, and cylinders) from integrated FM music synthesizers in an OpenGL context. Simultaneously, the sound from the built-in audio synthesizer can be heard. Future versions will be able to draw other kinds of figures from a musician's own music sources.
Some parts of Ula will take some time to do. It's very difficult to render hair in 3D software. Also skin detail is difficult, because the lighting and focus has to be set up to make it possible.
Ula will draw hair on a 3d model of a head from sound waves in real time. It won't attempt a photographic representation. That's difficult enough even with software designed specifically to do only that. Instead it will be artistic. And entirely new. There's nothing like it anywhere else in the world. Anywhere.
Ula is the newest member of the Metamusic series, which has been in development since 2001. Clients for the Metamusic series include music professionals providing audio and music compositions to major studios producing movies and prime-time TV, including 30 Rock, David Lynch Productions, E! Entertainment, Kings, and Sony Classic Movies.
The algorithms in a prior Metamusic design, Wittgenstein Redbook, will be available for note and controller transformations. These facilitate adjustment of external control surfaces, such as keyboards, ribbons, and so on, by mapping incoming controller, NRPN, and other MIDI data from one channel to another, from one controller to another, and also adjusting their response curves.
In future releases, the algorithms in other Metamusic ensembles, including Husserl and Godel, will be available for melodic transformations, as well as those in Zeno for making 'grooves' (variations in tempo across a musical bar). The Godel architecture permits combination of multiple arpeggios and chord generators in a flexible, multifaceted way. The Husserl sequencing architecture provides unique methods for melodic manipulation that have not been available anywhere else. These capabilities do not supplant, but instead complement, other software owned by music and 3D AV creators. The modules could, for example, provide interactive musical applications to accompany MP# recordings, with which listeners could also change the melody themselves on handheld and desktop devices.
Sound generators include oscillator matrices and SoundFont-based wavetable synthesizers. The oscillator matrices implement a technique invented by Yamaha called "phase modulation" to provide a wide range of deep, sharp, sonorous, harmonic, and resonant sounds. Internal sound generators may create OpenGL graphics, or external instruments may be embedded via an included VST host. A VST-to-Rewire and VST-to-Osc interface is also prototyped, and may be provided as free, independent downloads by the end of the year.
All sound can generate colored 3D wireframes and various filled 3D mesh constructions in real time, with different camera angles, zoom, and lights. Video will be able to change music, and vice versa. The mesh constructions export via the standard ".obj" format into 3D rendering programs for integration with other animation and volumetric image rendering systems.
As a result, artists may create visualizations to accompany sound recordings directly from the sound itself. Owners of handheld devices could change a playing music composition by pointing a handheld playback device with a camera in different directions; for example, the sound could change depending on color or movement wherever the listener may be.
A suite of sound processors will provide additional audio and visual processing. These will include tempo delay, chorus, and harmonic delay. Harmonic delay is similar to spectral delay, but based on musical intervals rather than frequency bands, thus for example, a particular sound may be shifted up an octave and delayed 120 ticks.
All saved design files will be human readable or in industry standard formats. There will be no encryption and artists won't be locked into using my software instead of someone else's because of proprietary formats. All real-time interaction with the user interface will be saved in a text file. It can be played back to display everything the artist does, at any time. The action list may be stored as a readable text file with each action on separate line, with a MIDI or SMPTE timestamp. It can be edited with a text editor, or piped through another software program. Music sequences and transformations are also stored in readable text files.