nxu/audio.md

UXN Audio Proposal

Problems

Currently the UXN audio device doesn't work very well for playing complex music. There are a few reasons for this:

• Note duration is conflated with envelope shape
• Envelope resolution (67ms) limits tempos/subdivisions
• Microtonal music is not possible (according to the spec)
• Using audio callback requires scheduling pauses/silence

Proposal outline

One way to improve the situation is to disentangle the envelope specification from the note duration, and more generally make it easier to specify things that a composer will frequently need to change (pitch, articulation, duration) without having to change the underlying voice (waveform/envelope settings).

This proposal does four things:

1. Add a two-byte duration port that configures a note's duration in milliseconds. The longest possible note is about 66 seconds.

2. Double the size of the adsr port. This means replacing the existing two-byte port with four one-byte ports for attack, decay, sustain, and release. Since we have 4 extra bits per stage, we will reduce the resolution of each stage from 66ms to 10ms (so 0x01 means 10ms). The longest envelope stage is now about 2.6s (up from 1s previously).

3. Add a one-byte mode port, which declares what kind of note or sound is being played. This provides an easy way to specify different behaviors such as:

   • staccato, legato, or standard playing styles
   • different sample rates (44.1, 22.05, 11.025)
   • looping or non-looping playback

4. Move the volume port to 0x5 and add a one-byte detune port. A zero value (0x00) indicates a "normal" semitone pitch, and non-zero values indicate a fractional amount to add. The calculation is that the pitch is raised by detune/256 cents. For example, a value of 0x80 will raise the pitch by a quarter-tone. The port is placed just before pitch so that microtonal music can write a "micro-pitch" using one DEO2 instruction.

Microtonal music

Here's how to encode the 17-tone equal temperment scale (17ET) as detune/pitch pairs starting from middle C (0x3c). Since each step of the scale consists of 70.588 cents, we can get accurate pitches and detunes by adding 70.588 for each step then dividing by 100 and using the quotient and remainder:

  pitch  1: #003c  (0 semitones +  0.00 cents)
  pitch  2: #b53c  (0 semitones + 70.59 cents)
  pitch  3: #693d  (1 semitones + 41.18 cents)
  pitch  4: #1e3e  (2 semitones + 11.76 cents)
  pitch  5: #d33e  (2 semitones + 82.35 cents)
  pitch  6: #883f  (3 semitones + 52.94 cents)
  pitch  7: #3c40  (4 semitones + 23.53 cents)
  pitch  8: #f140  (4 semitones + 94.12 cents)
  pitch  9: #a641  (5 semitones + 64.70 cents)
  pitch 10: #5a42  (6 semitones + 35.29 cents)
  pitch 11: #0f43  (7 semitones +  5.88 cents)
  pitch 12: #c443  (7 semitones + 76.47 cents)
  pitch 13: #7844  (8 semitones + 47.06 cents)
  pitch 14: #2d45  (9 semitones + 17.64 cents)
  pitch 15: #e245  (9 semitones + 88.23 cents)
  pitch 16: #9746 (10 semitones + 58.82 cents)
  pitch 17: #4b47 (11 semitones + 29.41 cents)
  pitch 18: #0048 (12 semitones +  0.00 cents)

While it's somewhat cumbersome to calculate these detune values in advance, it only has be done for one octave and the resulting microtonal pitches can be compactly stored and used.

Note duration and tempo

The duration and vector ports precisely specify the audio device behavior. The given note should be played for a number of milliseconds specified by duration, at which point the vector should be called to play the next note (or next silence). If the specified ADSR ports have a shorter duration, the mode defines how to extend the pitch (using the note type bits). If the ADSR ports have a longer duration, then the ADSR will be shortened to fit, starting with S/R but also truncating D and A if necessary.

Composers can choose a duration for the smallest subdivision needed (e.g. 125ms per 16th note to achieve 120 bpm) and then compute precise durations for 8th notes, quarter notes, dotted-8th notes, whole notes, and so on. Similarly, composers can use the same envelope with stacatto and legato notes to easily achieve different articulations for different passages.

More flexible envelope and waveform settings

The new envelope duration range (10ms to 25s) allows more more complex envelopes to be specified, from slow builds and fades to very fast attacks and releases. Similarly, allowing waveforms to be specified at lower sampling rates potentially allows more interesting percussion instruments to be specified without using too many bytes of the ROM.

For comparison, the NES uses variable frequency samples to allow basic voices/sounds without using too much space. For NTSC devices the supported range is 4182-33144 Hz.

Appending A: proposed specification:

ADDR SIZE NAME DESCRIPTION 0x30 2 bytes vector callback address to use when note finishes playing 0x32 2 bytes duration (new) duration to play sound in fractional seconds (1ms resolution) 0x34 1 byte mode (new) configures how to interpret addr/adsr/pitch (see below) 0x35 1 byte volume (moved) 4-bit volumes for left/right channels (6.7% resolution) 0x36 1 byte attack (new) envelope: attack duration (vol 0-100%, 10ms resolution) 0x37 1 byte decay (new) envelope: decay duration (vol 100-50%, 10ms resolution) 0x38 1 byte sustain (new) envelope: sustain duration (vol 50%, 10ms resolution) 0x39 1 byte release (new) envelope: release duration (vol 50-0%, 10ms resolution) 0x3a 2 bytes length length of waveform data to read (in bytes) 0x3c 2 bytes addr address to read waveform data from 0x3e 1 byte detune (new) fraction of semitone to raise (0x80 gives a quarter tone) 0x3f 1 byte pitch 1-bit loop and 7-bit MIDI note (0x00 gives silence)

MODES

Mode consists of the bits Lxxx WWNN.

The N bits correspond to note type:

0x00  (xxxx xx00)  standard note (uses ADSR, extends with silence)
0x01  (xxxx xx01)  staccato note (uses ADR, ignores S, extends with silence)
0x02  (xxxx xx10)  legato note (uses ADSR, extends S as needed)
0x03  (xxxx xx11)  slurred note (uses S, ignores ADR, extends S as needed)

Since we are no longer computing note duration from the ADSR durations, the note type specifies what to do when the duration is different than the envelope. For shorter durations, the sustain and/or release are truncated; for longer durations it varies by type.

The W bits correspond to waveform type:

0x00  (xxxx 00xx)  waveform sampled at 44100 Hz (44.1 kHz)
0x40  (xxxx 01xx)  waveform sampled at 22050 Hz
0x80  (xxxx 10xx)  waveform sampled at 11025 Hz
0xc0  (xxxx 11xx)  waveform sampled at  5512 Hz

Upsampling will be performed by repeating sample values as many times as needed (2x, 4x, or 8x). The underlying sound engine is still expected to play sounds at 44.1 kHz.

The L bit corresponds to whether to loop or not:

0x00 (0xxx xxxx) play once (do not loop)
0x80 (1xxx xxxx) repeat note indefinitely

Looping will continue until a new pitch is written (at which point that note's looping behavior will be used).

Appendix B: not currently supported

There are some features which would be nice to add but which are not strictly necessary and would require more significant changes. They could potentially be supported in the future using additional bits from the mode port, by new devices, or by a larger change.

• Vibrato/Tremelo
• Portamento/Glissando/Glide
• Effects (reverb, equalization, overdrive, etc.)
• Frequency generators/software synths

Appendix C: existing specification

ADDR SIZE NAME DESCRIPTION 0x30 2 bytes vector callback address to use when note finishes playing 0x32 2 bytes position read current position in sample 0x34 1 byte output read envelope loudness at this moment (0x000 to 0x888) 0x35 (unused) 0x36 (unused) 0x37 (unused) 0x38 2 bytes adsr four 4-bit envelope values (attack/decay/sustain/release) 0x3a 2 bytes length length of waveform data to read in bytes 0x3c 2 bytes addr address to read waveform data from 0x3e 1 byte volume 4-bit volumes for left/right channels 0x3f 1 byte pitch 1-bit loop and 7-bit MIDI note