nxu/audio.md

# UXN Audio Proposal

## Problems

Currently the UXN audio device doesn't work very well for playing
complex music. There are a few reasons for this:

 * Note duration is conflated with envelope shape
 * Envelope resolution (67ms) limits tempos/subdivisions
 * Microtonal music is not possible (according to the spec)
 * Using audio callback requires scheduling pauses/silence
 
## Proposal outline

One way to improve the situation is to disentangle the envelope
specification from the note duration, and more generally make it
easier to specify things that a composer will frequently need to
change (pitch, articulation, duration) without having to change the
underlying voice (waveform/envelope settings).

This proposal does four things:

 1. Add a two-byte `duration` port that configures a note's duration
    in milliseconds. The longest possible note is about 66 seconds.

 2. Double the size of the `adsr` port. This means replacing the
    existing two-byte port with four one-byte ports for `attack`,
    `decay`, `sustain`, and `release`. Since we have 4 extra bits per
    stage, we will reduce the resolution of each stage from 66ms to
    10ms (so 0x01 means 10ms). The longest envelope stage is now about
    2.6s (up from 1s previously).
    
 3. Add a one-byte `mode` port, which declares what kind of note or
    sound is being played. This provides an easy way to specify
    different behaviors such as:
     * articulation (e.g. staccato, legato, etc.)
     * different sample rates (44.1k, 22.05k, 11.025k, 5512)

 4. Move the `volume` port to `0x5` and add a one-byte `detune` port.
    A zero value (`0x00`) indicates a "normal" semitone pitch, and
    non-zero values indicate a fractional amount to add. The
    calculation is that the pitch is raised by `detune/256` cents. For
    example, a value of `0x80` will raise the pitch by a quarter-tone.
    The port is placed just before `pitch` so that microtonal music
    can write a "micro-pitch" using one `DEO2` instruction.

## Microtonal music

Here's how to encode the 17-tone equal temperment scale (17ET) as
`detune/pitch` pairs starting from middle C (`0x3c`, i.e. `#3c`).
Since each step of the scale consists of 70.588 cents, we can get
accurate pitches and detunes by adding 70.588 for each step then
dividing by 100 and using the quotient and remainder:

```
  pitch  1: #003c  (0 semitones +  0.00 cents) -- root note is C (#3c)
  pitch  2: #b53c  (0 semitones + 70.59 cents)
  pitch  3: #693d  (1 semitones + 41.18 cents)
  pitch  4: #1e3e  (2 semitones + 11.76 cents)
  pitch  5: #d33e  (2 semitones + 82.35 cents)
  pitch  6: #883f  (3 semitones + 52.94 cents)
  pitch  7: #3c40  (4 semitones + 23.53 cents)
  pitch  8: #f140  (4 semitones + 94.12 cents)
  pitch  9: #a641  (5 semitones + 64.70 cents)
  pitch 10: #5a42  (6 semitones + 35.29 cents)
  pitch 11: #0f43  (7 semitones +  5.88 cents) -- almost perfect 5th (#43)
  pitch 12: #c443  (7 semitones + 76.47 cents)
  pitch 13: #7844  (8 semitones + 47.06 cents)
  pitch 14: #2d45  (9 semitones + 17.64 cents)
  pitch 15: #e245  (9 semitones + 88.23 cents)
  pitch 16: #9746 (10 semitones + 58.82 cents)
  pitch 17: #4b47 (11 semitones + 29.41 cents)
  pitch 18: #0048 (12 semitones +  0.00 cents) -- octave is C (#48)
```

While it's somewhat cumbersome to calculate these detune values in
advance, it only has be done for one octave and the resulting
microtonal pitches can be compactly stored and used.

## Note duration and tempo

The `duration` and `vector` ports precisely specify the audio device
behavior. The given note should be played for a number of milliseconds
specified by `duration`, at which point the `vector` should be called
to play the next note (or next silence). If the specified ADSR ports
have a shorter duration, the *mode* defines how to extend the pitch
(using the *note type* bits). If the ADSR ports have a longer
duration, then the ADSR will be shortened to fit, starting with S/R
but also truncating D and A if necessary. If duration is zero the
duration will be calculated dynamically from ADSR, as it is now.

Composers can choose a duration for the smallest subdivision needed
(e.g. 125ms per 16th note to achieve 120 bpm) and then compute precise
durations for 8th notes, quarter notes, dotted-8th notes, whole notes,
and so on. Similarly, composers can use the same envelope with
stacatto and legato notes to easily achieve different articulations
for different passages.

## More flexible envelope and waveform settings

The new envelope duration range (10ms to 2.6s) allows more more
complex envelopes to be specified, from slower builds and fades to
very fast attacks and releases. Similarly, allowing waveforms to be
specified at lower sampling rates potentially allows more interesting
percussion instruments to be specified without using too many bytes of
the ROM.

For comparison, the NES uses variable frequency samples to allow basic
voices/sounds without using too much space. For NTSC devices the
supported range is 4182-33144 Hz.

## Appending A: proposed specification:

ADDR  SIZE     NAME      DESCRIPTION
0x30  2 bytes  vector    callback address to use when note finishes playing
0x32  2 bytes  duration  (new) duration to play sound in fractional seconds (1ms resolution)
0x34  1 byte   mode      (new) configures how to interpret addr/adsr/pitch (see below)
0x35  1 byte   volume    (moved) 4-bit volumes for left/right channels (6.7% resolution)
0x36  1 byte   attack    (new) envelope: attack duration (vol 0-100%, 10ms resolution)
0x37  1 byte   decay     (new) envelope: decay duration (vol 100-50%, 10ms resolution)
0x38  1 byte   sustain   (new) envelope: sustain duration (vol 50%, 10ms resolution)
0x39  1 byte   release   (new) envelope: release duration (vol 50-0%, 10ms resolution)
0x3a  2 bytes  length    length of waveform data to read (in bytes)
0x3c  2 bytes  addr      address to read waveform data from
0x3e  1 byte   detune    (new) fraction of semitone to raise (0x80 gives a quarter tone)
0x3f  1 byte   pitch     1-bit loop and 7-bit MIDI note (0x00 gives silence)

MODES

Mode consists of the bits `xxWW xAAA`.

The `A` bits correspond to articulation, which determines how to fill
extra space when the duration exceeds the envelope length, and also
which parts of the envelope to exclude (if any) which is denoted with
an underscore:

```
0x00  (xxxx x000)  regular (ADSR, pads with silence)
0x01  (xxxx x001)  short (AD_R, pads with silence)
0x02  (xxxx x010)  staccato (_D_R, pads with silence)
0x03  (xxxx x011)  staccatissimo (_D__, pads with silence)
0x04  (xxxx x100)  legato (ADSR, extends S)
0x05  (xxxx x101)  begin slur (ADS_, extends S)
0x06  (xxxx x110)  slur (__S_, extends S)
0x07  (xxxx x111)  end slur (__SR, extends S)
```

For shorter durations, the envelope is truncated starting from the
end. For example, with `short` articulation the release will be
truncated first, then the decay, and finally the attack.

The `L` bit corresponds to whether to loop or not:

```
0x00 (xxxx 0xxx) play once (do not loop)
0x80 (xxxx 1xxx) repeat note indefinitely
```

Looping will continue until a new `pitch` is written (at which point
that note's looping behavior will be used).

The `W` bits correspond to waveform type:

```
0x00  (xx00 xxxx)  waveform sampled at 44100 Hz (44.1 kHz)
0x40  (xx01 xxxx)  waveform sampled at 22050 Hz
0x80  (xx10 xxxx)  waveform sampled at 11025 Hz
0xc0  (xx11 xxxx)  waveform sampled at  5512 Hz
```

Upsampling will be performed by repeating sample values as many times
as needed (2x, 4x, or 8x). The underlying sound engine is still
expected to play sounds at 44.1 kHz.

## Appendix B: not currently supported

There are some features which would be nice to add but which are not
strictly necessary and would require more significant changes. They
could potentially be supported in the future using additional bits
from the `mode` port, by new devices, or by a larger change.

 * Vibrato/Tremelo
 * Portamento/Glissando/Glide
 * Effects (reverb, equalization, overdrive, etc.)
 * Frequency generators/software synths
 
If we want to support even slower builds/fades we could use existing
mode bits to change the units used by the envelope. That would allow
us to specify larger resolutions.

## Appendix C: existing specification

ADDR  SIZE     NAME      DESCRIPTION
0x30  2 bytes  vector    callback address to use when note finishes playing
0x32  2 bytes  position  read current position in sample
0x34  1 byte   output    read envelope loudness at this moment (0x000 to 0x888)
0x35                     (unused)
0x36                     (unused)
0x37                     (unused)
0x38  2 bytes  adsr      four 4-bit envelope values (attack/decay/sustain/release)
0x3a  2 bytes  length    length of waveform data to read in bytes
0x3c  2 bytes  addr      address to read waveform data from
0x3e  1 byte   volume    4-bit volumes for left/right channels
0x3f  1 byte   pitch     1-bit loop and 7-bit MIDI note