Algorithmic Timeline based Composer

by John Dunn

Global Controls
Track Controls
-- Clock group
-- Pitch group
-- Key group
-- Dynamics group
-- Control group
-- Algorithm group
-- Program select group


ImageMusic loads an image file (.gif, jpg, .png, .bmp, .tif), and processes the image pixels based on selectable algorithms to provide the raw note sources for music based on the picture. It does not convert the image to music. Rather, it provides a rich set of algorithmic processes that can be applied to the image pixel data, and a set of 8 2-channel timeline tracks to lay out your musical impressions of the image.

The general idea is to set up timeline labels that correspond to the values in the number grid. There are 8 timelines and 8 corresponding number grids. The numbers are sliders without the slider part, left click and drag the mouse up or down to set the numbers. Most parameters are in the 0-127 MIDI range. The 8 dual channel tracks use MIDI channels 1-16 (0 - 15), two channels per track.

As the timeline cursor progresses it passes over event labels numbered 1-8 that have been set up in the 8 tracks. Each time the cursor pass over an event label, the corresponding row of parameters change to the values previously set for that number. To set the parameter number for a particular label, just dial in the parameter while the timeline cursor is over it - the updated parameter is automatically saved.

This is a very large patch, over 6,000 modules, so it may not run well on slower computers.

To hear an example of music created with this patch, visit and select "Ten Minutes of Cold."

Global Controls

Starting at the top left, and working down, left to right....

On the Toolbar, only 3 controls are used:

[C] Canvas On/Off (Toolbar, leftmost of the row of screen control buttons [C] [1] [2] [3] [4]) - This shows the image being processed. The image displayed on the Canvas is show-only, the image information being processed is held in a memory buffer, unseen. This allows you to view the pixels as they are processed, and even to erase the image, without affecting the source pixel data.

[1] Screen1 On/Off (Toolbar, second of the row of screen control buttons [C] [1] [2] [3] [4]) - This shows the piano roll note view.

Play/Stop (Top middle of toolbar) -Starts and stops the progression of the timeline cursor. Also acts as a restart that causes updating of other global parameters that have been changed. For this reason it is important that global parameters should be changed when in Stop mode (timeline cursor not moving), so they update properly when Play is next pushed.

Reset (Toolbar Refresh button, just to the left of Play) - Resets the timeline cursor to position 0.

PlayOn (Top left, just above the timelines): When On (highlighted) the MIDI generating algorithms continue to play even when Play/Stop is at Stop, and the timeline cursor has stopped moving. Useful for adjusting parameters without having the timeline cursor run off the label. The data played is only an approximation of what it would be if the cursor were moving on the label; actual data resumes when Play is continued.

Control Timeline (Dark red, above the working timelines) - Click on this otherwise non functioning timeline to position the cursor. Any labels made on this timeline are ignored.

Solo buttons ( Leftmost white on blue buttons, numbered 1-8, 1-8, and X) - Click on any of these buttons to solo (all other lines stop sending MIDI). Click on the lower X to release solo. Also provides number labels for the timelines and their associated parameter boxes. You can click on the timeline solo button or the parameter box solo button, both work the same for each of the 8 tracks.

Mute buttons ( Red on green "M" buttons) - These mute individual tracks by stopping MIDI output for those tracks.. Like the Solo buttons, you can click on either the timeline button or the parameter box button, both work the same.

Skipping over the tracks and parameter boxes for the moment....

Cycle button (Just under the Mute buttons) - Normally the timeline cursor stops when it reaches the end of the timeline, thus ending the composition. With Cycle on (highlighted) the cursor cycles back to the start, running endlessly.

Parameter Box labels (Just right of the Cycle button) - These don't do anything, they are just labels.

Elapsed Time readout (Large black on green, to the right of the parameter box labels) - This shows the elapsed time in minutes:seconds at the timeline cursor, whether it is moving in Play mode or you move it in Stop mode by clicking on the control timeline.

Multi Fav / Single Fav (black on pink button, just right of the Time readout) - Controls whether you store your favorite patch and bank settings for each individual track (Multi Fav) or in a single location that all tracks use (Single Fav). The storage arrays are separate, so you can do both, switching between Multi and Single at any time.

Ticks readout, tempo slider, tempo readout (orange on dark red just right of the Multi/Single Fav button) - This sets the global tick rate, which also translates to overall tempo (individual tracks each have their own Dur settings). Initially set to the default ArtWonk tick rate of 20 milliseconds per tick, which at 24 ticks per quarter note is tempo 125.

Line spread for 2 Line pixel reading algorithms (light blue input box to the right of the tempo slider) - use this to set the pixel reading spread for the 2 Line algorithm.

Steps per tick for Key durations (blue input box to the right of the light blue line spread input box) - Sets the tick multiplier for the two Key parameters, discussed below.

Image file name (light blue input box to the right of the Key steps box) - This is the filename of the source image. This image file must be placed in the Patches/Packages/ImageMusic folder. Any image .gif, .jpg, .bmp, .tif or .png file of any size can be used as source. Smaller images tend to have more intense changes than larger images as a rule, but your mileage may vary. In order for the program to load a new file, you will need to save then reload (or open the Workspace window and click on the Load toolbar button).

Playing MIDI / Playing MP3 (to the right of the Image file name input box) - If you create an .mp3 file of a composition, give it the same name as the image file, and place it in the same Patches/Packages/ImageMusic folder, when set to Playing MP3, the MP3 file will play instead of MIDI output, with the timeline cursor controlling the position in the mp3 file. This is useful if you want to show a composition running but don't have a computer set up with the original MIDI synths used to create the piece.

Cursor button (Yellow on yellow, to the right of Playing MIDI) - When on (highlighted), a cursor shows on the display image, echoing the timeline cursor position.

Clear button (White on green, to the right of Cursor button) - Clears the image by reloading it. Discussed further below.

Time set knobs (blue knobs, lower right) - Sets the total time the timeline will run, up to 60 minutes (large knob) plus 60 seconds (small knob). Changing the timeline time does not change tempo or clock step durations, it only changes the timeline step size and thus how fast the timeline cursor will travel. This makes it possible to change the time of a finished piece without greatly impacting the composition - each section, as defined by the track labels showing will play longer or shorter, and more or fewer pixels will be read and processed, so the composition will change in its details but the overall feel of the piece will remain.

Show Pixels played buttons (white on green column of buttons numbered 1-8 just above the Clear button) - When on (highlighted) pixels being played are written to the image in high white. This will show up as horizontal or vertical lines for the line read algorithms, and as a fractal cloud of dots in the fractal read algorithms. Use the Clear button to restore the image by reloading it, or right click on the image itself and choose "Erase" from the context menu to clear the image to better see the dot read pattern. The dual channel fractal algorithm will change the color to red and green to make it easer to see the separate dot read patterns.

Track Controls

Back to the top....

The 8 timeline tracks - The 8 tracks are dual MIDI channels, left and right. Each track can have up to 128 event labels, and there are 8 different label types, each with a different color and numbered 1-8. The 8 label numbers are basically indexes into 8 different parameter tables for each track. The parameters are what you set with the numbered spread sheet-like boxes below the tracks, also numbered 1-8 and corresponding to the 8 tracks. For example, if you set the Dur parameter to 24 when the timeline cursor is on label 1, and to 12 when the cursor is on label 2, then that track will play quarter notes (Dur=24) while the cursor is on a label 1, and eighth notes (Dur=12) when it is over a label 2. When the cursor is on the background, not over any labels, the previous label parameters are held, but the clock stops and the volume fades down to 0. This transition from background to label is used internally to initialize the next label's parameters, so there should be a little bit of background between labels, they should not butt up against each other. Likewise, it is a good idea to avoid placing a label on the start position, where the cursor goes when the reset button is pressed, in order to start clean with all tracks initially off.

To create an event label, double click on an empty part of the timeline track.

To delete an event label, double click on the label.

To change event label numbers, right click on the label. It will cycle from 1 to 8.

To change size, left click and drag on the right end of the label while holding down either the shift or ctrl key.

To move a label, left click and drag while holding down either the shift or ctrl key. For very small labels, hold down both shift and ctrl keys at once to move the label instead of stretching it.

The track activity LEDs (two red LEDs between the Mute buttons and the timeline tracks - these flicker at the track clock Dur rate when notes are played, but otherwise do nothing.

The volume LEDs (orange LEDs between the track LEDs and the parameter boxes) - these are highlighted when the MIDI channel volume for the track is non zero. When volume is 0 (LED off), the channel is mute.

The 8 rows of parameter boxes - This is where you set up the sounds for the (up to) 8 label values. The parameters are grouped by overall function with each group indicated by different color labels. To change a value, make sure the cursor is over the label number you want to change, and just dial in the value. Saving is automatic.

Clock group: Each timeline track is driven by a pair of MultiClock modules, one for each channel. These are Autosync modules, and they share their control inputs, so the clocks work in tandem as a single clock unless Dly, Nth or Skip inputs are used.

Dur - (Duration) Set the clock cycle duration in ticks (increments of the Main Loop interval). Smaller values run faster. Minimum value is 2 (one tick for the On cycle and one for the Off cycle). Set to 0 to disable playing, as when only the single shot Key notes are wanted. ArtWonk clocks run at the MIDI standard of 24 ticks per quarter note, so 3=thirtysecond, 6=sixteenth, 12=eighth, 24=quarter, 48=half, 96=whole.

%On - Sets the portion of a the clock duration that the clock output will be On. Values of 0% or less gives an On duration of exactly 1 tick; values of 100% or more sets the On duration to the full value of the Duration input except for the last tick at the end, which is Off for 1 tick. Because clock timings are in ticks, the accuracy of the On duration time is limited to the number of ticks in the Duration. When Legato is on (Lgto is non zero), %On is ignored.

Dly - Number of ticks to delay onset of clock beat. Range is 0 to Dur. This parameter is linked only to the right channel clock, so delay values beat against the non delayed left channel. Use Pan to reverse left/right if needed.

LNth, RNth - Left and right tick on every nth beat (4 = every 4th beat, 9 = every 9th beat, etc.). Useful for creating long time intervals, or to create odd beat rhythms by setting left and right to different Nth values.

Skip - Random Skip probability (0 = no skips, 64=50% skips, 127 = nearly all skips, etc.). Skip is randomly determined separately within each MultiClock, so the two clocks will skip beats against each other.

Pitch group:

Lgto - Legato. 0 = Off, anything else = On. This causes the track to ignore the clock note-off, and to simply produce a new note-on every time the note value changes. ArtWonk automatically handles MIDI Note Off messages by sending the previous Note Off immediately prior to sending the new Note On. The effect is that notes have a run together Legato sound, and new notes will not sound unless the note value actually changes, so the notes skip beats where a note is followed by another of the same pitch.

HLS - Hue-Luminance-Saturation (also Red, Green, Blue). Determines how pixels from the source image will be interpreted. 0=Hue, 1=Lum, 2=Sat, 3=Red, 4=Blue, 5=Green.

Max - Maximum range for the note value. Scales the raw HLS/RGB value of the pixel to 0-Max.

LTsp, RTsp - Transpose. Adds offset to the note value. Separate settings for left and right channels. So selecting 12 Max, 60 LTsp, and 67 RTsp will result in a one octave range starting at middle C in the left channel, running parallel 5ths in the right channel.

Scl - Scale. Force notes to fit a scale. There are 44 scales available plus 0, Chromatic, which is the unchanged MIDI note number:

00. Chromatic
01. Major (Ionian)
02. Nat. Minor (Aeolian)
03. Major Pentatonic
04. Whole Tone
05. Fifths
06. Octaves
07. Major Triad
08. Minor Triad
09. Dorian
10. Phrygian
11. Lydian
12. Mixolydian
13. Locrian
14. Bebop 1
15. Bebop 2
16. Major Harmonic
17. Harmonic Minor
18. Melodic Minor
19. Minor Pentatonic
20. Augmented
21. Blues
22. Octatonic 2:1 (Whole Diminished)
23. Octatonic 1:2 (Half Diminished)
24. Arabian
25. Balinese
26. Byzantine (Double Harmonic)
27. Enigmatic
28. Gypsy
29. Hirayoshi
30. Hungarian Gypsy
31. Japanese
32. Jewish
33. Overtone
34. Undertone
35. Pelog
36. Spanish
37. Spanish Octatonic
38. I Messian's Mode of Limited Transposition
39. II Messian's Mode of Limited Transposition
40. III Messian's Mode of Limited Transposition
41. IV Messian's Mode of Limited Transposition
42. V Messian's Mode of Limited Transposition
43. VI Messian's Mode of Limited Transposition
44. VII Messian's Mode of Limited Transposition

Additionally, setting the Scl parameter to 45 or above causes the line to use the 7 Messian scales, automatically selecting the next one up or down each time a new track label value is encountered. The selection is made by a pseudo random array value, so it will be the same selection when the piece is repeated.

The track pitch letter names, as labels to the right of the scale names, and as the large rust colored array of labels along the bottom, are the MIDI pitches in C Major notation as they go out to MIDI, after all permutations, transpositions and scaling have been done; they denote the pitch value seen by the MIDI device.

Key group: These are two single-shot notes, one left and one right. They will use the same MIDI Channel settings (Program, Volume, Modulation, etc.) as the clocked note parameters; and they can be used with, or separately from, the main notes. Useful for drone notes, or to trigger keyswitches in patches that use them.

Key1, Key2 - Keyboard note value, Key1 for left channel, Key2 for right channel. This is straight in, no transpose and no scale adjustment. What you dial in is what goes to MIDI.

LVel, RVel - Velocity. Must be nonzero for note to play.

LDly, RDly - Delay before onset of note, in ticks multiplied by the value in the blue Steps per Tick input box. Thus if Steps=12 and delay=4, the actual delay will be 48, a half note.

LDur, RDur - Duration of note, in ticks multiplied by Steps, as above. Set to 0 to disable playing.

Dynamics group: Volume and Velocity faders. These use the ArtWonk FadeMS module, which provides fading in and out over a constant time, regardless of the value difference in the input change. They are set so 127=10 seconds, 0 is instant.

VoFi - Volume fade in. MIDI volume is a real time control, so the fade is smooth over the time set.

VoFo - Volume fade out.

Vol - Volume. When Vol is 0 the track turns itself off, and the orange volume LED goes out. If VoFo is on, the volume (and track) will remain on until fade out, even after the end of the timeline label.

veFi - Velocity fade in. MIDI velocity is taken at the time the note is sent, and does not change over the duration of the note. However the velocity fade here is useful for changing note dynamics as they play, for the duration of the fade.

veFo - Velocity fade out.

Vel - Velocity. Must be at least 1 for note to sound.

Control group: These are the basic per-channel MIDI controller settings. Left and right channels receive the same settings (except for Pan, which see below).

mFi - Modulation fade in. MIDI modulation is a real time control, so the fade is smooth over the time set.

mFo - Modulation fade out.

Mod - Modulation.

Expr - Expression. This is often used as a 2nd volume, overriding the main Volume. If there is no sound, check that this is set to 127. Uses MIDI control change 11, but this can be changed in the ArtWonk MIDI dialog.

Pan - MIDI Pan. This parameter is wired to be reversed in the right channel, so a setting of 32 gives 32 (mid left) in the left channel and 95 (mid right) in the right channel. Thus, the range should be considered 0-64, and the parameter acts more as a pan spread than straight pan: 0 is hard left and right stereo, 64 is center mono. Values > 64 reverse pan (i.e. a setting of 96 is the same soft pan as 32, but with channels reversed)

Porta - Portamento. Sends both CC#5 (value) & CC#65 (on/off).

Algorithm group: There are five different pitch selection algorithms. These are encapsulated in the Notes Algo macro module, that is top mid right in each 2VoxTl macro, that controls each timeline track, so it should be fairly easy to add your own algorithms. Use the Show Pixels buttons to see the pixels being processed.

Algo - Algorithm 0 - 4. As you select the algorithm its name will display in the white on purple label on the far right of the track. The algorithms are:

0. 2 Line Pix - As the timeline progresses, pixels are processed in a double line across the image (landscape) or down (portrait). The spread for the two lines is set by the light blue Line Spread input box, just to the left of the dark blue Key Steps input box, with a default value of 3 which gives a spread of 6 pixels. These lines are spread evenly down (or across) the image, one set for each of the 8 timeline tracks.

1. Line Pix DL - A single line per track goes to the left channel, and the same values delayed ("DL") to the right channel.

2. Fractal X2 - Two related fractal patterns that more or less mimic eye scan patterns. Uses the Pickover fractal, with initializing patterns set slightly differently for each timeline track, and for the two stereo patterns on each track. Pickover was chosen for its similarity to eye scan patterning, but any of ArtWonk's fractal modules can be substituted to provide additional algorithms.

3. Fractal DL - Pickering fractal as above goes to the left channel, and the same values delayed to the right channel.

4. Random DL - This is a tutorial patch to show you how to create your own algorithms. It generates a pseudo random sequence that steps forward and backward, with delay and kick. It is well commented and uncluttered to make it easier follow. The #4 slot contains this algorithm, and there are 3 empty slots ready to drop in additional algorithms - however there is no real limit on algorithms, you can expand the switching modules that control which algorithm is being being used from the current 8 slots to any number you like. Since unused algorithms do not play, there is hardly any computer processing overhead to adding more algorithms.

Kick - Probability of randomly transposing a note up one or (1 in 4 chance) two octaves. The parameter sets the probability of the octave "kick," with 0 = never, 64 = 50%, and 127 = nearly always.

Dly - Delay amount for the 2nd "chaser" line in the DL algorithms, in steps. When the delay is off (0), both channels are individually processed by Kick and Expnd; when on (any non zero), only the left channel is processed, and the delay is sent to the right channel. This gives two different types of effects. With delay on, it is a simple echo with left and right otherwise the same; with delay off, Kick and Expand treat the two chanels separately, so the left and right values are related but not a simple echo.

Expnd - Probability of expanding the pitch value (0 = never, 64 = 50%, 127 = nearly always). If the pitch value is 1/4th or less of the Max setting, it will be multiplied by 4; if it is 1/2 or less, it will be multiplied by 2; otherwise it is not changed. This is is useful for livening up areas in an image that change slowly, such as sky or calm water.

Program select group: Uses MIDI Bank and Program Change to select the the program for each track. Both channels are set the same. Incremental, spin button, controls are to the left of the program and bank labels (following the parameter boxes for each track), plus a zero set button that instantly sets the program or bank to 0.

Prog - MIDI Program change. The labels are set to use Names folder program and bank names. Default is General Midi but you can easily set it to the patch names names your synth uses. See the readme.txt file in the Names folder for details on setting up custom names. You can change the Names folder with the ArtWonk Options/Patch Options.

Bank - MIDI Bank change. There are two different MIDI bank change protocols, one used by Sound Blaster and the other used by the rest of the world. If your synth is not changing banks, check (or uncheck) the SB Banks checkbox in the ArtWonk MIDI dialog.

Port - Sets the MIDI Port, 0-3. Set up the MIDI Ports in the ArtWonk MIDI dialog. This is useful mostly for those synths/samplers that do not use program or bank change, but do allow multiple ports, such as the Play series from Sounds Online.

Favorites - The yellow and red buttons, spin button, and pink number label just to the right of the parameter boxes are for storing and retriving favorite programs. To save a favorite, click on the yellow button to enable saving (it will stay down), then click on the red button to save the current prog and bank into the slot indicated by the pink number label. To retreive a favorite, pop the yellow (save enable) butto up, then use the spin button to locate the favorite. The small red LED next to the pink number label will highlight if there is anything other than 0,0 stored at the location. There are 100 storage locations, 0-99. Additionally there is a special location, -1, which will erase all previously stored favorites when the red (store) button is pressed while the yellow (enable) button is down. Note that when you set to -1 there will be an overflow error message on the bottom message line. This is normal and can be ignored.