Glitch art refers to the use of audio signals and audio effects to modify images. The Cycling74 BEAP modular synthesizer system is particularly suited to glitching. This article provides a prototype for the first glitch modular system in Max7. This is very advantages for glitch art, because numerous audio modules are already available in BEAP for waveform generation, filtering, and effects; the results can be viewed in real time; the modular system is openly extensible to support many artistic variants; and modules can be customized for unique results.

youtube

 

Yes, glitch art is growing up. Zyta Jurielska, for example, made this rendering of William Bouguereau's Pietà by creating layers in Photoshop and processing one layer with text processing.

Pieta by Zyta Jurielska
Pieta by Zyta Jurielska

If you, like Zyta, have ever tried creating image layers in Photoshop, however, you know the process is cumbersome and takes alot of manual work. Most glitch art these days is done by test editing of image files with Audacity. but the audio design environment does not particularly lend itself to rapid creativity and experimentation for images based on audio, and glitch art is usually made by post=processing. So glitch art has remained a fringe, but rapidly growing, field for modern artists.

But now, Max7 is particularly suited to gith art, because its Jitter library is particularly suited to processing of images. Max7 now contains a modular audio library which provides rapid real-time assembly of audio test rigs, but it does not natively support Glitch art.

The First Glitch Module

Yofiel's First Glitch Art
Yofiel's First Glitch Art

This is the first glitch module for BEAP. It reads an image file at its original resolution, then scans the image pixels row by row, like a traditional TV, at audio rate. As it reaches each pixel, it can adjust its color by mixing its RGB value with audio inputs, scaled to fit in the +/-5 CV audio-rate module interface for BEAP modules, then stores it in the same location.

Additionally, the read point and write point can be modulated. The rasterizer signals, HSYNC, VSYNC, current RGB source value, and horizontal scan frequency, and vertical scan frequency are output as CV values.

changing the glitch is easy with BEAP in Max7. A hundred prebuilt audio modules can simply be selected with the BEAP button in the left toolbar,which opens subcategory lists. Upon selection, the module drops into the current patch,ready for wiring up.

Adding modules from the BEAP toolbar panel
Adding modules from the BEAP toolbar panel

Here is an example using some oscillators to mix in a background texture, and feeding back the RGB output into the color inputs through a BEAP echo module, to create image echoes.

Texturing an image and adding image echoes
Texturing an image and adding image echoes

I/O and Controls

 

Function Name
Inputs
get(x), get(y) ~ Modulates the rasterizer H and V signals to change the current scanline image pixel read position. Modulation is clamped into the +/- 5 range.
set(x), set(y) ~ Modulates the rasterizer write position. If unconnected, the pixels are written back to the same location as that from which they were read. Modulation is clamped into the +/- 5 range.
Red, Green, Blue ~ Sets the maximum amount of color to mix with the sampled pixel between -5 (black) and +5 (saturated), at the current pixel location after rasterization is modulated by the SETX and SETY inputs.
Panel Controls
Read ~ Buttons that open a file dialog to read/write an image file. All image formats and options in Quicktime are supported.
H, V, and Autosize ~ The current horizontal image size in pixels. Changing this value after loading an image and turning AUTOSIZE off changes the horizontal scan range. This is an experimental feature. Most users will wish to keep AUTOSIZE on.
Original (RGB) ~ Amount of color kept from the original image, written to the location set after modulation by the module's GETX and GETY inputs.
Tint (RGB) ~ Adds constant RGB values to each pixel.
In (RGB) ~ Amount that the module's RGB inputs are mixed with each pixel, at the location set after the rasterization has been modulated by the SETX and SETY inputs.
Outputs
CVx, CVy ~ The horizontal and vertical image rasterization scan frequencies as a MIDI control voltage value, before modulation by the module's GETX and GETY inputs.
X, Y ~ The horizontal and vertical scan line values, scaled from -5 (left edge)to to +5 (right edge), before modulation by the module's GETX and GETY inputs. As the horizontal scan is continuous, the X output also functions as a non-antialiased ramp oscillator. The Y output is a stepped ramp, changing at horizontal flyback rate, with overall frequency set by the number of pixels in the source image file.
Red, Green, Blue ~ The source image color values, after read position modulation by the module's GETX and GETY inputs, between -5(black) and +5 (saturated).
HSYNC, VSYNC ~ Outputs a single-cycle positive pulse when the internal rasterizer has reached the end of its horizontal or vertical scan line and is on the first pixel of a new line, output before modulation by the module's GETX and GETY inputs. In conventional TV jargon, these are called the horizontal and vertical flyback signals.

 

Performance Design

While this may not sem initially important, image procesing in real time with audio is eventually limited by computational resources, so Yofiel has optimized for CPU from the beginning in this project.

Internally, the module uses gen~ for all audio processing, which reduced CPU usage by about 40%. the demo patch, which contains two oscillators, LFO,and some other miscellaneous modules, can pixel render at 192kHz with default Windows DirectSound drivers (recommended) with ~50% CPU usage and no significant processing peaks on a single Windows 7 thread with a 3GHz i7. The waveset oscillator, which provides many basic modulation waves, is also optimized for low CPU. It is believed this is sufficient to support FFTs and advanced image processing algorithms in the future.

Design Details

This BEAP image module uses the quicktime player for imAge read/write, and jit.peek and jit.poke operations for reading and writing individual pixels. No other Jitter or image functions are currently implemented. The imager peek and poke operations must take place in the same audio cycle, so the design contains two compiled gen~ objects, one to calculate the peek/poke locations, and one to calculate the pixel value to write at the poke location. the second gen~ object is necessary because it needs the jit,peek data, which cannot accessed in gen~, and attempting to feed it back into the first gem~ module causes a feedback hang.

Image Glitcher Top-Level Design
Image Glitcher Top-Level Design

The loader subpatch contains the image read and write operations, which have alot of options, so this design was made as simple as possible.

Glitcher file operations
Glitcher file operations

The horizontal and vertical scan frequencies are calculated once at load, then latched onto the output repeatedly by the image rendering clock, so that new module connections receive the values when they are wired up.

Calculating MIDI pitch for image dimensions
Calculating MIDI pitch for image dimensions

Peek and poke locations would take alot of MSP modules, but in gen~ it only required 12 lines of code and two expressions. The expressions are outside the codebox so they are only evaluated when changed. The codebox contains two rasterization counters and is executed every audio clock cycle.

Glitcher peek and poke calculation
Glitcher peek and poke calculation

The gen~ code is kept simple, as many future variants are envisioned.

Hide LIne Numbers
  1. Param w(320); Param h(240); Param rst(0); 
  2. snc = change(rst) ? 1 : 0;
  3. x, hsnc = counter(1, snc, w);
  4. y, vsnc = counter(hsnc, snc, h);
  5. out1 = clip(x *in7 -5, -5, 5);
  6. out2 = clip(y *in9 -5, -5, 5);
  7. out3 = hsnc;
  8. out4 = vsnc;
  9. out5 = clip(x + floor(in3 * in8 ), 0, w);
  10. out6 = clip(y + floor(in4 * in10), 0, h);
  11. out7 = clip(x + floor(in5 * in8 ), 0, w);
  12. out8 = clip(y + floor(in6 * in10), 0, h);

The color calculations combine 15 variables in only 7 lines of code:

Glitch color calculations
Glitch color calculations

Doing input clipping on the gen~ inputs reduces CPU usage and removes three lines of code. Future versions could use conditional clauses for colorspace conversion, other forms of input mixing, and combine other parameters.

Hide LIne Numbers
  1. Param rin, gin, bin, r, b, g, r0, g0, b0;
  2. out1 = in1 *10 -5;
  3. out2 = in2 *10 -5;
  4. out3 = in3 *10 -5;
  5. out4 = clip(r + r0 *in1 + rin *.1 *(in4 +5), 0, 1);
  6. out5 = clip(g + g0 *in2 + gin *.1 *(in5 +5), 0, 1);
  7. out6 = clip(b + b0 *in3 + bin *.1 *(in6 +5), 0, 1);

The fade operation and RGB output scaling would require 27 MSP~ objects. In the future it is thought to provide image calculations for pixel blocks, rather tnan individual pixels, but theoretically this would be best performed by Jitter slabs rather than gen~.

Other Future Possibilities

Many different glitching possibilities are still unexplored. For example, delay effects could ad image echoes with the existing design.

In the future, additional BEAP modules could automatically isolate particular image areas for processing based on hue proximity, for example. BEAP jitter modules could provide additional types of control and modulation, for example permitting block processing or using DSP such as FFTs for image areas. Additionally, this system can support real-time glitch video generation.

Download and Setup

Extract the files to any location, open, and modify as desired. It opens with a basic image san visible. You should be able to increase the audio samnpling rate for faster image processing.

The BEAP demo contains the Yofiel waveset oscillator, which loads in an audio file at boot. For the demoto work, please make sure the audio file is in the same folder as the patch.

For documentation on the waveset module functionality, please see:

5.5K Waveset Oscillator

and for the waveset oscillator BEAP module itself:

BEAP Waveset Oscillator Module

This file requires Max 7. Please note there are no presets for image glitching currently set up.

Version 2a

  • Preset module: save button works properly
  • Lfo module: sine and random waveform fixes. More robust copy/paste. Minor change to wave display

Version 2

  • Pattrstorage works
  • Some demo presets included
  • Loads default image at start
  • exports processed image correctly
  • Sample rate can change in Max options without restarting patch
  • New Lfo
  • Minor enhancements and fixes
Download: BEAP image glitcher 2a
Title
BEAP image glitcher 2a (Details)
File Type
zip
File Version
2.1
File Size
2,926.15 Kb
Brief Description: