Moving Pixels: Visualizing Music with p5.js
We all have seen visualizations on Windows Media Player or WinAMP at one point in our lives. I always wondered how they are made and how they moved and reacted with sound. A little research led me down a rabbit hole of fascinating ideas behind how pixels move with music.
The key to these visualizations is to extract audio data and utilize some rendering method to visualize it.
In this post, we’ll use p5.js to create music visualizations. Here’s how the final result looks like:
p5.js
p5.js is a JavaScript library designed to make drawing and creative coding on the HTML canvas simple and accessible. It provides a rich set of built-in tools for working with shapes, colors, animation, interaction, and even audio.
We will look at it in action now. To get started, click the Start Coding button on their official website. This opens the p5.js Web Editor where we can write and run code directly in the browser.
If you press Play icon on the top right of the Web Editor you will see a 400x400 canvas with Grey color appear in the Preview section. The preview represents a running p5.js sketch.
Let’s look at what these two methods in the sketch.js do -
setup() - This is called once before the sketch starts running. It does the initial setup like creating the canvas with the specified size.
draw() - This is called continuously once the sketch starts running. The code written in it loops which helps us animate and handle interactions.
These two methods are at a heart of a p5.js sketch. Let’s try to draw a circle on the center of the canvas. We will use the circle() method that p5.js provides -
sketch.js
----------------------------------------
function setup() {
createCanvas(400, 400);
}
function draw() {
background(220);
circle(0,0,100)
}If you press play you will notice that the circle is not at the center of the canvas! You’ll only see a small part of it in the top-left corner.
This is because, in a p5.js sketch, the origin (0, 0) is located at the top-left of the canvas. The x-axis increases to the right, and the y-axis increases downward.
The center of the canvas is actually located at width/2 and height/2. If we place the circle at these coordinates and press Play, we’ll see it appear in the center.
sketch.js
----------------------------------------
function setup() {
createCanvas(400, 400);
}
function draw() {
background(220);
circle(width / 2, height / 2, 100);
}
Let’s try drawing a flower. We will use two ellipse (horizontal and vertical) for the petals and a circle for the center -
sketch.js
----------------------------------------
function setup() {
createCanvas(400, 400);
noStroke()
}
function draw() {
background(220);
let X = width/2
let Y = height/2
let size = 100
fill(243, 58, 106);
ellipse(X, Y, size / 2, size); // vertical petal
ellipse(X, Y, size, size / 2); // horizontal petal
fill(255, 204, 0);
circle(X, Y, size / 2); // center
}
Now that we have a flower, let’s try to animate it. We’ll use use the frameCount variable that p5.js provides to gradually increase the flower’s size over time -
sketch.js
----------------------------------------
function setup() {
createCanvas(400, 400);
noStroke();
}
function draw() {
background(220);
let X = width / 2;
let Y = height / 2;
let flowerSize = frameCount;
fill(243, 58, 106);
ellipse(X, Y, flowerSize / 2, flowerSize);
ellipse(X, Y, flowerSize, flowerSize / 2);
fill(255, 204, 0);
circle(X, Y, flowerSize / 2);
}
There are a lot of methods in p5.js that we can use to animate things on the canvas. Here, I’m using translate(), along with sin() and cos(), to move the flower in a circular path of radius 100 while its size increases over time -
sketch.js
----------------------------------------
function setup() {
createCanvas(400, 400);
noStroke();
}
function draw() {
background(220);
let X = width / 2;
let Y = height / 2;
let flowerSize = frameCount;
translate(100 * cos(frameCount / 10), 100 * sin(frameCount / 10));
fill(243, 58, 106);
ellipse(X, Y, flowerSize / 2, flowerSize);
ellipse(X, Y, flowerSize, flowerSize / 2);
fill(255, 204, 0);
circle(X, Y, flowerSize / 2);
}Audio and FFT (Fast Fourier Transform)
Now that we know how to draw and animate visuals on the screen, let’s see how we can extract audio data and use it.
To extract audio data, we’re going to create a p5.FFT() analyzer object, which will help us performs a FFT (Fast Fourier Transform) on the audio. After applying FFT algorithm to the audio we will get frequency data which we can further use in draw() method.
Now you might be wondering What exactly is an FFT ? or what does the "frequency data" mean ? To answer that, it helps to first understand how audio is represented on a computer and how FFT helps us get frequency data from it -
Audio is captured by computers in form of digital signals. You might have heard that sound is a wave, and waves are continuous, but the audio captured by computers is not continuous - it is a series of numbers that describe the sound over a period of time.
A single audio signal usually contains many frequencies mixed together. FFT is an algorithm that takes this signal and outputs the contribution of each individual frequency it is composed of. The analyze() method of p5.FFT() does exactly this - it returns an array where each element represents the strength (or level) of a particular frequency range.
You can learn more about FFT and Audio from these resources Wikipedia, Nti Audio and Reducible Video
Here’s an example from the p5.FFT() analyzer documentation that visualizes the frequency data returned by analyze() and the waveform data returned by waveform() -
Levels Visualizer
With both the canvas to draw on and the audio data to drive our visuals, we’re ready to bring everything together.
We’re going to create a visualizer I like to call Levels - it does exactly what the name suggests: it shows the audio levels. The idea is to take the elements of the array returned by the analyze() method and place them in an evenly spaced grid, where each cell will represents the level of a frequency bin (range).
The first thing we will do is to get microphone input permission in our p5.js sketch. To do this, we will create an audio input object using the p5.AudioIn() class and call its start() method to begin processing audio. This prompts the browser to ask for microphone access.
Once we have the microphone input, we create a p5.FFT() analyzer and call its setInput() method, passing in the audio input so the FFT can analyze the microphone audio.
sketch.js
----------------------------------------
let audioIn;
let fft;
function setup() {
createCanvas(400, 400);
noStroke();
audioIn = new p5.AudioIn();
audioIn.start();
fft = new p5.FFT();
fft.setInput(audioIn);
}
function draw() {
background(220);
}
Now that we have an audio input, lets draw the grid. The analyze() method return an array with 1024 elements by default. We will divide the canvas into 32x32(=1024) Grid where each cell represents a frequency range corresponding with the element of the returned array. We can do this with a simple nested for loop, and for each grid cell we draw a rectangle using the rect() method.
sketch.js
----------------------------------------
let audioIn;
let fft;
function setup() {
createCanvas(512, 512);
noStroke();
audioIn = new p5.AudioIn();
audioIn.start();
fft = new p5.FFT();
fft.setInput(audioIn);
stroke(0)
}
function draw() {
background(220);
let cols = 32;
let rows = 32;
let w = width / cols;
let h = height / rows;
for (let i = 0; i < rows; i++) {
for (let j = 0; j < cols; j++) {
rect(i * w, j * h , w , h);
}
}
}
We are going to use the level of each frequency bin (or range) as the height of cell in this grid. This way, as the audio changes over time, the height of the cell corresponding to a particular frequency range will also change.
To get the frequency data, we call the analyze() method on p5.FFT() analyzer object in the draw() function. The value of each element in the array returned is between (0-255). We will use the map() method to re-map this value between (0-h), where h is the height of a cell of grid.
If we press play, initially we will just see straight black lines filling the grid, but as soon as you speak into the mic, you’ll see the rectangles change height based on the audio!
sketch.js
----------------------------------------
let audioIn;
let fft;
function setup() {
createCanvas(512, 512);
noStroke();
audioIn = new p5.AudioIn();
audioIn.start();
fft = new p5.FFT();
fft.setInput(audioIn);
stroke(0);
}
function draw() {
background(220);
let spectrum = fft.analyze();
let cols = 32;
let rows = 32;
let w = width / cols;
let h = height / rows;
for (let i = 0; i < rows; i++) {
for (let j = 0; j < cols; j++) {
let x = map(spectrum[i + j * cols], 0, 255, 0, h);
rect(i * w, j * h, w, x);
}
}
}In the demo, you will notice that the rectangles grow downwards. This is because the rect() method draws the rectangle from the specified top-left corner co-ordinates - width increases to right and height increase downwards.
Let’s see how individual cells react when we play audio of specific frequencies. To demonstrate this, we will use this online tone generator website that can play pure tones at different frequencies.
We cannot directly get the system audio as an input source - to get system audio you need to setup loopback - for MacOS you can use BlackHole.
Wrapping up
In the end, I updated the colors and added some spacing between the rectangle. Here’s the Levels Visualizer -
I hope you enjoyed reading through this post! You can check out the full source code here.
Here are two more visualizer I made -