Manipulating Pixels Using Canvas

Celý článek

Modern browsers support playing video via the <video> element. Most browsers also have access to webcams via the MediaDevices.getUserMedia() API. But even with those two things combined, we can’t really access and manipulate those pixels directly.

Fortunately, browsers have a Canvas API that allows us to draw graphics using JavaScript. We can actually draw images to the <canvas> from the video itself, which gives us the ability to manipulate and play with those pixels.

Everything you learn here about how to manipulate pixels will give you a foundation to work with images and videos of any kind or any source, not just canvas.

Adding an image to canvas

Before we start playing with video, let’s look at adding an image to canvas.

<img id="SourceImage" src="image.jpg">
<div class="video-container">
  <canvas id="Canvas" class="video"></canvas>
</div>

We created an image element that represents the image that is going to be drawn on the canvas. Alternatively we could use the Image object in JavaScript.

var canvas;
var context;

function init() {
  var image = document.getElementById('SourceImage');
  canvas = document.getElementById('Canvas');
  context = canvas.getContext('2d');

  drawImage(image);
  // Or
  // var image = new Image();
  // image.onload = function () {
  //    drawImage(image);
  // }
  // image.src = 'image.jpg';
}

function drawImage(image) {
  // Set the canvas the same width and height of the image
  canvas.width = image.width;
  canvas.height = image.height;

  context.drawImage(image, 0, 0);
}

window.addEventListener('load', init);

The code above draws the whole image onto the canvas.

See the Pen Paint image on canvas by Welling Guzman (@wellingguzman) on CodePen.

Now we can start playing with those pixels!

Updating the image data

The image data on the canvas allows us to manipulate and change the pixels.

The data property is an ImageData object with three properties — the width, height and data/ all of which represent those things based on the original image. All these properties are readonly. The one we care about is data, n one-dimensional array represented by an Uint8ClampedArray object, containing the data of each pixel in a RGBA format.

Although the data property is readonly, it doesn’t mean we cannot change its value. It means we cannot assign another array to this property.

// Get the canvas image data
var imageData = context.getImageData(0, 0, canvas.width, canvas.height);

image.data = new Uint8ClampedArray(); // WRONG
image.data[1] = 0; // CORRECT

What values does the Uint8ClampedArray object represent, you may ask. Here is the description from MDN:

The Uint8ClampedArray typed array represents an array of 8-bit unsigned integers clamped to 0-255; if you specified a value that is out of the range of [0,255], 0 or 255 will be set instead; if you specify a non-integer, the nearest integer will be set. The contents are initialized to 0. Once established, you can reference elements in the array using the object's methods, or using standard array index syntax (that is, using bracket notation)

In short, this array stores values ranging from 0 to 255 in each position, making this the perfect solution for the RGBA format, as each part it is represented by 0 to 255 values.

RGBA colors

Colors can be represented by RGBA format, which is a combination of Red, Green and Blue. The A represents the alpha value which is the opacity of the color.

Each position in the array represents a color (pixel) channel value.

• 1st position is the Red value
• 2nd position is the Green value
• 3rd position is the Blue value
• 4th position is the Alpha value
• 5th position is the next pixel Red value
• 6th position is the next pixel Green value
• 7th position is the next pixel Blue value
• 8th position is the next pixel Alpha value
• And so on...

If you have a 2x2 image, then we have a 16 position array (2x2 pixels x 4 value each).

The array will be represented as shown below:

// RED                 GREEN                BLUE                 WHITE
[ 255, 0, 0, 255,      0, 255, 0, 255,      0, 0, 255, 255,      255, 255, 255, 255]

Changing the pixel data

One of the quickest things we can do is set all pixels to white by changing all RGBA values to 255.

// Use a button to trigger the "effect"
var button = document.getElementById('Button');

button.addEventListener('click', onClick);

function changeToWhite(data) {
  for (var i = 0; i < data.length; i++) {
    data[i] = 255;
  }
}

function onClick() {
  var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
  
  changeToWhite(imageData.data);

  // Update the canvas with the new data
  context.putImageData(imageData, 0, 0);
}

The data will be passed as reference, which means any modification we make to it, it will change the value of the argument passed.

Inverting colors

A nice effect that doesn’t require much calculation is inverting the colors of an image.

Inverting a color value can be done using XOR operator (^) or this formula 255 - value (value must be between 0-255).

function invertColors(data) {
  for (var i = 0; i < data.length; i+= 4) {
    data[i] = data[i] ^ 255; // Invert Red
    data[i+1] = data[i+1] ^ 255; // Invert Green
    data[i+2] = data[i+2] ^ 255; // Invert Blue
  }
}

function onClick() {
  var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
  
  invertColors(imageData.data);

  // Update the canvas with the new data
  context.putImageData(imageData, 0, 0);
}

We are incrementing the loop by 4 instead of 1 as we did before, so we can from pixel to pixel that each fill 4 elements in the array.

The alpha value has not effect on inverting colors, so we skip it.

Brightness and contrast

Adjusting the brightness of an image can be done using the next formula: newValue = currentValue + 255 * (brightness / 100).

• brightness must be between -100 and 100
• currentValue is the current light value of either Red, Green or Blue.
• newValue is the result of the current color light plus brightness

Adjusting the contrast of an image can be done with this formula:

factor = (259 * (contrast + 255)) / (255 * (259 - contrast))
color = GetPixelColor(x, y)
newRed   = Truncate(factor * (Red(color)   - 128) + 128)
newGreen = Truncate(factor * (Green(color) - 128) + 128)
newBlue  = Truncate(factor * (Blue(color)  - 128) + 128)

The main calculation is getting the contrast factor that will be applied to each color value. Truncate is a function that make sure the value stay between 0 and 255.

Let’s write these functions into JavaScript:

function applyBrightness(data, brightness) {
  for (var i = 0; i < data.length; i+= 4) {
    data[i] += 255 * (brightness / 100);
    data[i+1] += 255 * (brightness / 100);
    data[i+2] += 255 * (brightness / 100);
  }
}

function truncateColor(value) {
  if (value < 0) {
    value = 0;
  } else if (value > 255) {
    value = 255;
  }

  return value;
}

function applyContrast(data, contrast) {
  var factor = (259.0 * (contrast + 255.0)) / (255.0 * (259.0 - contrast));

  for (var i = 0; i < data.length; i+= 4) {
    data[i] = truncateColor(factor * (data[i] - 128.0) + 128.0);
    data[i+1] = truncateColor(factor * (data[i+1] - 128.0) + 128.0);
    data[i+2] = truncateColor(factor * (data[i+2] - 128.0) + 128.0);
  }
}

In this case you don't need the truncateColor function as Uint8ClampedArray will truncate these values, but for the sake of translating the algorithm we added that in.

One thing to keep in mind is that, if you apply a brightness or contrast, there’s no way back to the previous state as the image data is overwritten. The original image data must be stored separately for reference if we want to reset to the original state. Keeping the image variable accessible to other functions will be helpful as you can use that image instead to redraw the canvas with the original image.

var image = document.getElementById('SourceImage');

function redrawImage() {
  context.drawImage(image, 0, 0);
}

Using videos

To make it work with videos, we are going to take our initial image script and HTML code and make some small changes.

HTML

Change the Image element with a video element by replacing this line:

<img id="SourceImage" src="image.jpg">

...with this:

<video id="SourceVideo" src="video.mp4"></video>

JavaScript

Replace this line:

var image = document.getElementById('SourceImage');

...with this:

var video = document.getElementById('SourceVideo');

To start working with the video, we have to wait until the video can be played.

video.addEventListener('canplay', function () {
    // Set the canvas the same width and height of the video
    canvas.width = video.videoWidth;
    canvas.height = video.videoHeight;    
    
    // Play the video
    video.play();

    // start drawing the frames  
    drawFrame(video);
});

The event canplay is triggered when enough data is available that the media can be played, at least for a couple of frames.

We cannot see any of the video displayed on the canvas because we are only displaying the first frame. We must execute drawFrame every n milliseconds to keep up with the video frames rate.

Inside drawFrame we call drawFrame again every 10ms.

function drawFrame(video) {
  context.drawImage(video, 0, 0);
  
  setTimeout(function () {
    drawFrame(video);
  }, 10);
}

After we execute drawFrame, we create a loop executing drawFrame every 10ms — enough time to keep the video in sync in the canvas.

Adding the effect to the video

We can use the same function we created before for inverting colors:

function invertColors(data) {
  for (var i = 0; i < data.length; i+= 4) {
    data[i] = data[i] ^ 255; // Invert Red
    data[i+1] = data[i+1] ^ 255; // Invert Green
    data[i+2] = data[i+2] ^ 255; // Invert Blue
  }
}

And add it into the drawFrame function:

function drawFrame(video) {
  context.drawImage(video, 0, 0);
  
  var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
  invertColors(imageData.data);
  context.putImageData(imageData, 0, 0);
  
  setTimeout(function () {
    drawFrame(video);
  }, 10);
}

We can add a button and toggle the effects:

function drawFrame(video) {
  context.drawImage(video, 0, 0);
  
  if (applyEffect) {
    var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
    invertColors(imageData.data);
    context.putImageData(imageData, 0, 0);
  }
  
  setTimeout(function () {
    drawFrame(video);
  }, 10);
}

Using camera

We are going to keep the same code we use for video with the only different is that we are going to change the video stream from a file to the camera stream using MediaDevices.getUserMedia

MediaDevices.getUserMedia is the new API deprecating the previous API MediaDevices.getUserMedia(). There's still browser support for the old version and some browser do not support the new version and we have to resort to polyfill to make sure the browser support one of them

First, remove the src attribute from the video element:

<video id="SourceVideo"><code></pre>

<pre rel="JavaScript"><code class="language-javascript">// Set the source of the video to the camera stream
function initCamera(stream) {
    video.src = window.URL.createObjectURL(stream);
}

if (navigator.mediaDevices.getUserMedia) {
  navigator.mediaDevices.getUserMedia({video: true, audio: false})
    .then(initCamera)
    .catch(console.error)
  );
}

Live Demo

Effects

Everything we’ve covered so far is the foundation we need in order to create different effects on a video or image. There are a lot of different effects that we can use by transforming each color independently.

GrayScale

Converting a color to a grayscale can done in different ways using different formulas/techniques, to avoid getting too deep into the subject I will show you five of the formulas based on the GIMP desaturate tool and Luma:

Gray = 0.21R + 0.72G + 0.07B // Luminosity
Gray = (R + G + B) ÷ 3 // Average Brightness
Gray = 0.299R + 0.587G + 0.114B // rec601 standard
Gray = 0.2126R + 0.7152G + 0.0722B // ITU-R BT.709 standard
Gray = 0.2627R + 0.6780G + 0.0593B // ITU-R BT.2100 standard

What we want to find using these formulas is the brightness intensity level of each pixel color. The value will range from 0 (black) to 255 (white). These values will create a grayscale (black and white) effect.

This means that the brightest color will be closest to 255 and the darkest color closest to 0.

Live Demo

Duotones

The difference between duotone effect and grayscale effect are the two colors being used. On grayscale you have a gradient from black to white, while on duotone you can have a gradient from any color to any other color, blue to pink as an example.

Using the intensity value of the grayscale, we can replace this from the gradient values.

We need to create a gradient from ColorA to ColorB.

function createGradient(colorA, colorB) {   
  // Values of the gradient from colorA to colorB
  var gradient = [];
  // the maximum color value is 255
  var maxValue = 255;
  // Convert the hex color values to RGB object
  var from = getRGBColor(colorA);
  var to = getRGBColor(colorB);
  
  // Creates 256 colors from Color A to Color B
  for (var i = 0; i <= maxValue; i++) {
    // IntensityB will go from 0 to 255
    // IntensityA will go from 255 to 0
    // IntensityA will decrease intensity while instensityB will increase
    // What this means is that ColorA will start solid and slowly transform into ColorB
    // If you look at it in other way the transparency of color A will increase and the transparency of color B will decrease
    var intensityB = i;
    var intensityA = maxValue - intensityB;
    
    // The formula below combines the two color based on their intensity
    // (IntensityA * ColorA + IntensityB * ColorB) / maxValue
    gradient[i] = {
      r: (intensityA*from.r + intensityB*to.r) / maxValue,
      g: (intensityA*from.g + intensityB*to.g) / maxValue,
      b: (intensityA*from.b + intensityB*to.b) / maxValue
    };
  }

  return gradient;
}

// Helper function to convert 6digit hex values to a RGB color object
function getRGBColor(hex)
{
  var colorValue;

  if (hex[0] === '#') {
    hex = hex.substr(1);
  }
  
  colorValue = parseInt(hex, 16);
  
  return {
    r: colorValue >> 16,
    g: (colorValue >> 8) & 255,
    b: colorValue & 255
  }
}

In short, we are creating an array of color values from Color A decreasing the intensity while going to Color B and increasing its intensity.

var gradients = [
  {r: 32, g: 144, b: 254},
  {r: 41, g: 125, b: 253},
  {r: 65, g: 112, b: 251},
  {r: 91, g: 96, b: 250},
  {r: 118, g: 81, b: 248},
  {r: 145, g: 65, b: 246},
  {r: 172, g: 49, b: 245},
  {r: 197, g: 34, b: 244},
  {r: 220, g: 21, b: 242},
  {r: 241, g: 22, b: 242},
];

Above there is an example of a gradient of 10 colors values from #0096ff to #ff00f0.

Now that we have the grayscale representation of the image, we can use it to map it to the duotone gradient values.

The duotone gradient has 256 colors while the grayscale has also 256 colors ranging from black (0) to white (255). That means a grayscale color value will map to a gradient element index.

var gradientColors = createGradient('#0096ff', '#ff00f0');
var imageData = context.getImageData(0, 0, canvas.width, canvas.height);
applyGradient(imageData.data);

for (var i = 0; i < data.length; i += 4) {
  // Get the each channel color value
  var redValue = data[i];
  var greenValue = data[i+1];
  var blueValue = data[i+2];

  // Mapping the color values to the gradient index
  // Replacing the grayscale color value with a color for the duotone gradient
  data[i] = gradientColors[redValue].r;
  data[i+1] = gradientColors[greenValue].g;
  data[i+2] = gradientColors[blueValue].b;
  data[i+3] = 255;
}

Live Demo

Conclusion

This topic can go more in depth or explain more effects. The homework for you is to find different algorithms you can apply to these skeleton examples.

Knowing how the pixels are structured on a canvas will allow you to create an unlimited number of effects, such as sepia, color blending, a green screen effect, image flickering/glitching, etc.

You can even create effects on the fly without using an image or a video:

The post Manipulating Pixels Using Canvas appeared first on CSS-Tricks.