Creating a Distorted Mask Effect on an Image with Babylon.js and GLSL

Celý článek

Nowadays, it’s really hard to navigate the web and not run into some wonderful website that has some stunning effects that seem like black magic.

Well, many times that “black magic” is in fact WebGL, sometimes mixed with a bit of GLSL. You can find some really nice examples in this Awwwards roundup, but there are many more out there.

Recently, I stumbled upon the Waka Waka website, one of the latest works of Ben Mingo and Aristide Benoist, and the first thing I noticed was the hover effect on the images.

It was obvious that it’s WebGL, but my question was: “How did Aristide do that?”

Since I love to deconstruct WebGL stuff, I tried to replicate it, and in the end I’ve made it.

In this tutorial I’ll explain how to create an effect really similar to the one in the Waka Waka website using Microsoft’s BabylonJS library and some GLSL.

This is what we’ll do.

The setup

The first thing we have to do is create our scene; it will be very basic and will contain only a plane to which we’ll apply a custom ShaderMaterial.

I won’t cover how to setup a scene in BabylonJS, for that you can check its comprehensive documentation.

Here’s the code that you can copy and paste:

import { Engine } from "@babylonjs/core/Engines/engine";
import { Scene } from "@babylonjs/core/scene";
import { Vector3 } from "@babylonjs/core/Maths/math";
import { ArcRotateCamera } from "@babylonjs/core/Cameras/arcRotateCamera";
import { ShaderMaterial } from "@babylonjs/core/Materials/shaderMaterial";
import { Effect } from "@babylonjs/core/Materials/effect";
import { PlaneBuilder } from "@babylonjs/core/Meshes/Builders/planeBuilder";

class App {
  constructor() {
    this.canvas = null;
    this.engine = null;
    this.scene = null;
  }

  init() {
    this.setup();
    this.addListeners();
  }

  setup() {
    this.canvas = document.querySelector("#app");
    this.engine = new Engine(this.canvas, true, null, true);
    this.scene = new Scene(this.engine);

    // Adding the vertex and fragment shaders to the Babylon's ShaderStore
    Effect.ShadersStore["customVertexShader"] = require("./shader/vertex.glsl");
    Effect.ShadersStore[
      "customFragmentShader"
    ] = require("./shader/fragment.glsl");

    // Creating the shader material using the `custom` shaders we added to the ShaderStore
    const planeMaterial = new ShaderMaterial("PlaneMaterial", this.scene, {
      vertex: "custom",
      fragment: "custom",
      attributes: ["position", "normal", "uv"],
      uniforms: ["worldViewProjection"]
    });
    planeMaterial.backFaceCulling = false;

    // Creating a basic plane and adding the shader material to it
    const plane = new PlaneBuilder.CreatePlane(
      "Plane",
      { width: 1, height: 9 / 16 },
      this.scene
    );
    plane.scaling = new Vector3(7, 7, 1);
    plane.material = planeMaterial;

    // Camera
    const camera = new ArcRotateCamera(
      "Camera",
      -Math.PI / 2,
      Math.PI / 2,
      10,
      Vector3.Zero(),
      this.scene
    );

    this.engine.runRenderLoop(() => this.scene.render());
  }

  addListeners() {
    window.addEventListener("resize", () => this.engine.resize());
  }
}

const app = new App();
app.init();

As you can see, it’s not that different from other WebGL libraries like Three.js: it sets up a scene, a camera, and it starts the render loop (otherwise you wouldn’t see anything).

The material of the plane is a ShaderMaterial for which we’ll have to create its respective shader files.

// /src/shader/vertex.glsl

precision highp float;

// Attributes
attribute vec3 position;
attribute vec3 normal;
attribute vec2 uv;

// Uniforms
uniform mat4 worldViewProjection;

// Varyings
varying vec2 vUV;

void main(void) {
    gl_Position = worldViewProjection * vec4(position, 1.0);
    vUV = uv;
}

// /src/shader/fragment.glsl

precision highp float;

// Varyings
varying vec2 vUV;

void main() {
  vec3 color = vec3(vUV.x, vUV.y, 0.0);
  gl_FragColor = vec4(color, 1.0);
}

You can forget about the vertex shader since for the purpose of this tutorial we’ll work only on the fragment shader.

Here you can see it live:

Good, we’ve already written 80% of the JavaScript code we need for the purpose of this tutorial.

The logic

GLSL is cool, it allows you to create stunning effects that would be impossible to do with HTML, CSS and JS alone. It’s a completely different world, and if you’ve always done “web” stuff you’ll get confused at the beginning, because when working with GLSL you have to think in a completely different way to achieve any effect.

The logic behind the effect we want to achieve is pretty simple: we have two overlapping images, and the image that overlaps the other one has a mask applied to it.

Simple, but it doesn’t work like SVG masks for instance.

Adjusting the fragment shader

Before going any further we need to tweak the fragment shader a little bit.

As for now, it looks like this:

// /src/shader/fragment.glsl

precision highp float;

// Varyings
varying vec2 vUV;

void main() {
  vec3 color = vec3(vUV.x, vUV.y, 0.0);
  gl_FragColor = vec4(color, 1.0);
}

Here, we’re telling the shader to assign each pixel a color whose channels are determined by the value of the x coordinate for the Red channel and the y coordinate for the Green channel.

But we need to have the origin at the center of the plane, not the bottom-left corner. In order to do so we have to refactor the declaration of uv this way:

// /src/shader/fragment.glsl

precision highp float;

// Varyings
varying vec2 vUV;

void main() {
  vec2 uv = vUV - 0.5;
  vec3 color = vec3(uv.x, uv.y, 0.0);
  gl_FragColor = vec4(color, 1.0);
}

This simple change will result into the following:

This is becase we moved the origin from the bottom left corner to the center of the plane, so uv‘s values go from -0.5 to 0.5. Since you cannot assign negative values to RGB channels, the Red and Green channels fallback to 0.0 on the whole bottom left area.

Creating the mask

First, let’s change the color of the plane to complete black:

// /src/shader/fragment.glsl

precision highp float;

// Varyings
varying vec2 vUV;

void main() {
  vec2 uv = vUV - 0.5;
  vec3 color = vec3(0.0);
  gl_FragColor = vec4(color, 1.0);
}

Now let’s add a rectangle that we will use as the mask for the foreground image.

Add this code outside the main() function:

vec3 Rectangle(in vec2 size, in vec2 st, in vec2 p, in vec3 c) {
  float top = step(1. - (p.y + size.y), 1. - st.y);
  float right = step(1. - (p.x + size.x), 1. - st.x);
  float bottom = step(p.y, st.y);
  float left = step(p.x, st.x);
  return top * right * bottom * left * c;
}

(How to create shapes is beyond of the scope of this tutorial. For that, I suggest you to read this chapter of “The Book of Shaders”)

The Rectangle() function does exactly what its name says: it creates a rectangle based on the parameters we pass to it.

Then, we redeclare the color using that Rectangle() function:

vec2 maskSize = vec2(0.3, 0.3);

// Note that we're subtracting HALF of the width and height to position the rectangle at the center of the scene
vec2 maskPosition = vec2(-0.15, -0.15);
vec3 maskColor =  vec3(1.0);

color = Rectangle(maskSize, uv, maskPosition, maskColor);

Awesome! We now have our black plane with a beautiful white rectangle at the center.

But, wait! That’s not supposed to be a rectangle; we set its size to be 0.3 on both the width and the height!

That’s because of the ratio of our plane, but it can be easily fixed in two simple steps.

First, add this snippet to the JS file:

this.scene.registerBeforeRender(() => {
  plane.material.setFloat("uPlaneRatio", plane.scaling.x / plane.scaling.y);
});

And then, edit the shader by adding this line at the top of the file:

uniform float uPlaneRatio;

…and this line too, right below the line that sets the uv variable

uv.x *= uPlaneRatio;

Short explanation

In the JS file, we’re sending a uPlaneRatio uniform (one of the GLSL data type) to the fragment shader, whose value is the ratio between the plane width and height.

We made the fragment shader wait for that uniform by declaring it at the top of the file, then the shader uses it to adjust the uv.x value.

Here you can see the final result: a black plane with a white square at the center; nothing too fancy (yet), but it works!

Adding the foreground image

Displaying an image in GLSL is pretty simple. First, edit the JS code and add the following lines:

// Import the `Texture` module from BabylonJS at the top of the file
import { Texture } from '@babylonjs/core/Materials/Textures/texture'

// Add this After initializing both the plane mesh and its material
const frontTexture = new Texture('src/images/lantern.jpg')
plane.material.setTexture("u_frontTexture", frontTexture)

This way, we’re passing the foreground image to the fragment shader as a Texture element.

Now, add the following lines to the fragment shader:

// Put this at the beginninng of the file, outside of the `main()` function
uniform sampler2D u_frontTexture;

// Put this at the bottom of the `main()` function, right above `gl_FragColor = ...`
vec3 frontImage = texture2D(u_frontTexture, uv * 0.5 + 0.5).rgb;

A bit of explaining:

We told BabylonJS to pass the texture to the shader as a sampler2D with the setTexture() method, and then, we made the shader know that we will pass that sampler2D whose name is u_frontTexture.

Finally, we created a new variable of type vec3 named frontImage that contains the RGB values of our texture.

By default, a texture2D is a vec4 variable (it contains the r, g, b and a values), but we don’t need the alpha channel so we declare frontImage as a vec3 variable and explicitly get only the .rgb channels.

Please also note that we’ve modified the UVs of the texture by first multiplying it by 0.5 and then adding 0.5 to it. This is because at the beginning of the main() function I’ve remapped the coordinate system to -0.5 -> 0.5, and also because of the fact that we had to adjust the value of uv.x.

If you now add this to the GLSL code…

color = frontImage;

…you will see our image, rendered by a GLSL shader:

Masking

Always keep in mind that, for shaders, everything is a number (yes, even images), and that 0.0 means completely hidden while 1.0 stands for fully visible.

We can now use the mask we’ve just created to hide the parts of our image where the value of the mask equals 0.0.

With that in mind, it’s pretty easy to apply our mask. The only thing we have to do is multiply the color variable by the value of the mask:

// The mask should be a separate variable, not set as the `color` value
vec3 mask = Rectangle(maskSize, uv, maskPosition, maskColor);

// Some super magic trick
color = frontImage * mask;

Et voilà, we now have a fully functioning mask effect:

Let’s enhance it a bit by making the mask follow a circular path.

In order to do that we must go back to our JS file and add a couple of lines of code.

// Add this to the class constructor
this.time = 0

// This goes inside the `registerBeforeRender` callback
this.time++;
plane.material.setFloat("u_time", this.time);

In the fragment shader, first declare the new uniform at the top of the file:

uniform float u_time;

Then, edit the declaration of maskPosition like this:

vec2 maskPosition = vec2(
  cos(u_time * 0.05) * 0.2 - 0.15,
  sin(u_time * 0.05) * 0.2 - 0.15
);

u_time is simply one of the uniforms that we pass to our shader from the WebGL program.

The only difference with the u_frontTexture uniform is that we increase its value on each render loop and pass its new value to the shader, so that it updates the mask’s position.

Here’s a live preview of the mask going in a circle:

Adding the background image

In order to add the background image we’ll do the exact opposite of what we did for the foreground image.

Let’s go one step at a time.

First, in the JS class, pass the shader the background image in the same way we did for the foreground image:

const backTexture = new Texture("src/images/lantern-bw.jpg");
plane.material.setTexture("u_backTexture", backTexture);

Then, tell the fragment shader that we’re passing it that u_backTexture and initialize another vec3 variable:

// This goes at the top of the file
uniform sampler2D backTexture;

// Add this after `vec3 frontImage = ...`
vec3 backgroundImage = texture2D(iChannel1, uv * 0.5 + 0.5).rgb;

When you do a quick test by replacing

color = frontImage * mask;

with

color = backImage * mask;

you’ll see the background image.

But for this one, we have to invert the mask to make it behave the opposite way.

Inverting a number is really easy, the formula is:

invertedNumber = 1 - <number>

So, let’s apply the inverted mask to the background image:

backImage *= (1.0 - mask);

Here, we’re applying the same mask we added to the foreground image, but since we inverted it, the effect is the opposite.

Putting it all together

At this point, we can refactor the declaration of the two images by directly applying their masks.

vec3 frontImage = texture2D(u_frontTexture, uv * 0.5 + 0.5).rgb * mask;
vec3 backImage = texture2D(u_backTexture, uv * 0.5 + 0.5).rgb * (1.0 - mask);

We can now display both images by adding backImage to frontImage:

color = backImage + frontImage;

That’s it, here’s a live example of the desired effect:

Distorting the mask

Cool uh? But it’s not over yet! Let’s tweak it a bit by distorting the mask.

To do so, we first have to create a new vec2 variable:

vec2 maskUV = vec2(
  uv.x + sin(u_time * 0.03) * sin(uv.y * 5.0) * 0.15,
  uv.y + cos(u_time * 0.03) * cos(uv.x * 10.0) * 0.15
);

Then, replace uv with maskUV in the mask declaration

vec3 mask = Rectangle(maskSize, maskUV, maskPosition, maskColor);

In maskUV, we’re using some math to add uv values based on the u_time uniform and the current uv.

Try tweaking those values by yourself to see different effects.

Distorting the foreground image

Let’s now distort the foreground image the same way we did for the mask, but with slightly different values.

Create a new vec2 variable to store the foreground image uvs:

vec2 frontImageUV = vec2(
  (uv.x + sin(u_time * 0.04) * sin(uv.y * 10.) * 0.03),
  (uv.y + sin(u_time * 0.03) * cos(uv.x * 15.) * 0.05)
);

Then, use that frontImageUV instead of the default uv when declaring frontImage:

vec3 frontImage = texture2D(u_frontTexture, frontImageUV * 0.5 + 0.5).rgb * mask;

Voilà! Now both the mask and the image have a distortion effect applied.

Again, try tweaking those numbers to see how the effect changes.

10 – Adding mouse control

What we’ve made so far is really cool, but we could make it even cooler by adding some mouse control like making it fade in/out when the mouse hovers/leaves the plane and making the mask follow the cursor.

Adding fade effects

In order to detect the mouseover/mouseleave events on a mesh and execute some code when those events occur we have to use BabylonJS’s actions.

Let’s start by importing some new modules:

import { ActionManager } from "@babylonjs/core/Actions/actionManager";
import { ExecuteCodeAction } from "@babylonjs/core/Actions/directActions";
import "@babylonjs/core/Culling/ray";

Then add this code after the creation of the plane:

this.plane.actionManager = new ActionManager(this.scene);

this.plane.actionManager.registerAction(
  new ExecuteCodeAction(ActionManager.OnPointerOverTrigger, () =>
    this.onPlaneHover()
  )
);

this.plane.actionManager.registerAction(
  new ExecuteCodeAction(ActionManager.OnPointerOutTrigger, () =>
    this.onPlaneLeave()
  )
);

Here we’re telling the plane’s ActionManager to listen for the PointerOver and PointerOut events and execute the onPlaneHover() and onPlaneLeave() methods, which we’ll add right now:

onPlaneHover() {
  console.log('hover')
}

onPlaneLeave() {
  console.log('leave')
}

Some notes about the code above

Please note that I’ve used this.plane instead of just plane; that’s because we’ll have to access it from within the mousemove event’s callback later, so I’ve refactored the code a bit.

ActionManager allows us to listen to certain events on a target, in this case the plane.

ExecuteCodeAction is a BabylonJS action that we’ll use to execute some arbitrary code.

ActionManager.OnPointerOverTrigger and ActionManager.OnPointerOutTrigger are the two events that we’re listening to on the plane. They behave exactly like the mouseenter and mouseleave events for DOM elements.

To detect hover events in WebGL, we need to “cast a ray” from the position of the mouse to the mesh we’re checking; if that ray, at some point, intersects with the mesh, it means that the mouse is hovering it. This is why we’re importing the @babylonjs/core/Culling/ray module; BabylonJS will take care of the rest.

Now, if you test it by hovering and leaving the mesh, you’ll see that it logs hover and leave.

Now, let’s add the fade effect. For this, I’ll use the GSAP library, which is the de-facto library for complex and high-performant animations.

First, install it:

yarn add gsap

Then, import it in our class

import gsap from 'gsap

and add this line to the constructor

this.maskVisibility = { value: 0 };

Finally, add this line to the registerBeforeRender()‘s callback function

this.plane.material.setFloat( "u_maskVisibility", this.maskVisibility.value);

This way, we’re sending the shader the current value property of this.maskVisibility as a new uniform called u_maskVisibility.

Refactor the fragment shader this way:

// Add this at the top of the file, like any other uniforms
uniform float u_maskVisibility;

// When declaring `maskColor`, replace `1.0` with the `u_maskVisibility` uniform
vec3 maskColor = vec3(u_maskVisibility);

If you now check the result, you’ll see that the foreground image is not visible anymore; what happened?

Do you remember when I wrote that “for shaders, everything is a number”? That’s the reason! The u_maskVisibility uniform equals 0.0, which means that the mask is invisible.

We can fix it in few lines of code. Open the JS code and refactor the onPlaneHover() and onPlaneLeave() methods this way:

onPlaneHover() {
  gsap.to(this.maskVisibility, {
    duration: 0.5,
    value: 1
  });
}

onPlaneLeave() {
  gsap.to(this.maskVisibility, {
    duration: 0.5,
    value: 0
  });
}

Now, when you hover or leave the plane, you’ll see that the mask fades in and out!

(And yes, BabylonJS has it’s own animation engine, but I’m way more confident with GSAP, that’s why I opted for it.)

Make the mask follow the mouse cursor

First, add this line to the constructor

this.maskPosition = { x: 0, y: 0 };

and this to the addListeners() method:

window.addEventListener("mousemove", () => {
  const pickResult = this.scene.pick(
    this.scene.pointerX,
    this.scene.pointerY
  );

  if (pickResult.hit) {
    const x = pickResult.pickedPoint.x / this.plane.scaling.x;
    const y = pickResult.pickedPoint.y / this.plane.scaling.y;

    this.maskPosition = { x, y };
  }
});

What the code above does is pretty simple: on every mousemove event it casts a ray with this.scene.pick() and updates the values of this.maskPosition if the ray is intersecting something.

(Since we have only a single mesh we can avoid checking what mesh is being hit by the ray.)

Again, on every render loop, we send the mask position to the shader, but this time as a vec2. First, import the Vector2 module together with Vector3

import { Vector2, Vector3 } from "@babylonjs/core/Maths/math";

Add this in the runRenderLoop callback function

this.plane.material.setVector2(
  "u_maskPosition",
  new Vector2(this.maskPosition.x, this.maskPosition.y)
);

Add the u_maskPosition uniform at the top of the fragment shader

uniform vec2 u_maskPosition;

Finally, refactor the maskPosition variable this way

vec3 maskPosition = vec2(
  u_maskPosition.x * uPlaneRatio - 0.15,
  u_maskPosition.y - 0.15
);

Side note; I’ve adjusted the x using the uPlaneRatio value because at the beginning of the main() function I did the same with the shader’s uvs

And here you can see the result of your hard work:

Conclusion

As you can see, doing these kind of things doesn’t involve too much code (~150 lines of JavaScript and ~50 lines of GLSL, including comments and empty lines); the hard part with WebGL is the fact that it’s complex by nature, and it’s a very vast subject, so vast that many times I don’t even know what to search on Google when I get stuck.

Also, you have to study a lot, way more than with “standard” website development. But in the end, it’s really fun to work with.

In this tutorial, I tried to explain the whole process (and the reasoning behind everything) step by step, just like I want someone to explain it to me; if you’ve reached this point of this tutorial, it means that I’ve reached my goal.

In any case, thanks!

Credits

The lantern image is by Vladimir Fetodov

Creating a Distorted Mask Effect on an Image with Babylon.js and GLSL was written by Francesco Michelini and published on Codrops.