Server-Side Visualization With Nightmare

Celý článek

This is an extract from chapter 11 of Ashley Davis’s book Data Wrangling with JavaScript now available on the Manning Early Access Program. I absolutely love this idea as there is so much data visualization stuff on the web that relies on fully functioning client side JavaScript and potentially more API calls. It’s not nearly as robust, accessible, or syndicatable as it could be. If you bring that data visualization back to the server, you can bring progressive enhancement to the party. All example code and data can be found on GitHub.

When doing exploratory coding or data analysis in Node.js it is very useful to be able to render a visualization from our data. If we were working in browser-based JavaScript we could choose any one of the many charting, graphics, and visualization libraries. Unfortunately, under Node.js, we don’t have any viable options, so how otherwise can we achieve this?

We could try something like faking the DOM under Node.js, but I found a better way. We can make our browser-based visualization libraries work for us under Node.js using a headless browser. This is a browser that has no user interface. You can think of it as a browser that is invisible.

I use Nightmare under Node.js to capture visualizations to PNG and PDF files and it works really well!

The headless browser

When we think of a web-browser we usually think of the graphical software that we interact with on a day to day basis when browsing the web. Normally we interact with such a browser directly, viewing it with our eyes and controlling it with our mouse and keyboard as shown in Figure 1.

A headless browser on the other hand is a web-browser that has no graphical user interface and no direct means for us to control it. You might ask what is the use of a browser that we can’t directly see or interact with.

Well, as developers we would typically use a headless browser for automating and testing web sites. Let’s say that you have created a web page and you want to run a suite of automated tests against it to prove that it works as expected. The test suite is automated, which means it is controlled from code and this means that we need to drive the browser from code.

We use a headless browser for automated testing because we don’t need to directly see or interact with the web page that is being tested. Viewing such an automated test in progress is unnecessary, all we need to know is if the test passed or failed — and if it failed we would like to know why. Indeed, having a GUI for the browser under test would actually be a hindrance for a continuous-integration or continuous-deployment server, where many such tests can run in parallel.

So headless browsers are often used for automated testing of our web pages, but they are also incredibly useful for capturing browser-based visualizations and outputting them to PNG images or PDF files. To make this work we need a web server and a visualization, we must then write code to instance a headless browser and point it at our web server. Our code then instructs the headless browser to take a screenshot of the web page and save it to our file system as a PNG or PDF file.

Nightmare is my headless browser of choice. It is a Node.js library (installed via npm) that is built on Electron. Electron is a framework normally used for building cross-platform desktop apps that are based on web-technologies.

Why Nightmare?

It’s called Nightmare, but it’s definitely not a Nightmare to use. In fact, it’s the simplest and most convenient headless browser that I’ve used. It automatically includes Electron, so to get started we simply install Nightmare into our Node.js project as follows:

npm install --save nightmare

That’s all we need to install Nightmare and we can start using it immediately from JavaScript!

Nightmare comes with almost everything we need: A scripting library with an embedded headless browser. It also includes the communication mechanism to control the headless browser from Node.js. For the most part it’s seamless and well-integrated to Node.js.

Electron is built on Node.js and Chromium and maintained by GitHub and is the basis for a number of popular desktop applications.

Here are the reasons that I choose to use Nightmare over any other headless browser:

• Electron is very stable.
• Electron has good performance.
• The API is simple and easy to learn.
• There is no complicated configuration (just start using it).
• It is very well integrated with Node.js.

Nightmare and Electron

When you install Nightmare via npm it automatically comes with an embedded version of Electron. So, we can say that Nightmare is not just a library for controlling a headless browser, it effectively is the headless browser. This is another reason I like Nightmare. With some of the other headless browsers, the control library is separate, or it’s worse than that and they don’t have a Node.js control library at all. In the worst case, you have to roll your own communication mechanism to control the headless browser.

Nightmare creates an instance of the Electron process using the Node.js child_process module. It then uses inter-process communication and a custom protocol to control the Electron instance. The relationship is shown in Figure 3.

Our process: Capturing visualizations with Nightmare

So what is the process of capturing a visualization to an image file? This is what we are aiming at:

1. Acquire data.
2. Start a local web server to host our visualization
3. Inject our data into the web server
4. Instance a headless browser and point it at our local web server
5. Wait for the visualization to be displayed
6. Capture a screenshot of the visualization to an image file
7. Shutdown the headless browser
8. Shutdown the local web server

Prepare a visualization to render

The first thing we need is to have a visualization. Figure 4 shows the chart we’ll work with. This a chart of New York City yearly average temperature for the past 200 years.

To run this code you need Node.js installed. For this first example we’ll also use live-server (any web server will do) to test the visualization (because we haven’t created our Node.js web server yet), install live server as follows:

npm install -g live-server

Then you can clone the example code repo for this blog post:

git clone https://github.com/Data-Wrangling-with-JavaScript/nodejs-visualization-example

Now go into the repo, install dependencies and run the example using live-server

cd nodejs-visualization-example/basic-visualization
bower install
live-server

When you run live-server your browser should automatically open and you should see the chart from Figure 4.

It’s a good idea to check that your visualization works directly in a browser before you try and capture it in a headless browser; there could easily be something wrong with it and problems are much easier to troubleshoot in a real browser than in the headless browser. live-server has live reload built-in, so now you have a nice little setup here when you can edit and improve the chart interactively before you try to capture it under Node.js.

This simple line chart was constructed with C3. Please take a look over the example code and maybe look at some of the examples in the C3 gallery to learn more about C3.

Starting the web server

To host our visualization, we need a web server. It’s not quite enough that we have a web server, we also need to be able to dynamically start and stop it. Listing 1 shows the code for our web server.

Listing 1 – Code for a simple web server that can be started and stopped

const express = require('express');
const path = require('path');
 
module.exports = {
  start: () => { // Export a start function so we can start the web server on demand.
    return new Promise((resolve, reject) => {
      const app = express();

      const staticFilesPath = path.join(__dirname, "public"); // Make our 'public' sub-directory accessible via HTTP. 
      const staticFilesMiddleWare = express.static(staticFilesPath);
      app.use('/', staticFilesMiddleWare);
     
      const server = app.listen(3000, err => { // Start the web server!
        if (err) {
          reject(err); // Error occurred while starting web server.
        }
        else {
          resolve(server); // Web server started ok.
        }
      });                        
    });
  }
}

The code module in listing 1 exports a start function that we can call to kickstart our web server. This technique, being able to start and stop our web server, is also very useful for doing automated integration testing on a web site. Imagine that you want to start your web server, run some tests against it and then stop it at the end.

So now we have our browser-based visualization and we have a web server that can be started and stopped on demand. These are the raw ingredients we need for capturing server-side visualizations. Let’s mix it up with Nightmare!

Rendering the web page to an image

Now let’s flesh out the code to capture a screenshot of the visualization with Nightmare. Listing 2 shows the code that instances Nightmare, points it at our web server and then takes the screenshot.

Listing 2 – Capturing our chart to an image file using Nightmare

const webServer = require('./web-server.js');
const Nightmare = require('nightmare');
 
webServer.start() // Start the web server.
.then(server => {
  const outputImagePath = "./output/nyc-temperatures.png";

  const nightmare = new Nightmare(); // Create the Nightmare instance.
  return nightmare.goto("http://localhost:3000") // Point the browser at the web server we just started.
    .wait("svg") // Wait until the chart appears on screen.
    .screenshot(outputImagePath) // Capture a screenshot to an image file.
    .end() // End the Nightmare session. Any queued operations are completed and the headless browser is terminated.
    .then(() => server.close()); // Stop the web server when we are done.
})
.then(() => {
  console.log("All done :)");
})
.catch(err => {
  console.error("Something went wrong :(");
  console.error(err);
});

Note the use of the goto function, this is what actually directs the browser to load our visualization.

Web pages usually take some time to load. That’s probably not going to be very long, especially as we are running a local web server, but still we face the danger of taking a screenshot of the headless browser before or during its initial paint. That’s why we must call the wait function to wait until the chart’s <svg> element appears in the browser’s DOM before we call the screenshot function.

Eventually, the end function is called. Up until now we have effectively built a list of commands to send to the headless browser. The end function actually sends the commands to the browser, which takes the screenshot and outputs the file nyc-temperatures.png. After the image file has been captured we finish up by shutting down the web server.

You can find the completed code under the capture-visualization sub-directory in the repo. Go into the sub-directory and install dependencies:

cd nodejs-visualization-example/capture-visualization
cd public 
bower install
cd ..
npm install
live-server

Now you can try the code for yourself:

node index.js

This has been an extract from chapter 11 of Data Wrangling with JavaScript now available on the Manning Early Access Program. Please use this discount code fccdavis3 for a 37% discount. Please check The Data Wrangler for new updates on the book.

The post Server-Side Visualization With Nightmare appeared first on CSS-Tricks.