Do you have a NodeJS application serving production traffic? Ever wonder how Yahoo might handle this? If yes, at some point or other you must have thought it would be nice to have a clean and simple solution for high availability.

Read below for a detailed overview of how you can achieve this just like we did here at Yahoo.

What modules are needed?

• monitr https://npmjs.org/package/monitr
• process-watcher https://npmjs.org/package/process-watcher
• mod_statuspage https://npmjs.org/package/mod_statuspage

How does it work?

Use monitr to start monitoring when your application gets loaded with this simple statement monitor.start() and stop monitoring with monitor.stop() when your application shuts down.

The monitr module creates a thread and starts monitoring the application. It looks for process status from various system files and reports the status on a socket.

The process-watcher module provides a listener on this socket. This listener is also responsible for sending a SIGHUP and SIGKILL if the process did not honor the performance boundaries given to the watcher.

How do we add it to an existing/new node app?

MONITOR

// Your includes

var monitor = require(‘monitr’);
monitor.start();

// Your application logic

process.on(‘exit’, function () {
    monitor.stop();
});

LISTENER

var watcher = require('process-watcher');

/*
 * Dummy metric monitoring object.
 */
var watcher_metric = {
    increment : function (name, v) {
    },
    set : function (name, v) {
    }
};

var dgpath = '/tmp/nodejs.mon',
    statusPath = '/tmp/watcher_status_path_test',
    watcher_config = {
	    max_inactive : 0.001,
	    monitor : 0.001,
	    monPath: dgpath,
	    timeout : 3,
	    timeout_start : 60
    };

//Instantiate watcher
var watcher_instance = new watcher.Watcher({
    metric: watcher_metric,
    config : watcher_config
});

In this ecosystem, how do we view application behavior?

The mod_statuspage module provides a simple middleware to view each process status on the specified server.

//Your includes
var app = express();

app.use(status({
        url: '/status',
        check: function(req) {
            if (req.something == false) {
                return false;
            }
            return true;
        },
        responseContentType : 'html'
    }));

//Other middleware
app.use(...
//URLs served
app.get(...

Putting it all together …

Source

Example application and listener

Running it

Run your application and watcher on separate terminals

node ./application.js node ./listener.js

Status page

Once you have your application and the watcher running, you can view your application status at http://localhost:8000/status

Test URL to trigger watcher to kill the worker process

http://localhost:8000/late-response

You can see from the application that the URL ‘late-response’ deliberately engages CPU for 5 seconds. The watcher is configured to kill the worker processes that are idle more than 3 seconds.

Hence hitting this URL above will cause the watcher to send a SIGHUP to the worker process that took the request.

Console Output (on a 2 core box)

Starting worker 0
Starting worker 1
Worker 23107 online
Worker 23108 online
Go to: http://127.0.0.1:8000/status
Go to: http://127.0.0.1:8000/status
                   >>>> Access URL http://localhost:8000/late-response
worker 23107 died
Worker 23114 online
Go to: http://127.0.0.1:8000/status

Make sure you check out all of our other NodeJS modules on our Github page, they are all used in production here at Yahoo.