node_modules.xml

<?xml version="1.0"?>

<!--
  Copyright (C) Nginx, Inc.
  -->

<!DOCTYPE article SYSTEM "../../../../dtd/article.dtd">

<article name="Using node modules with njs"
        link="/en/docs/njs/node_modules.html"
        lang="en"
        rev="6">

<section id="intro">

<para>
Often, a developer wants to use 3rd-party code,
usually available as a library of some kind.
In the JavaScript world, the concept of a module is relatively new,
so there was no standard until recently.
Many platforms (browsers) still don't support modules, which makes code
reuse harder.
This article describes ways to reuse
<link url="https://nodejs.org/">Node.js</link> code in njs.
</para>

<note>
Examples in this article use features that appeared in
<link doc="index.xml">njs</link>
<link doc="changes.xml" id="njs0.3.8">0.3.8</link>
</note>

<para>
There is a number of issues
that may arise when 3rd-party code is added to njs:

<list type="bullet">

<listitem>
Multiple files that reference each other and their dependencies
</listitem>

<listitem>
Platform-specific APIs
</listitem>

<listitem>
Modern standard language constructions
</listitem>

</list>
</para>

<para>
The good news is that such problems are not something new or specific to njs.
JavaScript developers face them daily
when trying to support multiple disparate platforms
with very different properties.
There are instruments designed to resolve the above-mentioned issues.

<list type="bullet">

<listitem>
Multiple files that reference each other, and their dependencies
<para>
This can be solved by merging all the interdependent code into a single file.
Tools like
<link url="http://browserify.org/">browserify</link> or
<link url="https://webpack.js.org/">webpack</link>
accept an entire project and produce a single file containing
your code and all the dependencies.
</para>
</listitem>

<listitem>
Platform-specific APIs
<para>
You can use multiple libraries that implement such APIs
in a platform-agnostic manner (at the expense of performance, though).
Particular features can also be implemented using the
<link url="https://polyfill.io/v3/">polyfill</link> approach.
</para>
</listitem>

<listitem>
Modern standard language constructions
<para>
Such code can be transpiled:
this means performing a number of transformations
that rewrite newer language features in accordance with an older standard.
For example, <link url="https://babeljs.io/"> babel</link> project
can be used to this purpose.
</para>
</listitem>

</list>
</para>

<para>
In this guide, we will use two relatively large npm-hosted libraries:

<list type="bullet">

<listitem>
<link url="https://www.npmjs.com/package/protobufjs">protobufjs</link>&mdash;
a library for creating and parsing protobuf messages used by the
<link url="https://grpc.io/">gRPC</link> protocol
</listitem>

<listitem>
<link url="https://www.npmjs.com/package/dns-packet">dns-packet</link>&mdash;
a library for processing DNS protocol packets
</listitem>

</list>
</para>

</section>


<section id="environment" name="Environment">

<para>
<note>
This document mostly employs a generic approach
and avoids specific best practice advices concerning Node.js
and JavaScript.
Make sure to consult the corresponding package's manual
before following the steps suggested here.
</note>
First (assuming Node.js is installed and operational), let's create an
empty project and install some dependencies;
the commands below assume we are in the working directory:
<example>
$ mkdir my_project &amp;&amp; cd my_project
$ npx license choose_your_license_here > LICENSE
$ npx gitignore node

$ cat &gt; package.json &lt;&lt;EOF
{
  "name":        "foobar",
  "version":     "0.0.1",
  "description": "",
  "main":        "index.js",
  "keywords":    [],
  "author":      "somename &lt;[email protected]&gt; (https://example.com)",
  "license":     "some_license_here",
  "private":     true,
  "scripts": {
    "test": "echo \"Error: no test specified\" &amp;&amp; exit 1"
  }
}
EOF
$ npm init -y
$ npm install browserify
</example>
</para>

</section>


<section id="protobuf" name="Protobufjs">

<para>
The library provides a parser
for the <literal>.proto</literal> interface definitions
and a code generator for message parsing and generation.
</para>

<para>
In this example, we will use the
<link url="https://github.com/grpc/grpc/blob/master/examples/protos/helloworld.proto">helloworld.proto</link>
file
from the gRPC examples.
Our goal is to create two messages:
<literal>HelloRequest</literal> and
<literal>HelloResponse</literal>.
We will use the
<link url="https://github.com/protobufjs/protobuf.js/blob/master/README.md#reflection-vs-static-code">static</link>
mode of protobufjs instead of dynamically generating classes, because
njs doesn't support adding new functions dynamically
due to security considerations.
</para>

<para>
Next, the library is installed and
the JavaScript code implementing message marshalling
is generated from the protocol definition:
<example>
$ npm install protobufjs
$ npx pbjs -t static-module helloworld.proto > static.js
</example>
</para>

<para>
Thus, the <literal>static.js</literal> file becomes our new dependency,
storing all the code we need to implement message processing.
The <literal>set_buffer()</literal> function contains code that uses the
library to create a buffer with the serialized
<literal>HelloRequest</literal> message.
The code resides in the <literal>code.js</literal> file:
<example>
var pb = require('./static.js');

// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
    // set fields of gRPC payload
    var payload = { name: "TestString" };

    // create an object
    var message = pb.helloworld.HelloRequest.create(payload);

    // serialize object to buffer
    var buffer = pb.helloworld.HelloRequest.encode(message).finish();

    var n = buffer.length;

    var frame = new Uint8Array(5 + buffer.length);

    frame[0] = 0;                        // 'compressed' flag
    frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24;  // length: uint32 in network byte order
    frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
    frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt;  8;
    frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt;  0;

    frame.set(buffer, 5);

    return frame;
}

var frame = set_buffer(pb);
</example>
</para>

<para>
To ensure it works, we execute the code using node:
<example>
$ node ./code.js
Uint8Array [
    0,   0,   0,   0,  12, 10,
   10,  84, 101, 115, 116, 83,
  116, 114, 105, 110, 103
]
</example>
You can see that this got us a properly encoded <literal>gRPC</literal> frame.
Now let's run it with njs:
<example>
$ njs ./code.js
Thrown:
Error: Cannot find module "./static.js"
    at require (native)
    at main (native)
</example>
</para>

<para>
Modules are not supported, so we've received an exception.
To overcome this issue, let's use <literal>browserify</literal>
or other similar tool.
</para>

<para>
An attempt to process our existing <literal>code.js</literal> file will result
in a bunch of JS code that is supposed to run in a browser,
i.e. immediately upon loading.
This isn't something we actually want.
Instead, we want to have an exported function that
can be referenced from the nginx configuration.
This requires some wrapper code.
<note>
In this guide, we use
njs <link doc="cli.xml">cli</link> in all examples for the sake of simplicity.
In real life, you will be using nginx njs module to run your code.
</note>
</para>

<para>
The <literal>load.js</literal> file contains the library-loading code that
stores its handle in the global namespace:
<example>
global.hello = require('./static.js');
</example>
This code will be replaced with merged content.
Our code will be using the "<literal>global.hello</literal>" handle to access
the library.
</para>

<para>
Next, we process it with <literal>browserify</literal>
to get all dependencies into a single file:
<example>
$ npx browserify load.js -o bundle.js -d
</example>
The result is a huge file that contains all our dependencies:
<example>
(function(){function......
...
...
},{"protobufjs/minimal":9}]},{},[1])
//# sourceMappingURL..............
</example>
To get final "<literal>njs_bundle.js</literal>" file we concatenate
"<literal>bundle.js</literal>" and the following code:
<example>
// Example usage of protobuf library: prepare a buffer to send
function set_buffer(pb)
{
    // set fields of gRPC payload
    var payload = { name: "TestString" };

    // create an object
    var message = pb.helloworld.HelloRequest.create(payload);

    // serialize object to buffer
    var buffer = pb.helloworld.HelloRequest.encode(message).finish();

    var n = buffer.length;

    var frame = new Uint8Array(5 + buffer.length);

    frame[0] = 0;                        // 'compressed' flag
    frame[1] = (n &amp; 0xFF000000) &gt;&gt;&gt; 24;  // length: uint32 in network byte order
    frame[2] = (n &amp; 0x00FF0000) &gt;&gt;&gt; 16;
    frame[3] = (n &amp; 0x0000FF00) &gt;&gt;&gt;  8;
    frame[4] = (n &amp; 0x000000FF) &gt;&gt;&gt;  0;

    frame.set(buffer, 5);

    return frame;
}

// functions to be called from outside
function setbuf()
{
    return set_buffer(global.hello);
}

// call the code
var frame = setbuf();
console.log(frame);
</example>
Let's run the file using node to make sure things still work:
<example>
$ node ./njs_bundle.js
Uint8Array [
    0,   0,   0,   0,  12, 10,
   10,  84, 101, 115, 116, 83,
  116, 114, 105, 110, 103
]
</example>
Now let's proceed further with njs:
<example>
$ njs ./njs_bundle.js
Uint8Array [0,0,0,0,12,10,10,84,101,115,116,83,116,114,105,110,103]
</example>
The last thing will be to use njs-specific API to convert
array into byte string, so it could be usable by nginx module.
We can add the following snippet before the line
<literal>return frame; }</literal>:
<example>
if (global.njs) {
    return String.bytesFrom(frame)
}
</example>
Finally, we got it working:
<example>
$ njs ./njs_bundle.js |hexdump -C
00000000  00 00 00 00 0c 0a 0a 54  65 73 74 53 74 72 69 6e  |.......TestStrin|
00000010  67 0a                                             |g.|
00000012
</example>
This is the intended result.
Response parsing can be implemented similarly:
<example>
function parse_msg(pb, msg)
{
    // convert byte string into integer array
    var bytes = msg.split('').map(v=>v.charCodeAt(0));

    if (bytes.length &lt; 5) {
        throw 'message too short';
    }

    // first 5 bytes is gRPC frame (compression + length)
    var head = bytes.splice(0, 5);

    // ensure we have proper message length
    var len = (head[1] &lt;&lt; 24)
              + (head[2] &lt;&lt; 16)
              + (head[3] &lt;&lt; 8)
              + head[4];

    if (len != bytes.length) {
        throw 'header length mismatch';
    }

    // invoke protobufjs to decode message
    var response = pb.helloworld.HelloReply.decode(bytes);

    console.log('Reply is:' + response.message);
}
</example>
</para>

</section>


<section id="dnspacket" name="DNS-packet">

<para>
This example uses a library for generation and parsing of DNS packets.
This a case worth considering because the library and its dependencies
use modern language constructions not yet supported by njs.
In turn, this requires from us an extra step: transpiling the source code.
</para>

<para>
Additional node packages are needed:
<example>
$ npm install @babel/core @babel/cli @babel/preset-env babel-loader
$ npm install webpack webpack-cli
$ npm install buffer
$ npm install dns-packet
</example>
The configuration file, webpack.config.js:
<example>
const path = require('path');

module.exports = {
    entry: './load.js',
    mode: 'production',
    output: {
        filename: 'wp_out.js',
        path: path.resolve(__dirname, 'dist'),
    },
    optimization: {
        minimize: false
    },
    node: {
        global: true,
    },
    module : {
        rules: [{
            test: /\.m?js$$/,
            exclude: /(bower_components)/,
            use: {
                loader: 'babel-loader',
                options: {
                    presets: ['@babel/preset-env']
                }
            }
        }]
    }
};
</example>
Note we are using "<literal>production</literal>" mode.
In this mode webpack does not use "<literal>eval</literal>" construction
not supported by njs.
The referenced <literal>load.js</literal> file is our entry point:
<example>
global.dns = require('dns-packet')
global.Buffer = require('buffer/').Buffer
</example>
We start the same way, by producing a single file for the libraries:
<example>
$ npx browserify load.js -o bundle.js -d
</example>
Next, we process the file with webpack, which itself invokes babel:
<example>
$ npx webpack --config webpack.config.js
</example>
This command produces the <literal>dist/wp_out.js</literal> file, which is a
transpiled version of <literal>bundle.js</literal>.
We need to concatenate it with <literal>code.js</literal>
that stores our code:
<example>
function set_buffer(dnsPacket)
{
    // create DNS packet bytes
    var buf = dnsPacket.encode({
        type: 'query',
        id: 1,
        flags: dnsPacket.RECURSION_DESIRED,
        questions: [{
            type: 'A',
            name: 'google.com'
        }]
    })

    return buf;
}
</example>
Note that in this example generated code is not wrapped into function and we
do not need to call it explicitly.
The result is in the "<literal>dist</literal>" directory:
<example>
$ cat dist/wp_out.js code.js > njs_dns_bundle.js
</example>
Let's call our code at the end of a file:
<example>
var b = set_buffer(global.dns);
console.log(b);
</example>
And execute it using node:
<example>
$ node ./njs_dns_bundle_final.js
Buffer [Uint8Array] [
    0,   1,   1, 0,  0,   1,   0,   0,
    0,   0,   0, 0,  6, 103, 111, 111,
  103, 108, 101, 3, 99, 111, 109,   0,
    0,   1,   0, 1
]
</example>
Make sure this works as expected, and then run it with njs:
<example>
$ njs ./njs_dns_bundle_final.js
Uint8Array [0,1,1,0,0,1,0,0,0,0,0,0,6,103,111,111,103,108,101,3,99,111,109,0,0,1,0,1]
</example>

</para>

<para>
The response can be parsed as follows:
<example>
function parse_response(buf)
{
    var bytes = buf.split('').map(v=>v.charCodeAt(0));

    var b = global.Buffer.from(bytes);

    var packet = dnsPacket.decode(b);

    var resolved_name = packet.answers[0].name;

    // expected name is 'google.com', according to our request above
}
</example>
</para>

</section>

</article>