github

ehmicky / portable-node-guide

  • вторник, 12 марта 2019 г. в 00:16:49
https://github.com/ehmicky/portable-node-guide


📗 Practical guide on how to write portable/cross-platform Node.js code



License Gitter

Practical guide on how to write portable/cross-platform Node.js code.

Why you should care

According to the 2018 Node.js user survey (using the raw data), 24% of Node.js developers use Windows locally and 41% use Mac. In production 85% use Linux and 1% use BSD.

About this guide

If you find this document too long, you can jump to the summary.

If you want to keep up to date on portability issues introduced with each new Node.js release, follow @ehmicky on Twitter.

Did you find an error or want to add more information? Issues and PRs are welcome!

Table of contents

Installing and updating Node

Installers for each major OS are available on the Node.js website.

To install, switch and update Node.js versions nvm can be used on Linux/Mac. It does not support Windows but nvm-windows and nvs are alternatives that do.

To upgrade npm on Windows, it is convenient to use npm-windows-upgrade.

Core utilities

Each OS has its own set of (from the lowest to the highest level):

Directly executing one of those binaries (e.g. calling sed) won't usually work on every OS.

There are several approaches to solve this:

  • Most Node.js API core modules abstract this (mostly through libuv). E.g. the child_process methods are executing OS-specific system calls under the hood.
  • some projects abstract OS-specific core utilities like:
  • some projects like opn abstract common user applications.

Few lower-level tools attempt to bring cross-platform compatibility by emulating or translating system calls:

  • Wine: to run Windows API calls on Linux, Mac, BSD and Solaris.
  • Cygwin: to run POSIX on Windows.
  • WSL: to run the Linux command line on Windows (ELF binary execution, system calls, filesystem, Bash, core utilities, common applications).

Testing

Any OS can be run locally using virtual machines. Windows provides with official images.

It is recommended to run automated tests on a continuous integration provider that supports Linux, Mac and Windows, which most high-profile providers now do.

C/C++ addons

Windows users must first run npm install -g windows-build-tools as an admin before being able to install C/C++ addons.

Directory locations

Typical directory locations are OS-specific:

  • the main temporary directory could for example be /tmp on Linux, /var/folders/.../T on Mac or C:\Users\USER\AppData\Local\Temp on Windows. os.tmpdir() can be used to retrieve it on any OS.
  • the user's home directory could for example be /home/USER on Linux, /Users/USER on Mac or C:\Users\USER on Windows. os.homedir() can be used to retrieve it on any OS.

Man pages are Unix-specific so the package.json's man field does not have any effects on Windows.

System configuration

While Unix usually stores system configuration as files, Windows uses the registry, a central key-value database. Some projects like node-winreg, rage-edit or windows-registry-node can be used to access it from Node.

This should only be done when accessing OS-specific settings. Otherwise storing configuration as files or remotely is easier and more portable.

Character encoding

The character encoding on Unix is usually UTF-8. However on Windows it is usually either UTF-16 or one of the Windows code pages. Few non-Unicode character encodings are also popular in some countries. This can result in characters not being printed properly, especially high Unicode code points and emoji.

The character encoding can be specified using an encoding option with most relevant Node.js core methods. UTF-8 is always the default value except for readable streams (including fs.createReadStream()), fs.readFile() and most crypto methods where buffer is the default instead.

To convert between character encodings string_encoder (decoding only), Buffer.transcode(), TextDecoder and TextEncoder can be used.

Node.js supports UTF-8, UTF-16 little endian, Latin-1 and ASCII, except for TextDecoder and TextEncoder which support UTF-8, UTF-16 little endian and UTF-16 big endian by default. If Node.js is built with full internationalization support or provided with it at runtime, many more character encodings are supported by TextDecoder and TextEncoder. If doing so is inconvenient, iconv-lite or iconv can be used instead.

It is recommended to always use UTF-8. When reading from a file or terminal, one should either:

When writing to a terminal the character encoding will almost always be UTF-8 on Unix and CP866 on Windows (cmd.exe). figures and log-symbols can be used to print common symbols consistently across platforms.

Newlines

The character representation of a newline is OS-specific. On Unix it is \n (line feed) while on Windows it is \r\n (carriage return followed by line feed).

Newlines inside a template string translate to \n on any OS.

const string = `this is
an example`

Some Windows applications, including the cmd.exe terminal, print \n as newlines, so using \n will work just fine. However some Windows applications don't, which is why when reading from or writing to a file the OS-specific newline os.EOL should be used instead of \n.

EOF and BOM

The substitute character (CTRL-Z) stops file streams on some Windows commands when in text mode. This includes the type command in cmd.exe. As a consequence that character should be avoided in non-binary files.

As opposed to Windows, Unix does not implicitely add a newline at the end of files. Thus it is recommended to end files with a newline character. However please remember that Windows will print these as if two newlines were present instead.

The BOM is a special character at the beginning of a file indicating its endianness and character encoding. Since it creates issues with shebangs and adds little value to UTF-8, it is better not to add it to new files. However if a BOM is present in input, it should be properly handled. Fortunately this is the default behavior of Node.js core methods, so this should usually not create any issues.

Character encoding, newlines and EOF behavior should be specified with editorconfig. They can also be enforced with tools like ESLint and Prettier.

File paths

While / is used as a file path delimiter on Unix (/file/to/path), \ is used on Windows instead (\file\to\path). The path delimiter can be retrieved with path.sep. Windows actually allows using or mixing in / delimiters in file paths most of the time, but not always so this should not be relied on.

Furthermore absolute paths always start with / on Unix, but on Windows they can take many shapes:

  • \: the current drive.
  • C:\: a specific drive (here C:). This can also be used with relative paths like C:file\to\path.
  • \\HOST\: UNC path, for remote hosts.
  • \\?\: allows to overcome file path length limit of 260 characters. Those can be produced in Node.js with path.toNamespacedPath().
  • \\.\: device path.

When file paths are used as arguments to Node.js core methods:

When file paths are returned by Node.js core methods:

Outside of Node.js, i.e. when the path is input from (or output to) the terminal or a file, its syntax is OS-specific.

To summarize:

  • if a path must be output outside of Node.js (e.g. terminal or file), path.normalize() should be used to make it OS-specific.
  • if a path comes from outside of Node.js or from a core method, it will be OS-specific. However all Node.js core methods will properly handle it.
  • in all other cases using Unix paths will just work.

Filenames

Each OS tends to use its own file system: Windows uses NTFS, Mac uses APFS (previously HFS+) and Linux tends to use ext4, Btrfs or XFS. Each file system has its own restrictions when it comes to naming files and paths.

Portable filenames need to avoid:

  • any other characters but a-z, 0-9, -._,=()~
  • starting with -
  • ending with a .
  • uppercase characters (Mac and Windows are case-insensitive).
  • being more than 255 characters long.
  • being one of those names: com1, com2, com3, com4, com5, com6, com7, com8, com9, lpt1, lpt2, lpt3, lpt4, lpt5, lpt6, lpt7, lpt8, lpt9, con, nul, prn, aux.

Portable file paths need to avoid:

Shell

Unix usually comes with Bash but not always. Popular alternatives include Fish, Dash, tcsh, ksh and zsh.

Writing interoperable shell code can be somewhat achieved by using either:

  • sh the ancestor of most of those shells.
  • projects like modernish.

However this won't work on Windows which uses two other shells:

  • cmd.exe which comes by default.
  • Powershell which is more recent, featureful and complex.

cmd.exe is very different from Bash and has quite many limitations:

  • ; cannot be used to separate statements. However && can be used like in Bash.
  • CLI flags often use slashes (/opt) instead of dashes (-opt). But Node.js binaries can still use -opt.
  • Globbing (e.g. wildcard *) does not work.
  • Exit code are accessed with %errorlevel% instead of $?.
  • Escaping is done differently with double quotes and ^. This is partially solved with the child_process.spawn() option windowsVerbatimArguments which defaults to true when cmd.exe is used.

When the option shell of child_process.spawn() is true, /bin/sh will be used on Unix and cmd.exe (or the environment variable ComSpec) will be used on Windows. Since those shells behave differently it is better to avoid that option.

As a consequence it is recommended to:

  • keep shell commands to simple command arguments... calls
  • use execa() (not execa.shell()) to fire those.

Files execution

Shebang like #!/usr/bin/node do not work on Windows, where only files ending with .exe, .com, .cmd or .bat can be directly executed. Portable file execution must either:

  • use an interpreter, e.g. node file.js instead of ./file.js.
  • use cross-spawn (which is included in execa).

During file execution the extension can be omitted on Windows if it is listed in the PATHEXT environment variable, which defaults to .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC. This won't work on Unix.

The PATH environment variable uses ; instead of : as delimiter on Windows. This can be retrieved with path.delimiter.

When the option detached: false of child_process.spawn() is used, the child process will be terminated when its parent is on Windows, but not on Unix.

When the option detached: true is used instead, a new terminal window will appear on Windows unless the option windowsHide: true is used (requires Node >= 8.8.0).

Finally the option argv0 does not modify process.title on Windows.

Many of those differences can be solved by using execa.

Package binaries

Package binaries (package.json's bin field) are installed in the node_modules/.bin folder by npm install.

On Unix those are symlinks pointing to the executable files. They can be executed directly inside a terminal.

On Windows, each package binary creates instead two files for the same purpose:

  • a Windows batch file ending with .cmd which can be executed directly inside cmd.exe.
  • a Unix shell file with no file extension which can be executed with sh or bash.

Environment variables

The syntax to reference environment variables is $VARIABLE on Unix but %VARIABLE% on Windows. Also if the variable is missing, its value will be '' on Unix but '%VARIABLE%' on Windows.

To pass environment variables to a command, it must be prepended with VARIABLE=value ... on Unix. However on Windows one must use Set VARIABLE=value or setx VARIABLE value as separate statements. cross-env can be used to both reference and pass environment variables on any OS.

To list the current environment variables env must be used on Unix and set on Windows. However process.env will work on any OS.

Environment variables are case insensitive on Windows but not on Unix. path-key can be used to solve this for the PATH environment variable.

Finally most environment variables names are OS-specific:

The project osenv can be used to retrieve OS-specific environment variables names.

Symlinks

Windows (but not Unix) can use junctions. fs.symlink() allows creating these.

Creating regular symlinks on Windows will most likely fail because it requires a "create symlink" permission which by default is off for non-admins. Also some file systems like FAT do not allow symlinks. As a consequence it is more portable to copy files instead of symlinking them.

Neither junctions nor hard links (fs.link()) require permissions on Windows.

File metadata

The blksize and blocks values of fs.stat() are undefined on Windows. On the other hand the birthtime and birthtimeMs values are undefined on Unix.

The O_NOATIME flag of fs.open() only works on Linux.

fs.watch() is not very portable. For example the option recursive does not work on Linux. chokidar can be used instead.

Permissions

Unix uses POSIX permissions but Windows is based on a combination of:

  • file attributes like readonly, hidden and system. winattr and hidefile can be used to manipulate those.
  • ACLs (also called NTFS permissions or just "file permissions").
  • share permissions.

Node.js does not support Windows permissions. fs.chmod(), fs.stat()'s mode, fs.access(), fs.open()'s mode, fs.mkdir()'s options.mode and process.umask() only work on Unix with some minor exceptions:

  • fs.access() F_OK works.
  • fs.access() W_OK checks the readonly file attribute on Windows. This is quite limited as it does not check other file attributes nor ACLs.
  • The readonly file attribute is checked on Windows when the write POSIX permission is missing for any user class (user, group or others).

On the other hand fs.open() works correctly on Windows where flags are being translated to Windows-specific file attributes and permissions.

Another difference on Windows: to execute files their extension must be listed in the environment variable PATHEXT.

Finally fs.lchmod() is only available on Mac.

Users

Unix users are identified with a UID and a GID while Windows users are identified with a SID.

Consequently all methods based on UID or GID fail on Windows:

The privileged user is root on Unix and admin on Windows. Those are triggered with different mechanisms. One can use is-elevated (and the related is-admin and is-root) to check it on any OS.

Time resolution

The resolution of process.hrtime() is hardware-specific and varies between 1 nanosecond and 1 millisecond.

OS identification

The main way to identify the current OS is to use process.platform (or the identical os.platform()).

The os core module offers some finer-grained identification methods but those are rarely needed:

  • os.type() is similar but slighly more precise.
  • os.release() returns the OS version number, e.g. 3.11.0-14-generic (Linux), 18.0.0 (Mac) or 10.0.17763 (Windows).
  • os.arch() (or the identical process.arch) returns the CPU architecture, e.g. arm or x64.
  • os.endianness() returns the CPU endianness, i.e. BE or LE.

Some projects allow retrieving:

Device information

Uptime, memory and CPUs can be retrieved on any OS using os.uptime(), process.uptime(), os.freemem(), os.totalmem(), process.memoryUsage(), os.cpus() and process.cpuUsage().

However:

systeminformation can be used for more device information.

Networking

os.networkInterfaces() and os.hostname() work on any OS.

However on Windows:

  • sockets / named pipes must be prefixed with \\.\pipe\
  • TCP servers cannot listen() on a file descriptor.
  • cluster.schedulingPolicy SCHED_RR is inefficient, so the default value is SCHED_NONE.

Processes

process.pid, process.ppid, process.title, os.getPriority() and os.setPriority() work on any OS.

Other projects can be used to manipulate processes:

Signals

Windows do not use signals like Unix does.

However processes can be terminated using the taskkill command. The taskkill project can be used to do it from Node.js. fkill builds on it to terminate processes on any OS.

Which signals can be used is OS-specific:

Each signal has both an OS-agnostic name and an OS-specific integer constant. process.kill() can use either. It is possible to convert between both using os.constants.signals. However it is more portable to use signal names instead of integer constants.

--diagnostic-report-on-signal does not work on Windows.

Errors

Node errors can be identified with either:

It is possible to convert between both using os.constants.errno and util.getSystemErrorName.

Most available error.code start with E and can be fired on any OS. However few start with W and can only be fired on Windows.

Anti-virus

Some anti-virus software on Windows have been reported to lock directories and make fs.rename() fail. graceful-fs or rimraf solves this by retrying few milliseconds later.

Summary

Further reading

Contributing

Did you find an error or want to add more information? Did you have problems understanding a specific section? Issues and PRs are welcome!

We also invite you to submit links to related projects but only if:

  • it's specific to Node.js
  • it provides cross-platform support that is not available in core Node.js
  • it's maintained

Contributors

Thanks goes to these wonderful people:

ehmicky
ehmicky
💻 🎨 🤔 📖		 thatalextaylor
thatalextaylor
🤔 📖		 Ben Noordhuis
Ben Noordhuis
🤔 📖		 Steve Lee
Steve Lee
🤔 📖 📢		 Michael J. Ryan
Michael J. Ryan
🤔