Knowledge is only real when shared.
January 2022
How Rust based low-level tooling is improving workflows.
JavaScript and TypeScript continue to be the most popular programming languages. Originating from the browser JavaScript is optimized to work well for Web Development. Because of this JavaScript was never designed to be performant at basic low-level tasks. Other things like a reasonable learning-curve for newcomers and portability of the code were more important. However, browser based applications are increasingly becoming more complex. Although, not visible to the user the build chain is an important part of the daily Web Development experience. There are bundlers like webpack, transpilers like babel, linters like eslint, formatters like prettier and last but not least type checkers like TypeScript.
All these tools have one thing in common: They are written and distributed in JavaScript.
As mentioned JavaScript was never intended for these low-level tasks that perform the same simple operations again and again. Still, these tools work increadibly well and have managed to keep up with the fast paced innovation in the Web Development space. Tools like esbuild have shown that there is still a lot of room for performance improvements among JavaScript based tools using only JavaScript. Replacing these tools with Rust based binaries however puts performance in a whole new ballgame. This article will discuss these old and new approaches that use lower level languages in order to gain an edge.
JavaScript isn't performant when compared to low-level system languages like C or C++. For simple tasks that are run over-and-over again like compilation it makes sense to use a faster runtime. So in order to harness better performance some node_modules include binary code written in languages other than JavaScript. An example is node-sass which compiles SASS styles to CSS styles parsable by the browser. The source code for node-sass is written in C++ and based on the now deprecated LibSass. When installed on a Mac the binary code is found under /node_modules/node-sass/vendor/darwin-x64-102.node. This .node file includes binary code written in C++ and can be imported and used directly from JavaScript code running with node. These files leverage the Node-API and are called native addons.
Unlike JavaScript or Java Bytecode the binary code is different depending on the system it runs on. npm has no specific mechanism to distribute system specific code. Therefore, various workarounds are used to allow users to download or generate the binaries they need. As for node-sass the package includes the source code written in C++ and the native addon is compiled on the target system itself. This compilation is triggered in the postinstall script of the package using node-gyp. When installing dependencies with the flag --ignore-scripts the compilation will not happen. Obviously, this compilation slows down the installation and as most developers are aware requires a rebuild when the major node version changes. As for SASS the developers now recommend to use the sass package which has the same functionality but includes the compiler as JavaScript built from the source code in Dart. The compiler is somewhat slower but the advantages outweigh the drawbacks as SASS compilation is usually only a small step in a larger build chain.
Just like JavaScript WebAssembly is a platform independent language. It's intended to allow high performance applications to be run in the browser. Since it's an assembly language it's usually not written directly but created as the compiled result of some C/C++ or Rust. In the Web Development timeline WebAssembly has been supported in node since October 2017 with version 8. Browser support is very good at about 95% comparable to CSS Grid. Figma a browser based design tool runs on WebAssembly while the source is written in C++.
Rust has recently gained attraction in the Web Development community. It probably won't replace JavaScript used by regular developers anytime soon, but it's being used to improve the performance and interoperability of various background development tools. An example is SWC which is a compiler for JavaScript and TypeScript similar to babel and esbuild. SWC is already enabled by default in Next.js and has greatly increased its performance. One of the goals is to fully replace webpack some time in the future so that it could also be integrated into tools like create-react-app. @swc/jest can already be used productively as a replacement for babel-jest and ts-jest used as transformers with jest. Bundling similar to webpack is still under construction and will be called swcpack.
Rust is usually compiled to a native Node-API addon. This way it can be used to interoperate with regular JavaScript code while still having better performance. The compiled native addon is a single file ending with .node. Unlike with node-sass described above the binaries are downloaded precompiled for the system of the user. Below, the workaround using optionalDependencies used by napi-rs is explained.
In a recent post about Generating Social Media Share Images with Vercel I tried several ways to generate dynamic images. Rendering the images using a headless browser like puppeteer works fine but requires almost all of the available disk space provided to Vercel Functions and obviously has a lot of overhead. Therefore, I've explored alternatives like generating the images with canvas (also called node-canvas) and rendering and SVG to a PNG with sharp. Both these node plugins work well locally but are hard to get to work on Vercel Functions as they rely on system dependencies not available there. Unable to configure these tools to work properly with Vercel Functions I stumbled upon some brand new Rust-based tooling to achieve the same result without requiring system dependencies missing in the Vercel AWS environment. The previous post details how these plugins are used, while here we'll analyse how these packages work and leverage Rust.
@napi-rs/canvasBrowsers already include a canvas renderer. However, in the backend with node one usually wants to avoid firing up a full browser just to render some small image dynamically. Until recently, canvas was the go-to option to render canvas with node. Recently published was @napi-rs/canvas which like canvas is a canvas renderer for node. It doesn't rely on any system dependencies and instead uses the Skia 2D graphics engine. The whole package is about 20 MB in size for the macOS version. A lot smaller than canvas which after installation requires 45 MB spread out over 61 dependencies requiring a postinstall build using node-gyp.
The package doesn't offer any exciting features as the goal is simply to mirror the browser canvas API and offer increased performance and ease-of-use compared to previous renderers. Interesting is the workaround used to distribute the binaries which vary depending on the host system. Diving into the package.json of @napi-rs/canvas we can see that it includes a list of optionalDependencies one for each supported architecture (different CPU architectures and operating systems require differently compiled binaries).
optionalDependencies were introduced fairly recently to npm. When an installed package lists any of those npm will try to install them. If the installation fails nothing will happen and it's simply skipped. As can be seen below in the @napi-rs/canvas-darwin-x64 package used for Mac it lists an os and the applicable cpu. In this case the package will successfully install on any Intel-x64 based Macs while all the others will fail and are skipped.
This approach will work even with the --ignore-scripts flag, but fail with the --no-optional flag. Postinstall scripts are disabled on certain corporate environments due to security precautions. Unlike older approaches which used to download the binaries in the postinstall script from GitHub this approach will only contact the npm registry. This can be useful when a firewall in front of a continuous integration environment is blocking other parts of the internet.
@resvg/resvg-jsThis plugin is wrapping resvg for use with node. resvg itself is written in Rust and together with napi-rs this plugin is making it accessible to node. Just like the canvas plugin it can be installed without any postinstall scripts using the --ignore-scripts flag. The whole package clocks in at below 3 MB for the macOS version.