craigtaub.dev

To subscribe to this blog, you have several options.

The RSS Feed

The RSS feed is located at:

Historically most browsers had built-in support for feeds, but unfortunately most do not anymore. However RSS feeds are still extremely well supported by content producers. If you're reading blogs or news sites today the chance is incredibly high that an RSS is offered.

I recommend Thunderbird as both an email and news reader. Let me know if you have any other suggestsion and I can list them here.

Email list

I set up a mailing list if you prefer email.

Other ways

I post blog updates on Twitter.

I post JavaScript videos on YouTube.

Thank you!

Thank you for being interested. If you have any feedback I would love to hear it!

NodeJS vs Golang - an AWS costing exercise

Tue, 20 Sep 2022 00:00:00 GMT

When you look at the backend landscape, currently there are many popular languages - I would say Golang and Python are the most popular, with NodeJS/TypeScript towards the top of the list especially when the role requires some full-stack work.

I am not going to argue against Golang today - it is a great language with many years of hard work to build out its performance, syntax, tooling, community and many more assets.

The part I want to look at today is whether there is much value in migrating from TypeScript to Golang for purely cost reasons. I want to answer what kind of savings would it offer to a company who uses many of AWS products??

We will use an example company so we have something to compare against.

Introducing CompanyX

They have a website, native mobile apps and TV app, all powered by an API run entirely on AWS. The surve to tens of thousands of users which produces millions of requests. A summary of the tech stack includes

Fargate (via ECS and EKS) - for the API
EC2 - for the website
Serverless Lambdas - for processing user input (which is collected very frequently) into features for different platforms as well as for storage purposes

They also use most other AWS products such as

RDS, S3, Cloudwatch, Cloudfront, API Gateway, ElastiCache, Kinesis and more.

CompanyX - January AWS bill

So its February and we check out the bill for the first month of the Year, the costings look like this, in order of highest price.

RDS - £3k
S3 - £2.5k
Cloudwatch - £2k
Cloudfront - £1.5k
EC2 - £2k
Fargate - £1k
API Gateway - £1k
ElastiCache - £700
Lambda - £150
Kinesis - £70

The total bill came to £13,920.

Here you can see the % of products of the entire bill.

What can we ignore?

So many of those services it would not matter if we were using Golang, those services are listed below

RDS
S3
Cloudwatch
Cloudfront
API Gateway
ElastiCache
Kinesis

That leaves just below:

EC2
Fargate
Lambda

That means the maximum saving from the bill is £3,150, that is 22% from the overall bill. That is pretty good it's just under a quarter. However we still need those services, they will just be run using Golang rather than NodeJS.

So now let's actually dig into what changes with those if we swap to Golang.

What is the savings?

Here is a link to the pricing pages to review my comments below yourselves:

EC2 on-demand here.
Fargate here.
Lambda here.

EC2

We are paying per "on-demand instance hour", which means we pay for the compute capacity we used from the time the instance launched until it was terminated or stopped.

Let us assume Golang is twice as fast as NodeJS - many different benchmarks do support a similar theory.

Of the £2k bill £1.5k is the "on-demand per instance hour" (the rest is related to NatGateway), so that is brought down to £750. So the total EC2 bill is £1,250. Down from £2k.

Fargate

Fargate pricing is very similar to EC2 on-demand model, in that we pay for the resources used from time the pod starts until it's terminated. It applies for memory, cpu and storage all per hour.

Of the £1000 bill £800 is for the hourly vCPU charges.

Similar to above we assume Golang to be twice as fast, we reduce £800 to £400 bringing the overall Fargate bill to £600.

Lambda

The last application type is a serverless lambda. With lambdas we typically pay for:

Invoke numbers
Duration

In general it seems duration costs are higher than invoke numbers.

CompanyX currently only pays £160, £140 of which is for compute time, the other £20 is for request count.

We will need the same number of invoke numbers, as the same number of clients will be requesting data, but the latency is lower. So by speeding up our compute time we can possibly halve the cost to £70. Bringing the total for Lambda to £90 a month.

The savings

Above produces to following savings:

EC2 - £750
Fargate - £400
Lambda - £70

That is a total saving of £1,220. That is 8% from our overall total bill of £13,920.

After applying the saving, here you can see the % of products of the entire bill.

Overall

So that's it - for CompanyX they will save £1,220 a month - which will bring their yearly bill down from £167,040 to £152,400. So £15,000 they can spend on something else.

From my experience to most companies, including startups, £15,000 is pretty small.

Considering the amount of developer time and effort which will have to be invested in order to learn and implement Golang, it does not seem worth it purely for cost reasons. However as I said at the start, if the reasons are more than cost than it definitely can reap long-term benefits.

Please do remember this is just a numbers exercise, but I have tried to base it on a real scenario.

Thanks for reading my article - I hope you found it useful.

Under-the-hood of GraphQL DataLoader

Fri, 11 Mar 2022 00:00:00 GMT

In recent years GraphQL has really taken off as a pattern/library/type system. It offers much which REST does not and its standardization and flexibility has really helped in its adoption. I have an article focused on digging deeper into GraphQL here, so today we will focus on another tool in the GraphQL ecosystem - one that is very important and interesting in how it assists applications - that tool is the DataLoader.

This is part of my "under-the-hood of" series:

The article today will be broken down into 2 parts:

Overview
- Batching
- Caching
Building our own GraphQL DataLoader

1: Overview

The NodeJS repository for GraphQL's DataLoader is found at https://github.com/graphql/dataloader, however it can be found in many different language implementations. It can be used as part of your applications data fetching layer, and its basic job is to reduce requests to backends by 2 means.

Batching
Caching

It utilizes different logic and functionality to perform the above efficiently.

The first question is what does GraphQL have to do with this?

It pairs nicely with GraphQL as GraphQL has fields which are designed to be stand-alone functions (resolvers) and it is very easy to share a class instance via the context. The class instance would be our instance of DataLoader.

The natural pairing of DataLoader and GraphQL has produced high success rates - some examples have seen 13 database queries reduced down to 3 or 4.

Batching

Batching is the primary feature of DataLoader, you must pass the library a "batch function" to detail how to process the batch. Within a single tick of the event loop DataLoader gathers all individual loads, then calls the "batch loading function" with all requested keys.

VS your ORM

Its important to note DataLoader does not optimize the queries itself - you can look to an ORM for help there. For example Objection-JS has logic to avoid "N+1 selects" by utilizing "where in" queries.

Batch function

This is the function given to the library when you create a new instance

const ourInstance = new DataLoader(keys => myBatchFunction(keys))

The basic idea is that you check your cache first for a given key, if it exists return that value, else hit the data-source e.g database.

It passes in an array of keys, but there is a constraint that:

the returned array size must match the keys coming in
the returned array indexes must match the keys coming in

There is a sound reason for that limitation and it's related to the implementation - it is covered in part 2 below.

Its worth highlighting that keys are suited to table column ID's, so it stands to reason that having a database table for each entity in your data model would fit this mechanism well.

Caching

DataLoader uses a simple in-memory memoization cache. You can swap the memory store for something else e.g. SQL-lite.

Caching pairs really well with the Batching, because the batch can ensure the requested data has already been pulled from the database, the cache can be utilized to retrieve from there. We will go over this more in the next section

2. Building our own GraphQL Dataloader

In this section we will focus on the and save for another time. Hopefully it will provide enough context on how caching is utilized.

Within a single tick of the event loop DataLoader gathers all individual loads, then calls the "batch loading function" with all requested keys.

You might be wondering how it does this - so let's look at the most simple example.

const run = async () => {
  const keys = [];
  const batchFunction = (keys) => {
    // bad point 1 - called with [1,2], [1,2]
    console.log("keys: ", keys);
  };

  const load = async (id) => {
    keys.push(id);
    process.nextTick(() => {
      batchFunction(keys);
    });
    // bad point 2 - promise not connected to batch function
    return Promise.resolve(`id: ${id}`);
  };

  const a = await load(1);
  const b = await load(2);
  console.log("a", a); // id: 1
  console.log("b", b); // id: 2
};

run();

This calls our batch function twice - both times with both keys. The order of events is this:

call load asynchronously with id 1
call load asynchronously with id 2
async load(1)
- store key 1 in the global keys array
- schedule a node process to, on the next tick, run our batchFunction with those keys
- Return a resolved promise with the id.
async load(2)
- store key 2 in the global keys array
- schedule a node process to, on the next tick, run our batchFunction with those keys
- Return a resolved promise with the id.
The first scheduled process runs, with both ids 1 and 2 in the keys array
The second scheduled process runs, with both ids 1 and 2 in the keys array.

So here you can see the basic mechanism of how batching works.

Good 👍🏻

Runs our batch function with both keys - this will mean we can cache the database response, and next time those keys are included only utilize cache

Bad 👎🏻

Unnecessarily calling the batch function with the same keys, unnecessarily running the code even if it is hitting the cache.
load does not return anything useful, its a completely isolated resolved promise.

The below example looks to improve on that.

let resolvedPromise;
let batchFunction;
let batch;

const dispatchBatch = () => {
  batch.hasDispatched = true;
  const batchPromise = batchFunction(batch.keys);

  batchPromise.then((values) => {
    for (var i = 0; i < batch.callbacks.length; i++) {
      var value = values[i];
      // resolve promise callback
      batch.callbacks[i].resolve(value);
    }
  });
};

const batchScheduleFn = (cb) => {
  // add more logic if scheduling
  process.nextTick(cb);
};

const getCurrentBatch = () => {
  // !hasDispatched only needed if using a 2nd tick - this example isnt
  if (batch && !batch.hasDispatched) {
    return batch;
  }
  const newBatch = { hasDispatched: false, keys: [], callbacks: [] };
  batch = newBatch;
  batchScheduleFn(() => {
    dispatchBatch();
  });
  return newBatch;
};

const load = async (id) => {
  const localBatch = getCurrentBatch();
  localBatch.keys.push(id);
  // add promise callback to batch
  const promise = new Promise((resolve, reject) => {
    localBatch.callbacks.push({ resolve, reject });
  });

  return promise;
};

async function threadTwo() {
  const user = await load(2);
  console.log("threadTwo user", user.id);
}

async function threadOne() {
  const user = await load(1);
  console.log("threadOne user", user.id);
}

const run = async () => {
  // make async
  batchFunction = async (keys) => {
    console.log("keys:", keys);
    // keys: [ 1, 2 ]
    return keys.map((key) => ({ id: key }));
  };

  threadOne();
  threadTwo();
};

run();

It introduces batches which can be sheduled - this is exactly how DataLoader manages it (here).

The order of events is this:

call threadOne - call load async with id 1
call threadTwo - call load async with id 2
async load(1)
- get the current batch
  - batch is currently undefined so a newBatch is created
  - we schedule a dispatch by calling dispatchBatch() inside our scheduler batchScheduleFn()
    - this adds dispatchBatch callback to the nextTick.
  - lastly we return the batch
- we add the id to the keys array on the current batch
- we create a new promise, add the reject and resolve to our current batch callbacks (so the list index is important)
- lastly we return the new promose
async load(2)
- get current batch
  - batch currently exists and has not been dispatched so we return that
- as above we add the id and reject/resolve to the current batch
- as well as return the promise
process.nextTick
- the tick runs dispatchBatch
- call our batchFunction with the current batches keys
  - batchFunction returns a promise
- when that promise resolves (.then), it returns an array of our keys
- we iterate over our batch callbacks - for each callback
  - we find the associated batchFunction key value this is why the batch function response indexes are so important
  - resolve the callback with that value
await load(1) resolves
- returning object {id}
await load(2) resolves
- returning object {id}

This calls the batch function once with both keys, it returns correct values from batch function - dealing with both "bad" points from the first example.

Thanks so much for reading, I learnt a huge amount about DataLoader and GraphQL from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Under-the-hood of Docker

Fri, 28 Jan 2022 00:00:00 GMT

Docker is defined as

A set of platform as a service products that use OS-level virtualization to delivery software in packages called containers

In essence Docker allows us to turn applications into binaries which can be stored externally and then be pulled to run or deployed anywhere. It was initially released in 2013 (8 years ago from now) and is predominantly written in Go. The two main public registries of docker images are Docker Hub and Docker Cloud. The former being the default registry Docker checks.

This is part of my "under-the-hood of" series:

The article today will be broken down into:

Overview
Building our own Docker

1: Overview

There are several components to Docker, lets start with:

Client
Registry
Host
- Daemon
- Images
- Containers
- Storage/volumes
- Networking

Client

The client is the means to interact with a Docker Host via the Daemon (see below). The client usually consists of a CLI (to run the commands directly e.g. docker pull x) or a piece of software which runs these connads for you (e.g. Docker Desktop).

The client can actually connect to more than 1 Docker daemon at a time.

Registry

This is the external database which stores the images. See my "under the hood of NPM" for more details on a registry Database.

Host

This is the environment to execute and run containers on, its not the same as the local machine. To get access you have to go into a container and then hit localhost:<port>.

The host includes several objects such as:

Daemon

The Daemon is a background process whose job is to listen for Docker API requests and respond to them. It manages objects like - images, containers, networks and volumes.

The binary can be run under

dockerd

Example - build command

CLI tells Deamon
Damon opens Dockerfile runnning 1 instruction at a time - It commits the result of each instruction to a new image
Finally outputs ID of image - Uses the build cache. Print message in CLI with each step

Build cache

For each instruction Daemon checks cache to see if exists already
Pairs "parent image" + "instruction" (key) to find a hit
Compares the instruction against all child images derived from base/parent images

Images

There is alot to images so I have tried to cover the main points.

Images are read-only templates, they are immutable snapshots.

Images come with a manifest.json file, which holds details of the image tags and signatures. The image directory contains the image layer and manifest, only 1 layer per image. The base has no parent, the children have the ID of the parent. The bottom-most image is called the base image. The unique ID is a sha256 hash

Image layers

Layers are intermediate images, an image has 1 layer, each layer is an instruction. The instruction is stored inside the images JSON config, along with the folder details (i.e. lower, merged, upper, work). This system promotes smaller images as each is stacked on top of the previous.

Each layer is stored in its own directory, inside the Docker host local storage area. The directory contains (unique) image contents, but the directory name IS NOT the layer ID.

> ls /var/lib/docker/overlay2 # list image layers
> 16802227a96c24dcbeab5b37821e2
> ls 16802227a96c24dcbeab5b37821e2 # image layer contents
> diff link lower merged work

Directories

link - shortened layer id
diff - the layers contents from the root (the folders and files)
lower - Points to parent/previous layer (higher layers have this).
merged - Unified contents of higher layer and itself (higher layers have this)
work - used internally by OverlayFS storage driver

You can use the below commands to verify the cryptographic IDs

> docker image ls
> docker history

Use the below to view the layer ID and hte instruction which created it (note Size can be 0 if just adding to metadata)

> docker image history <image>

Containers

Containers are the virtualized runtime environment, they run the image.

Container layer

Each new container adds a new writeable layer on top of the underlying layers. All changes made to the running container (e.g. alter file) are written to this thin writeable layer. When the container is deleted so is this writeable layer, the underlying image is untouched. Multiple containers can share the same image but have their own data state.

If a volume is used the volume becomes this writeable layer.

Storage/volumes

I personally find this the trickiest one to understand from the Docker subject.

Storage driver

This controls how the "image layer (stack)" and "container (r/w) layer" are managed (the contents), unrelated to volumes.

Generally files are not persisted after a container is deleted, but the driver gives the ability to write data to the "containers (writeable) layer". The driver handles details of how the image layers interact with each other. They have different advantages and disadvantages based on the situation.

All drivers use the same stackable image layers and most use the CoW (copy-on-write) strategy (see below).

The overlay2 docker storage driver uses the OverlayFS Linux kernel driver (see below).

The storage driver uses the backing filesystem driver (in the kernel) to alter files (the filesystem of the host on which Docker is installed). Some storage drivers only work with specific backing file-systems e.g. extfs.

Copy-on-Write strategy

It is a very efficient storage technique. Essentially if the image is duplicated but not modified, don't create a new image. So you can share images until one is changed, but modifications create a new image.

OverlayFS

OverlayFS layers 2 directories on a single Linux host and presents them as a single directory. Is is a very modern union filesystem (i.e. container vs image layer) and it has efficient inode utilization.

Volumes

Volumes are used to persist data created in containers, for write-heavy applications. They are preferable to bind mounts, as volumes are managed by docker whereas bind mounts are via the OS.

To use a volume point local contents to the area used by the container and using the below format.

volume:directory

Volumes are bind-mounted into existing (or created) paths within the filesystem image, outside the image layer. They are preserved in their source location and become the "container layer". They can share data by sharing a single volume across multiple containers.

They are created initially and then re-used, managed via the Docker API. You can create and manage them outside the scope of a container. You can name the volume so it has a source from outside the container, or have it anonymous so when the container is removed the Daemon will remove it.

They live on the host filesystem (usually found under the docker data directory e.g. /var/lib/docker/volumes but depends on FS).

Some useful commands:

> docker volume create my-vol
> docker volume inspect my-vol
> docker volume rm my-vol
>  -v ./packages/website:/usr/src/app

Networking

The docker Deamon acts as a DHCP service for containers i.e. assigning IPS etc. The container uses DNS settings of the host (defined in /etc/resolv.conf).

The default network settings specify the container is part of the docker network stack and creates a bridge (a gateway and IP subnet). All containers that belong to this network can talk to each other via IP addressing.

My favourite Docker architecture image is this one - it shows an overview of the components but also the flow of operations (see the build, pull, run).

2: Building our own Docker

For this proof-of-concept we will be including the below aspects:

Host
- Daemon
- Existing images
CLI
- build
- run
Running the app

My plan was to use a copy-on-write mechanism and custom JS storage driver, but I ran out of time and decided it was more important to focus on the image creation and running aspect.

So our app is going to:

Read a local Dockerfile committing instructions into a new image layer, based off a parent layer, creating a new image.
Run the new image - use a tmp directory as a writeable container layer over the specified image. Then start our container in a virtualized runtime environment and execute a command.

Ignored aspects:

Daemon acting as DHCP for containers assigning ips
Host to run Daemon (will be on local machine)
Deleting container layer when stopped
Container layer using stackable "copy-on-write" strategy
Allow mounting volume
Container not using DNS settings of the host (no host)

A final thing which was omitted is that in real Docker each instruction creates a new image layer, in our PoC we have run all the instructions into a single image layer, hopefully this does simplify things.

THE CODE !!

Daemon

The Deamon is a simple express server executing commands in a child process.

import express from "express";
import path from "path";
import { promisify } from "util";
import run from "./commands/run.js";
import build from "./commands/build.js";

const app = express();
const port = 3000;

app.get("/:command/:args", async (req, res) => {
  console.log("Command: ", req.params.command);
  console.log("Args: ", req.params.args);

  switch (req.params.command) {
    case "run":
      await run(req.params.args);
      break;
    case "build":
      await build(req.params.args);
      break;
  }
  res.send("success");
});

app.listen(port, () => {
  console.log(`Example app listening at http://localhost:${port}`);
});

Existing images

Before we look at the commands, I have created a couple of fake image layers inside images.

To summarize what is found in the above link:

images/lowest-layer contains:

/diff
- contains folders /etc, /home, /lib, /mount, /opt, /user, /var
link
- contents LOWEST-ID
config.json
- JSON object with empty values
manifest.json

images/middle-layer contains:

/diff
- contains folders /media + /usr/src/app
- Note that /usr/src/app now has a small JS application with index.prod.js and index.dev.js
/merged
- contains folders /etc, /home, /lib, /mount, /opt, /user, /var, /media
- /media is new
link
- contents MIDDLE-ID
lower
- contents l/LOWEST-ID
config.json
- JSON object with values for Env, Cmd and WorkingDir
- Cmd runs index.dev.js
manifest.json

CLI

const daemon = 'http://localhost:3000';
const supported = ['build', 'run'];

async function run() {
  const command = process.argv[2];
  const details = process.argv[3];
  if (!supported.includes(command)) {
    console.log("Not supported");
    return;
  }
  const response = await fetch(`${daemon}/${command}/${details}`);
  if (response.status) {
    console.log("SUCCESS");
    return;
  }
  console.log("Failure :(");
}

run();

It is very simple. It takes an argument and some details and HTTP GET's the Daemon with those details. The HTTP GET mechanism is a simplification of the real CLI->Daemon communication, but it works easily for a PoC.

Our cli package script will just run node cli.js.

Example commands are:

  > npm run cli <command> <argument>

Commands

Build

We start with some utils, they return paths or update config values.

const utils = {
  getFullPath: () => path.resolve(path.dirname("")),
  grabConfig: async () => {
    await delay(1000);
    const fullPath = utils.getFullPath();
    const fullConfig = await import(`${fullPath}/tmp/config.json`);
    return fullConfig.default[0];
  },
  updateConfig: async (config) => {
    const fullPath = utils.getFullPath();
    return asyncWriteFile(
      `${fullPath}/tmp/config.json`,
      JSON.stringify([config])
    );
  },
};

The main build fuction.

export default async function (buildImage) {
  // 1
  if (buildImage === ".") {
    // Default local image
    // 2
    const dockerFilePath = path.resolve(path.dirname(""), "./OurDockerfile");
    const file = await asyncReadFile(dockerFilePath, {
      encoding: "utf-8",
    });
    // 3
    // good for small files, NOT big ones
    const linesArray = file.split(/\r?\n/);
    await linesArray.map(async (line) => await commitLine(line));
    // required for above OS ops to finish
    await delay(1000);
    // create new image
    const layerName = "highest-layer";

    const fullPath = utils.getFullPath();
    // 4
    // update link (HIGHEST-LAYER) + lower (MIDDLE-ID)
    const link = await asyncReadFile(`${fullPath}/tmp/link`, {
      encoding: "utf-8",
    });
    await asyncWriteFile(`${fullPath}/tmp/link`, layerName.toUpperCase());
    await asyncWriteFile(`${fullPath}/tmp/lower`, link);

    console.log(`SUCCESS - Created layer: ${layerName}`);
    await delay(1000);
    // 5
    // move tmp to new image
    await asyncNcp(`${fullPath}/tmp`, `images/${layerName}`);
    // remove tmp
    await asyncRimraf(`${fullPath}/tmp/`, {});
  }
}

It works like this:

Checks for build image, currently only supports local file i.e. .
Grab the OurDockerfile contents.
Split the file by its lines and run commitLine to process each line
Update the link and lower of the new image, the references to itself and parent image.
Move the new image layer from /tmp into /images and remove /tmp

The commit maps job is to action the comamnds inside the Dockerfile. In real Docker it would create a new layer for each instruction. There is also a commitLine which runs against the map. It currently supports some of the most popular commands in Docker:

from
env
workdir
copy
cmd

const commitMap = {
  from: async (layer) => {
    // move to tmp for processing
    const fullPath = utils.getFullPath();
    await asyncNcp(`${fullPath}/images/${layer}`, `tmp`);
    // remove diff as specific to layer
    await asyncRimraf(`${fullPath}/tmp/diff`, {});
  },
  env: async (values) => {
    const config = await utils.grabConfig();
    if (config.Config.Env) {
      config.Config.Env.push(...values); // merge incoming array into config one
    } else {
      config.Config.Env = values;
    }
    await utils.updateConfig(config);
  },
  workdir: async ([value]) => {
    const config = await utils.grabConfig();
    config.Config.WorkingDir = value; // a string
    await utils.updateConfig(config);
  },
  copy: async (values) => {
    const fullPath = utils.getFullPath();
    const cpyLoc = values.pop();
    // required for diff deletion to finish
    await delay(1000);
    values.map(async (file) => {
      // create folder recursively
      await asyncMkdir(`${fullPath}/tmp/diff${cpyLoc}/`, { recursive: true });
      // copy files
      await asyncCopyFile(file, `${fullPath}/tmp/diff${cpyLoc}/${file}`);
    });
  },
  cmd: async (values) => {
    const config = await utils.grabConfig();
    config.Config.Cmd = values;
    await utils.updateConfig(config);
  },
};

async function commitLine(line) {
  const args = line.split(" ");
  // 1
  const command = args[0];
  if (!command) return; // empty line or something
  args.shift();
  // 2
  // call command function
  if (!commitMap[command.toLowerCase()]) return; // invalid instruction
  await commitMap[command.toLowerCase()](args);
}

The break down the flow once commitLine(line) is called

Grab the command
Make sure it exists in the commitMap then execute it with arguments

If we take the following OurDockerfile

FROM middle-layer
ENV NODE_VERSION=13.0.0
WORKDIR /usr/src/app
COPY fileA.txt fileB.txt /usr/src/app
CMD npm run start.prod

Note that fileA.txt and fileB.txt both exist at the top level of the test repository and have simple content inside.

We can step through how each instruction gets processed:

FROM

gets the full path
moves the image (e.g. middle-layer) located in images into tmp
removes the tmp/diff folder as it is specific to the layer

I have 2 example images already build under images

middle-layer
lowest-layer

ENV

Grab the config
if there are values set under Env add to them, else create the section and add this value
Update the config

WORKDIR

Grab the config
Set the WorkingDir to be the new value
update the config

COPY

Grab the copy location
Map over each file to copy and move to /tmp/diff/<copy location>

CMD

Grab the config
Set the Cmd to be the new value, i.e. run index.prod.js
update the config

Notice the similarities between ENV, WORKDIR and CMD. They are mostly updating the image layers config.json file, so when it is run it has the correct values.

In real Docker each of the instructions would be committed into a new image layer and at the end the image would be created, however we have merged those 2 concepts for simplicity sake so that all the instructions create a single image layer.

Run

Now we will look at what it takes to build a new image.

export default async function (image) {
  // 1. copy image contents, exclude configs
  const fullImgPath = path.resolve(path.dirname(""), "./images");
  await asyncNcp(`${fullImgPath}/${image}/merged`, `tmp`);
  console.log("copied");

  // 2. process config
  const fullConfig = await import(`../../images/${image}/config.json`);
  const config = fullConfig.default[0].Config;
  const splitCommand = config.Cmd;
  // env is key:value pairs
  const environment = config.Env.reduce((acc, curr) => {
    const [key, value] = curr.split("=");
    acc[key] = value;
    return acc;
  }, {});
  const workingDir = config.WorkingDir;

  // 3. run command in child
  const startCmd = splitCommand[0];
  splitCommand.shift();
  const childProcess = spawn(startCmd, splitCommand, {
    cwd: `tmp/${workingDir}`,
    env: environment,
  });
  // 4. outputs
  childProcess.stdout.on("data", (data) => {
    console.log(`stdout: ${data}`);
  });
  childProcess.stderr.on("data", (data) => {
    console.error(`stderr: ${data}`);
  });
  childProcess.on("error", (error) => {
    console.log(`child process error ${error}`);
  });
  childProcess.on("close", (code) => {
    console.log(`child process exited with code ${code}`);
  });
  // remove ref might close open conn, but not sure it will considering above
  childProcess.unref();
}

Lets review

We move the images merged contents into tmp for processing
We process the config.json grabbing the values we need such as Cmd, WorkingDir and Env for the environment variables.
Spawn a child process and run the command using the working directory given and the environment variables
Create listeners on events such as stdout and error. Logging the output.

Running the app

To test we will perform the following

Run middle-layer -> runs tiny js app which prints run DEV app
Build a new highest-layer images -> creates images/highest-layer
Run highest-layer -> the same tiny js app except runs run PROD app

Lets go 💪

In terminal A we run below starting the Daemon

> npm run daemon

In terminal B we run our other commands.

Run middle-layer image

> npm run cli run middle-layer

Built with command npm run start.dev
Outputs run DEV app. Check

Build highest-layer image

> npm run cli build .

Run highest-layer image

> npm run cli run highest-layer

Built with command npm run start.prod
Outputs run PROD app

So thats it 🙌, our mini Docker works.

Thanks so much for reading, I learnt a huge amount about Docker from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Under-the-hood of NPM

Sun, 30 May 2021 00:00:00 GMT

NPM is the official package manager for the NodeJS ecosystem. Since the first release of NodeJS it has come with NPM built-in. The initial release date for NPM was 12 January 2010 and since then has grown into the Worlds largest software registry.

I would expect most engineers in the JavaScript ecosystem to know all about either NPM or Yarn, as they are critical to most local, CI and CD flows. However in the overview section I am going to cover the core functionality, as this is what we will focus on building later.

This is part of my "under-the-hood of" series:

A video for this talk can be found here. Part of my "under-the-hood of" video series here.

The article today will be broken down into:

Overview
Module vs Package
Building our own NodeJS package manager

1: Overview

There are 3 parts to NPM

CLI tool

The CLI is open-source code available on Github. It is currently on version 7 and has had over 700 contributors. The CLI commands are NodeJS scripts and require the format npm <command>.

You can point the CLI to any registry e.g. npm adduser --registry http://localhost:4873

The most popular commands are:

`npm init`

Setup a new package
Creates a package.json (among other things)

`npm install`

Code located in install.js.
Uses npm-registry-fetch https://github.com/npm/npm-registry-fetch
Generates node_modules
Downloads dependencies defined in package.json (see "Module resolution" below)
Generates lockfile (see "Lockfile" below)
Utilises a cache (see "Cache" below)

Example output below:

Lockfile

Generates package-lock.json - describes dependency tree that was installed. For deterministic installs (subsequent installs). If it exists, the install is driven by that. The lockfile stores an 'integrity' hash for each dep. Ths hash (checksum) is of the package tarball uploaded to registry. Can be SHA-1 (older NPM) or SHA-512 (newer NPM). Its akin to a Subresource Integrity used in HTML/browsers.

Module resolution

NPM installs per package and sequentially i.e. moves on once package installed. This means it can be quite slow to finish.
Currently installs all of the nested dependencies as efficiently (or flat) as possible. If a version is the first for a dependency it is top-level, if it is not the first version then it is stored with the parent who requires it.
The old package resolution (pre npm v5) was done on disk by NodeJS (node_modules), this was much slower and is not used anymore.
Example resolution below

Cache

Stores http request/response data and other package-related data
Uses pacote. Library responsible for package fetching
All data passed through the cache is verified for integrity on insertion and extraction
Cache corruption will trigger a refetch, so clearing cache should only be necessary for reclaiming disk space

`npm version`

Bump local version, update package.json and package-lock.json
Create version commit and tag for git

`npm pack`

Create tarball (a zip) of the package (package.tar.gz)
Example screenshot below:

`npm publish`

Code located in publish.js
Runs npm pack as part of it
Sends tarball to registry
Publishing a package consists of at least 1 step (http PUT) with a metadata payload and tarball
See in the logs printed "npm http fetch PUT 200"

Registry

Large public database of JavaScript packages and the meta-information surrounding it. Versioned like git.

Uses Apache's NoSQL database Couch DB to manage publicly available data (it offers good storage performance and replication). It has database for "users" and one for "registry", the latter holds the packages.

Tarballs

It uses Couch DB's attachments for package tarballs. It is incredlby easy to upload or download an attachment from Couch, simpler than most other NoSQL.

API

Couch naturally exposes HTTP endpoints which means it has an API built into it by default. For auth there is a /_session endpoint (to create a new cookie-based session) or it accepts an Authentication header for basic auth, both natively. Using design document Rewrites (see below section) you can build redirects to different parts of the database

Querying

Couch DB is very dynamic in that it allows you to create a kind of schema called a "design document" for a given section. This can include javascript functions (yes it can store and run JS code internally) which execute at given events e.g. Updates is list of functions which run when a document is updated. It also allows Views which are function that take document data and produce searchable lists of information based on the document's contents. There are other types of dynamic mechanisms, more details here.

NPM links

The Couch DB tables and API for registry details are here (a repository for Registry documentation). Couch DB views used by the Registry are found in npm-registry-couchapp. An example Docker image for their DB setup is npm-docker-couchdb. It details how to update a local NPM client to use a local CouchDB registry (here), example below.

npm config set registry=http://localhost:5984/registry/_design/app/_rewrite

Some apps use the popular lightweight private proxy app verdaccio, for many reasons one of which is independent caching incase NPM is down.

Website

Found at https://www.npmjs.com/, a domain first registered on 19th March 2010. It is built using Webpack, React and Lodash. Its assets are served via CDN CloudFlare. Any payments are processed via Stripe.

Connects and reads data from the registry Couch DB instance.

Packages are located at https://www.npmjs.com/package/<package name>.

The README markdown file is loaded as the landing page content with markdown rendered as HTML.

NPM also shows a load of other information such as the weekly download number, latest version, unpacked size and github location.

2. Module vs Package

A module is a file or directory that can be loaded by Nodejs, located inside node_modules.

CLI packages are not modules, they cannot be loaded by NodeJS until unzipped into a folder first.

2 versions of the same module can exist (module@1.0.0 and module@1.0.1) and they will not conflict. Generally npm packages are modules, loaded via require() or import

3: Building our own NodeJS package manager

We will be building an app which includes all 3 aspects of NPM.

Registry
CLI
Website

The aim is to get a basic PoC which will work for all 3.

Using an example-lib (a small package which returns some text) and example-app (a small express server using the previous package) we can test it works.

Wrapping all of the above in a single mono-repository with multiple packages.

Registry

For this we will use the default couchdb Docker image. It includes a basic unauthenticated CouchDB instance.

We will use docker-compose to set up our apps.

Our docker-compose.yml starts like this:

version: "3.0"

services:
  couchdb_container:
    image: couchdb:1.6.1
    ports:
      - 5984:5984

It does not need a volume mounted or anything else at this point.

We will use a Makefile to assist in running. We start by building and creating the registry database. Later I added a stop and reset so we could kill the docker application and reset the database quickly.

Makefile below:

up:
	docker-compose up --build

db:
	curl -XPUT http://localhost:5984/registry

stop:
	docker-compose stop

reset:
	curl -X DELETE \
		'http://localhost:5984/registry' \
		-H 'content-type: application/json'

Opening the browser confirms the database and couchDb instance exists

CLI

Located in our monorepo in packages/cli/. We will only be creating 2 commands for this PoC.

Install
Publish

The package.json will look like below

{
  "bin": {
    "our-npm-install": "./scripts/install.js",
    "our-npm-publish": "./scripts/publish.js"
  }
}

Using bin means inside the packages/cli folder we can run below:

npm install -g .

And from any directory in a terminal we can run the commands like this.

our-npm-publish

our-npm-install

I could have chosen to split these into a single script and use arguments, like the real NPM, but it did not seem worth it for this PoC. If I had gone that route I would have used yargs.

Install script

Located at packages/cli/scripts/install.js

It has 4 steps:

Grab the package name and ourDeps object, from the current working directory package.json
Iterate over each item in ourDeps, the version is ignored for our PoC.
1. make a fetch request to the couch-db tarball attachment (looks like http://localhost:5984/registry/{repo-name}/{repo-name}.tar.gz)
2. write the file to a local tmp.tar.gz file (for processing)
Extract the tmp.tar.gz contents using the tar library, into the current working directory node_modules/{repo-name}.
Finally delete the tmp.tar.gz file

The code is here.

#!/usr/bin/env node

const fetch = require("node-fetch")
const { writeFile } = require("fs")
const { promisify } = require("util")
const tar = require("tar")
const fs = require("fs")

const writeFilePromise = promisify(writeFile)
const apiUrl = "http://localhost:5984/registry"
const outputPath = `${process.cwd()}/tmp.tar.gz`

async function extractPackage(repoName) {
  const zipExtractFolder = `${process.cwd()}/node_modules/${repoName}`

  if (!fs.existsSync(zipExtractFolder)) {
    // create package in node_mods
    fs.mkdirSync(zipExtractFolder)
  }
  try {
    // Step 3
    await tar.extract({
      gzip: true,
      file: "tmp.tar.gz",
      cwd: zipExtractFolder, // current extract
    })
    console.log("Extract complete")
  } catch (e) {
    console.log("Extract error: ", e.message)
  }
}
async function downloadPackage(repoName) {
  // Step 2.1
  return (
    fetch(`${apiUrl}/${repoName}/${repoName}.tar.gz`)
      .then(x => x.arrayBuffer())
      // Step 2.2
      .then(x => writeFilePromise(outputPath, Buffer.from(x)))
      .catch(e => console.log("Download Error: ", e.message))
  )
}

async function run() {
  // Step 1
  const package = require(`${process.cwd()}/package.json`)
  // Step 2 - process each dep
  Object.keys(package.ourDeps).map(async repoName => {
    await downloadPackage(repoName)

    await extractPackage(repoName)

    // Step 4 - remove tar
    fs.unlinkSync(outputPath)
    console.log(`Downloaded: ${repoName}`)
  })
}

run()

Publish script

Located at packages/cli/scripts/publish.js

It has 4 steps:

From the current working directory grab
1. package.json file name field
2. The README.md contents
Create a tarball of the current working directory
Send the package tarball (as an attachment) and README contents (as plain text) to our couchdb instance, the document name is the package name.
Delete the local tarball file

#!/usr/bin/env node

const { unlinkSync, readFile } = require("fs")
const tar = require("tar")
const { promisify } = require("util")
const nano = require("nano")("http://localhost:5984")

const readFileAsync = promisify(readFile)

async function sendPackage(repoName, readmeContents) {
  const tarballName = `${repoName}.tar.gz`
  const filePath = `${process.cwd()}/${tarballName}`

  const tarballData = await readFileAsync(filePath)

  const registry = nano.db.use("registry")
  let response
  try {
    const docName = repoName

    // Step 3
    const response = await registry.insert({ readmeContents }, docName)
    await registry.attachment.insert(
      docName,
      tarballName,
      tarballData,
      "application/zip",
      { rev: response.rev }
    )
  } catch (e) {
    console.log("Error:", e)
  }
  console.log("Response success: ", response)
}

async function packageRepo(repoName) {
  try {
    // Step 2
    await tar.create(
      {
        gzip: true,
        file: `${repoName}.tar.gz`,
        cwd: process.cwd(),
      },
      ["./"]
    )
  } catch (e) {
    console.log("gzip ERROR: ", e.message)
  }
}

async function run() {
  // Step 1.1
  const repoName = require(`${process.cwd()}/package.json`).name
  // Step 1.2
  const readmeContents = await readFileAsync(`${process.cwd()}/README.md`, {
    encoding: "utf8",
  })

  await packageRepo(repoName)

  await sendPackage(repoName, readmeContents)

  // Step 4 - remove file
  unlinkSync(`${repoName}.tar.gz`)
}

run()

Website

Located under packages/website we will use Docker to create a basic NodeJS website.

Our packages/website/Dockerfile looks like.

FROM node:14-alpine

# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

# Install dependencies
COPY package.json package-lock.json ./
RUN npm install

# Bundle app source
COPY . ./

# Exports
EXPOSE 3000
CMD [ "npm", "run", "start.dev" ]

The website details are located in packages/website/src/server.js

For a request to the /packages/:package-name url

Query the couchdb instance for the package name
Using showdown library, render the README markdown as HTML.

If no package is found a nice message is printed.

// deps...
const nano = require("nano")("http://couchdb_container:5984") // no auth for GET

// Constants
const PORT = 3000
const HOST = "0.0.0.0"

// couchdb
async function findOne(packageName) {
  try {
    const registry = nano.db.use("registry")
    // Step 1
    const doc = await registry.get(packageName)
    console.log("client result: ", doc)
    return doc
  } catch (err) {
    console.log("ERROR: ", err.message)
  }
}

// App
const app = express()
app.get("/packages/:packageName", async (req, res) => {
  const packageName = req.params["packageName"]

  const result = await findOne(packageName)
  if (result) {
    const converter = new showdown.Converter()
    // Step 2
    const html = converter.makeHtml(result.readmeContents)
    res.send(html)
  } else {
    res.send("No package found")
  }
})

app.listen(PORT, HOST)
console.log(`Running on http://${HOST}:${PORT}`)

Lastly we will add the website to our docker-compose.yml so we can run it with the registry database.

The docker-compose.yml now looks like this

version: "3.0"

services:
  web:
    build: packages/website
    ports:
      - "3000:3000"
    restart: always
    volumes:
      - ./packages/website:/usr/src/app

  couchdb_container:
    image: couchdb:1.6.1
    ports:
      - 5984:5984

Testing it works

We now have 3 of the core 3 jobs carried out by NPM, so we need to test it works using an example application and example library.

`example-lib`

The library will just return a string.

The packages/example-lib/package.json looks like below.

{
  "name": "example-lib",
  "main": "index.js"
}

The packages/example-lib/index.js is below, it just returns a string.

module.exports = () => "example-lib data"

`example-app`

It will print information from a real library (e.g. express) and our example library.

Our packages/example-app/package.json looks like this. As mentioned the version is ignored for our PoC.

{
  "ourDeps": {
    "example-lib": null
  }
}

The packages/example-app/src/index.js is below.

const express = require("express")
const exampleLib = require("example-lib")

console.log("express function", express.urlencoded)
console.log("example-lib function", exampleLib())

Lastly we update the reset in our Makefile to also remove the package installed and uninstall the global binary commands. The final file is below:

up:
	docker-compose up --build

db:
	curl -XPUT http://localhost:5984/registry

stop:
	docker-compose stop

reset:
	curl -X DELETE \
		'http://localhost:5984/registry' \
		-H 'content-type: application/json'
	rm -rf packages/example-app/node_modules/example-lib
	cd packages/cli && npm uninstall -g our-npm-cli

Running the example

Setup commands

make up

make reset
Check website for the package
GET http://localhost:3000/packages/example-lib -> Notice the package does not exist yet
Setup the database and CLI
make db
cd packages/cli
npm install -g . -> Install the 2 CLI scripts globally (see bin inside package.json)
Publish the package then check website
cd ../example-lib
our-npm-publish -> Publish the example-lib package to our registry
GET http://localhost:5984/registry/example-lib/example-lib.tar.gz -> Location of the registry package tarball
GET http://localhost:3000/packages/example-lib -> Package README on website (screenshot below)
Use package
cd ../example-app
npm start -> See the missing package error (screenshot below)
our-npm-install -> Install the packages from package.json -> ourDeps list (screenshot below)
npm start -> Package found, now it works (screenshot below)

So thats it 🙌, our mini NPM works.

I encourage anyone that is interested to check out the code and play with the mechanism yourself.

What have we missed?

As mentioned there is functionality to each of the 3 core elements to NPM which we have omitted from our app. Some of those are:

CLI

Various commands including init and pack
Ability to download a package via argument
Creating lockfile (including versions and dependency info)
Caching and request/response data
Flags e.g --devDeps
Dependency module resolution (NPM has a lot of logic for managing this, see overview section)

Registry

Better authentication on the CouchDB instance
sha512sum hash of package/tarball contents ("Subresource Integrity" check)
Install counter
Semvar package versions

Website

Show install increment
Show versions and intergrity hash
A decent UI

Thanks so much for reading, I learnt a huge amount about NPM from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Under-the-hood of Git

Sat, 27 Feb 2021 00:00:00 GMT

For many years now Git has been the SCM (source control management aka version control) of choice. It offered many features which alternatives such as CVS did not, and combined with GitHub website created an entire CI pipeline which any teams Dev practices could be built around.

When I began reading about the mechanics of Git it was obvious that it's combination of many different techniques, all of which produce the "replicated versioned file system" know as Git, for example:

Linked lists,
File system objects database
Hashing (stat SHA-1 vs content SHA-1 vs content Deflate)
Differential encoding

So I decided to create a mini-working version with some of the core version control features. Thankfully there are many helpful books which break down how things work, so I have attempted to strip the internals down to its bare minimum.

This post will focus on:

repositories,
working directories,
staging,
committing
status checks.

I have omitted packfiles, deltas, branches, tags, merging and comparing staged chunks (diffing). I may do a follow up post/repository on those.

This is part of my "under-the-hood of" series:

A video for this talk can be found here. Part of my "under-the-hood of" series here.

The article today will be broken down into:

Overview
Building our own Git
- Our git code
- Testing it works
What have we missed?

1: Overview

Git is described as a distributed version-control system, which tracks changes in any set of files. It was initially released 15 years ago (in 2005) and has grown in functionality and popularity massively since then. As any developer who uses Github knows (or an alternative e.g. BitBucket/GitLab) it has become a staple in the world of software as a best practice.

Workflow

I am not going to review how it is used but the basic workflow can be summarised by:

initialise a new git repository
A file/s change is made locally and saved
The file/s is added to staging
The file/s in the staging area are comitted
The commit is pushed to a remote repository (pulling the latest before doing so).

We will break down each step, but before we do we need to review the mechanism at the core of Git, the "Object model".

Object model

The object model is essentially an incredibly efficient versioned file system (with replication).

Each file in the repository exists in the file system and the object database. The object database is a hash of the contents. A hash is an object, there are 4 types in total but today we will look at (excluding "tags"):

Blob -> a sequence of bytes. A blob in Git will contain the same exact data as a file, it’s just that a blob is stored in the Git object database. Basically the file contents.
Tree -> corresponds to UNIX directory entries. Can contain blobs or sub trees (sub directory). The commit tree has the entire project in blob and trees at time of the commit. It can recreate the entire project from that tree. Always from root directory even if a sub directory file is being updated in the commit.
Commit -> single tree id and commits preceding it

Each tree node, commit and file have their own unique 40 character long SHA-1 representation. The filename is a hash of the contents. If the contents change, so does the hash. Each time it changes a new entry/hash is added but keeps the old ones.

Inside a git repository they are found under the .git/objects folder.

This is my favourite image to describe the structure.

Hash

Within the object model, the filename is a 2-way SHA-1 encoding of the contents.

Git prefixes any Blob objects with blob, followed by the length (as a human-readable integer), followed by a NUL character Example:

> s='abc'
> printf "$s" | git hash-object --stdin

Equivalent to

> printf "blob $(printf "$s" | wc -c)\0$s" | sha1sum

Object file contents are compressed via DEFLATE zlib algorithm, it is less human readable or filename-friendly but a more efficient encoding.

Components

I will be covering the components we will be building in our mini-working version.

Working directory

The current system folder with git repository in, also known as the working tree.

HEAD

A file holding a ref to current working branch. Basically the last checked out workspace. It holds a reference to the parent commit, usually last branch checked out.

Found in the file .git/HEAD. Example

> ls  .git/HEAD

ref: refs/heads/master

> ls .git/refs/heads/master

2e1803ee08fa9aa36e4c5918220e283380a4c385

Branches

A branch is actually just a named pointer to specific snapshot. When it is checked out

moves HEAD pointer to point to the feature ref (branch)
moves all content from the current branch repo into the index file, so it's easy to track changes.
Make working dir match content of commit pointing to (using tree and blob objects to update working dir contents)

Repository

A git project stored on disk i.e. not in-memory. Essentially a collection of objects.

Staging

Area between working directory and repository. All changes in staging will be in the next commit.

Index file

The index is a binary file, it does not hold objects (blobs/trees), it stores info about files in repository. It is a virtual working tree state.

The index file is located at .git/index. You can see the status of the Index file via > git ls-files --stage

Information stored

For each file it stores

time of last update, name of file,
file version in working dir,
file version in index,
file version in repository

File versions are marked with checksums, a SHA-1 hash of stat(), not a hash of the contents. This is more efficient.

Refresh

It is updated when you checkout a branch or the working directory is updated. Runs in the background automatically.

Hashing

It uses uses a filesystem stat() to get the files information, to check quickly if the working tree file content has changed from version recorder in index file. Checks the file modification time under st_mtime.

The refresh literally calls stat() for all files.

Additional reading

The main goal of this post is the mini-working version below so we have only just touched briefly on how git works. Here are websites which go into far more details

2: Building our own Git

our git code

The code consists of 4 files, one for each command, plus a util.

init.mjs
status.mjs
add.mjs
commit.mjs
util.mjs

init.mjs

// imports excluded, see linked repo for details
const init = () => {
  const workingDirectory = workingDir()
  const files = glob.sync("**/*.txt", { cwd: workingDirectory }) // (1)

  const indexData = files.reduce((acc, curr) => { // (2)
    const hash = hashFileStats(curr)
    acc[curr] = {
      cwd: hash,
      staging: "",
      repository: "",
    }
    return acc
  }, {})

  fs.mkdirSync(`${workingDirectory}/.repo`) // (3)
  updateIndex(indexData)
  fs.writeFileSync(`${workingDirectory}/.repo/HEAD`) // (4)
  fs.mkdirSync(`${workingDirectory}/.repo/objects`) // (4)
}

(1) Grab all the files from the current working directory
(2) Build the index file using files stat() SHA-1 hash for each file
(3) Write a repository folder under .repo
(4) Inside repository write a HEAD file and objects folder

status.mjs

// imports excluded, see linked repo for details
const status = () => {
  const indexData = getIndexData() // (1)

  const notStaged = []
  const notComitted = []
  const updatedIndexData = Object.keys(indexData).reduce((acc, curr) => { // (2)
    const hash = hashFileStats(curr) // (2a)
    if (hash !== indexData[curr].cwd) { // (2b)
      acc[curr] = {
        cwd: hash,
        staging: indexData[curr].staging,
        repository: indexData[curr].repository,
      }
      notStaged.push(curr)
    } else {
      if (indexData[curr].cwd !== indexData[curr].staging) {
        notStaged.push(curr) // (2c)
      } else if (indexData[curr].staging !== indexData[curr].repository) {
        notComitted.push(curr) // (2d)
      }
      acc[curr] = indexData[curr]
    }

    return acc
  }, {})

  updateIndex(updatedIndexData) // (3)

  console.log("\nChanged locally but not staged:")
  notStaged.map(message => console.log(`- ${message}`)) // (4)
  console.log("\nStaged but not comitted:")
  notComitted.map(message => console.log(`- ${message}`)) // (5)
}

(1) Grab the index data
(2) For each item in the index data
(2a) Grab files stat() SHA-1 hash
(2b) If doesnt match current working dir stored hash of file, flag as changed not staged
(2c) If does match above but doesnt match staged, flag as not staged
(2d) If does match staged but not repository, flag as not comitted
(3) Update index file
(4) Output local changes not staged
(5) Output staged changes not comitted

add.mjs

// imports excluded, see linked repo for details
const add = () => {
  const workingDirectory = workingDir()

  const files = process.argv.slice(2) // (1)

  const indexData = getIndexData()

  console.log("[add] - write blob objects")
  const updatedFiles = files.map(file => {
    const blobHash = hashBlobContentsInFile(file) // (2)
    const blobDir = blobHash.substring(0, 2)
    const blobObject = blobHash.substring(2)

    // TODO - check exists first - for re-adding file with earlier contents
    fs.mkdirSync(`${workingDirectory}/.repo/objects/${blobDir}`)

    const blobCompressed = compressBlobContentsInFile(file) // (3)
    fs.writeFileSync(
      `${workingDirectory}/.repo/objects/${blobDir}/${blobObject}`,
      blobCompressed
    )

    const hash = hashFileStats(file) // (4)

    return {
      file,
      hash,
    }
  })

  const updatedIndexData = Object.keys(indexData).reduce((acc, curr) => { // (5)
    if (!updatedFiles.find(item => item.file === curr)) { // (5a)
      acc[curr] = {
        cwd: indexData[curr].cwd,
        staging: indexData[curr].staging,
        repository: indexData[curr].repository,
      }
      return acc
    }
    acc[curr] = {
      cwd: indexData[curr].cwd,
      staging: updatedFiles.find(item => item.file === curr).hash, // (5b)
      repository: indexData[curr].repository,
    }
    return acc
  }, {})

  updateIndex(updatedIndexData) // (6)
}

(1) Explicitly give files e.g. one.txt and two/three.txt
(2) For each file, get contents in SHA-1 and use for directory name and filename
(3) Get DEFLATED value and use for content
(4) Get SHA-1 value for files stat()
(5) Update the index
(5a) If file was not touched, just proxy values
(5b) If file was touched, update staging for the file
(6) Override old index data with new index data

commit.mjs

// imports excluded, see linked repo for details

// array of dir (name) and files (children), ordered by bottom-up
const _buildTree = paths => {
  return paths.reduce(
    (parent, path, key) => {
      path.split("/").reduce((r, name, i, { length }) => {
        if (!r.children) {
          r.children = []
        }
        let temp = r.children.find(q => q.name === name)
        if (!temp) {
          temp = { name }
          if (i + 1 === length) {
            temp.type = "blob"
            temp.hash = hashBlobContentsInFile(path)
          } else {
            temp.type = "tree"
          }
          r.children.push(temp)
        }
        return temp
      }, parent)

      return parent
    },
    { children: [] }
  ).children
}

const commit = () => {
  const workingDirectory = workingDir()
  const indexData = getIndexData()
  // TODO - if comitted already then dont recreate tree?? PROB chek first
  const paths = Object.keys(indexData).filter( // (1)
    item => indexData[item].staging || indexData[item].repository
  )

  const rootTrees = _buildTree(paths) // (2)

  const flattenedTrees = rootTrees.reverse().reduce((acc, curr, key) => { // (3)
    if (curr.children) {
      const hash = createTreeObject(curr.children) // (3a)
      const clone = Object.assign({}, curr)
      delete clone.children
      clone.hash = hash
      acc.push(curr.children) // (3b)
      acc.push([clone])
    } else {
      acc[key].push(curr) // (3c)
    }
    return acc
  }, [])

  const rootTree = flattenedTrees.reverse()[0]
  const treeForCommit = createTreeObject(rootTree) // (4)

  const parent = getParentCommit()

  const commit = { // (5)
    tree: treeForCommit,
    parent: parent === "undefined" ? null : parent,
    author: "CRAIG", // hardcoded for now
    committor: "CRAIG",
    message: "Initial commit",
  }

  const commitHash = createCommitObject(commit) // (6)

  const updatedIndexData = Object.keys(indexData).reduce((acc, curr) => { // (7)
    const { cwd, staging, repository } = indexData[curr]
    let updatedRepo = repository
    if (staging !== repository) { // (7a)
      updatedRepo = staging
    }
    acc[curr] = {
      cwd: indexData[curr].cwd,
      staging: indexData[curr].staging,
      repository: updatedRepo,
    }
    return acc
  }, {})
  updateIndex(updatedIndexData)

  fs.writeFileSync(`${workingDirectory}/.repo/HEAD`, commitHash) // (8)
}

(1) Grab files of files to commit
(2) Build tree for files in staging or comitted, excluded working dir only
(3) Iterate items root "tree" into a flattened array of trees
(3a) If tree, create tree for children
(3b) Then add children to flattened tree
(3c) If not a tree, push with previous tree
(4) Create tree object for root
(5) Create commit object, using parent commit if exists and the tree hash
(6) From commit object get commit hash
(7) Update index file
(7a) If staging hash does not match repository hash then update. An existing file has been updated.
(8) Update HEAD with the latest commit

utils.mjs

I have included the helper file but hopefully the names are pretty self-explanatory.

The largest is createTreeObject and createCommitObject. Both of which:

Process given contents into a hash
Compress given contents
Writes compressed contents to the respective directory and file - The first 2 characters of a hash become the directory and the rest the filename.

import fs from "fs"
import crypto from "crypto"
import zlib from "zlib"

export const workingDir = () => {
  const cwd = process.cwd()
  return cwd + "/src"
}

export const sha1 = object => {
  const string = JSON.stringify(object)
  return crypto
    .createHash("sha1")
    .update(string)
    .digest("hex")
}

const getFilePath = file => {
  const workingDirectory = workingDir()
  return `${workingDirectory}/${file}`
}
const getContentsInFile = file => {
  const path = getFilePath(file)
  return fs.readFileSync(path, { encoding: "utf-8" })
}

export const compressBlobContentsInFile = file => {
  const contents = getContentsInFile(file)
  return zlib.deflateSync(contents)
}

// always same based on contents
export const hashBlobContentsInFile = file => {
  const contents = getContentsInFile(file)
  return sha1({ type: "blob", contents })
}

// different based on midified time
// remove atime + atimeMs which are different each stat() call
export const hashFileStats = file => {
  const path = getFilePath(file)
  const contents = fs.statSync(path)
  delete contents["atime"]
  delete contents["atimeMs"]
  return sha1(contents)
}

export const getIndexData = () => {
  const workingDirectory = workingDir()
  return JSON.parse(
    fs.readFileSync(`${workingDirectory}/.repo/index`, { encoding: "utf-8" })
  )
}

export const updateIndex = indexData => {
  const workingDirectory = workingDir()
  fs.writeFileSync(`${workingDirectory}/.repo/index`, JSON.stringify(indexData))
}

// hash contents, create tree, return hash
export const createTreeObject = contents => {
  const contentsClone = Object.assign([], contents)
  const flatContents = contentsClone.map(item => {
    delete item.children // dont need full children depth
    return item
  })
  const workingDirectory = workingDir()
  const stringContents = JSON.stringify(flatContents)
  const treeHash = sha1(stringContents)
  const treeDir = treeHash.substring(0, 2)
  const treeObject = treeHash.substring(2)
  const treeCompressed = zlib.deflateSync(stringContents)
  // create tree object
  fs.mkdirSync(`${workingDirectory}/.repo/objects/${treeDir}`)
  fs.writeFileSync(
    `${workingDirectory}/.repo/objects/${treeDir}/${treeObject}`,
    treeCompressed
  )
  return treeHash
}

export const createCommitObject = contents => {
  const workingDirectory = workingDir()
  const stringContents = JSON.stringify(contents)
  const commitHash = sha1(stringContents)
  const commitDir = commitHash.substring(0, 2)
  const commitObject = commitHash.substring(2)
  const commitCompressed = zlib.deflateSync(stringContents)
  // create commit object
  fs.mkdirSync(`${workingDirectory}/.repo/objects/${commitDir}`)
  fs.writeFileSync(
    `${workingDirectory}/.repo/objects/${commitDir}/${commitObject}`,
    commitCompressed
  )
  return commitHash
}

export const getParentCommit = () => {
  const workingDirectory = workingDir()
  return fs.readFileSync(`${workingDirectory}/.repo/HEAD`, {
    encoding: "utf-8",
  })
}

Testing it works

I wrote a small project to test the version control. 3 files each with a line of text, 2 of which inside a folder.

The above scripts are found inside bin/

A working directory / application is found in src/

one.txt
two/three.txt
two/four.txt

Then I wrote some inegration tests (test/index.integration.spec.js) to help track what happens to our repository for a given command, the steps (and results) are:

repo:init => created INDEX with current working directory files stat() hash
repo:status => flag 3 new local changes not staged (those above)
repo:add one.txt two/three.txt =>
- should create blob objects, inside 2 character-long directories, with content compressed
- should update INDEX, move items to staged
repo:status => flag 1 new local changes not staged and 2 changes not comitted
Manually update one.txt
repo:status => similar to previous except now flags one.txt as locally changed
repo:add one.txt => re-add updated file one.txt should update blob object
repo:status => re-added file should show with old added file
repo:add two/four.txt => add two/four.txt so 2 items in tree object
repo:commit => should create tree and commit object and update HEAD and INDEX

What have we missed?

As mentioned there are many additional parts to the real Git version control which we have omitted from our library. Some of those are:

Comparing change chunks (diffing)
Packfiles
Deltas
Branches
Tags
Merging

Thanks so much for reading, I learnt a huge amount about Git from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Under-the-hood of GraphQL

Sat, 28 Nov 2020 00:00:00 GMT

In recent years GraphQL has really taken off as a pattern/library/type system. It offers much which REST does not and its standardisation and flexibily has really helped in its adoption. When I began looking for a detailed explanation of how graphql works internally I struggled to find anything which was not too high-level. I wanted to see how the schema, query, types and resolvers all worked together mechanically.

Its not possible to focus on all parts in detail in this article so I have focused on the execution part of the query lifecycle.

This is part of my "under-the-hood of" series:

A video for this talk can be found here. Part of my "under-the-hood of" series here.

The article today will be broken down into 2 parts:

Overview
Building our own GraphQL executor

1: Overview

First we need to ask what is graphql? There are a couple of answers

It is:

A type system - type definitions define how data should look. They are written to show what is included and highlight the relationships (how things relate). This system definition language can be transformed into AST.
A formal language for querying data - what to fetch from where.
Rules for validating or executing a query against the Schema.

Point (3) above references the official Specification which defines the rules for types, validation and executing the schema. It can be found at:

spec.graphql.org

There is also a documentation-friendly website graphql.org/learn

All languages follow the spec, and the JS graphql library frequently references part of it. We will be looking at that library as part of this section.

Building the schema

Building the schema is important part for a graphql application, as mentioned above it defines all types and their relationships. There are 2 steps to this

1. Parses "schema notation" (usually found in a `schema.graphql` file) into AST

The parser will throw errors if it is not a GraphQL schema. See snippet from types/schema.js (below)

export function isSchema(schema) {
  return instanceOf(schema, GraphQLSchema)
}

2. Transform AST into objects and instances

We need a schema which is a type instance of GraphQLSchema (see above snippet). We then need objects inside the schema which match types, for example a scalar or an object.

Example of a scalar (below snippet)

const OddType = new GraphQLScalarType({
  name: "Odd",
  serialize(value) {
    if (value % 2 === 1) {
      return value
    }
  },
})

The native scalar types defined for GraphQL are ID, Int, Float, String and Boolean. You can define your own inside your type system.

Summarise

Essentially for building the schema we turn this graphql schema notation:

type Book {
  id: ID!
  title: String
  authors: [Author]
}

Into this Javascript

const Book = new GraphQLObjectType({
  name: 'Book',
  fields: () => ({
    id: { type: new GraphQLNonNull(GraphQLID) },
    title: { type: new GraphQLString },
    author: { type: new GraphQLList(Author) },
  })
}

Almost all of the GraphQL types that you define will be object types. Object types have a name, but most importantly describe their fields.

A GraphQLSchema looks like the below in raw POJO form:

GraphQLSchema {
  astNode: {
    kind: 'SchemaDefinition',
    ...
  },
  extensionASTNodes: [],
  _queryType: Query,
   ...
  _typeMap: {
    Query: Query,
    ID: ID,
    User: User,
    String: String,
    ...
  },
  ...
}

It holds AST information too but the root Query and types are found under the _typeMap property.

Adding resolvers

Resolvers cannot be included in the GraphQL schema language, so they must be added separately.

They are added to the _typeMap property, under _typeMap.<Type>._fields.<field> So after adding the resolvers, a schema object might look like below:

e.g.

 _typeMap: {
   Query: { _fields: { users: { resolve: [function] } } },
   User: { _fields: { address: { resolve: [function] } } }
}

From the HTTP request side the community has largely standardized on a HTTP POST method. But what happens when the server receives a query?

Query lifecycle

The GraphQL spec outlines what is known as the "request lifecycle". This details what happens when a request reaches the server to produce the result.

There are 3 steps that occur once the lifecycle is triggered.

1. Parse Query

Here the server turns the query into AST. This includes:

Lexical Analysis
- GraphQL's Lexer identifies the pieces (words/tokens) of the GraphQL query and assigns meaning to each
Syntactic Analysis
- GraphQL's parser than checks whether the pieces conforms to the language syntax (grammar rules)

If both these pass the server can move on.

In graphql-js this is all found under the parser.js file and function.

This query:

query homepage { posts { title author } }

Becomes the AST

{
  "kind": "Document",
  "definitions": [
    {
      "kind": "OperationDefinition",
      "operation": "query",
      "name": {
        "kind": "Name",
        "value": "homepage"
      },
      "selectionSet": {
        "kind": "SelectionSet",
        "selections": [
          {
            "kind": "Field",
            "name": {
              "kind": "Name",
              "value": "posts"
            },
            "selectionSet": {
              "kind": "SelectionSet",
              "selections": [
                {
                  "kind": "Field",
                  "name": {
                    "kind": "Name",
                    "value": "title"
                  }
                },
                {
                  "kind": "Field",
                  "name": {
                    "kind": "Name",
                    "value": "author"
                  }
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

You can see this for yourself on astexplorer under GraphQL.

2. Validate Query

This step ensures the request is executable against the provided Schema. Found under the validate.js in the graphql-js library.

While it is usally run just before execute, it can be useful to run in isolation. For example by a client before sending the query to the server. The benefit is that the validator could flag an invalid query before it is sent to the server, saving a HTTP request.

It works by checking each field in the query AST document against its corresponding type definition in the schema object. It will compare argument type compatibility as well as coercion checks.

3. Execute Query

This step is by far the most intensive and the step I often found the most confusing in its mechanism. We will be digging deeper into this step in part 2, so lets look at the process involved from a high-level.

Identify the operations i.e. a query or mutation? (several queries can be made at once)
Then resolve each operation.

For step (2) each query/mutation is run in isolation.

Resolving each operation

For step 2 GraphQL iterates over each field in the selection-set, if it is a scalar type resolve the field (executeField) else recurse the selection-sets until it resolves to a scalar.

The way this works is that the engine calls all fields on the root level at once and waits for all to return (this includes any promises). Then after reading the return type it cascades down the tree calling all sub-field resolvers with data from the parent resolver. Then it repeats this cascade on those field return types. So simply speaking it calls the top-level Query initially and then the root resolver for each type.

The mechanism caled "scalar coercion" comes into play here. Any values returned by the resolver are converted (based on the return type) in order to uphold the API contract. For example a resolver returning string "123" with a number type attached to it will return Number("123") (i.e a number).

This is found under the execute.js file and function inside graphql-js.

Lastly the result is returned.

Introspection system

Its worth mentioning the introspection system. This is a mechanism used by the GraphQL API schema to allow clients to learn what types and operations are supported and in what format.

The clients can query the __schema field in the GraphQL API, which is always available on the root type of a Query. Any interactive GraphQL UIs rely on sending an IntrospectionQuery request to the server, which builds documentation and auto-completion with it.

Libraries

As part of my research I covered many different libraries, so I thought it was worth giving a quick overview of the main ones in the JS ecosystem.

graphql-js

The reference implementation of the GraphQL spec, but also full of useful tools for building GraphQL servers, clients and tooling. It's a GitHub organisation with many mono-repositories.

It performs the entire Query Lifecycle (including parsing schema notation).

Schema

Requires library specific GraphQLSchema instance. In order to be an executable schema it requires resolver functions.

Example functions

buildClientSchema

Take output of introspection query and builds the schema out of it

buildASTSchema

Once you have parsed the schema into AST this then transforms into a GraphQLSchema type

graphql-tools

It's an abstraction on top of graphql-js. Houses lots of functionality including generating a fully spec-supported schema and stitching multiple schemas together.

Example functions

makeExecutableSchema

Takes arguments
- typeDefs - "GraphQL schema language string" or array
- resolvers - is an object or array of objects
Returns a graphql-js GraphQLSchema instance

loadSchemaSync

Point this to the source to load your schema from and it returns a GraphQLSchema.

addResolversToSchema

Takes a GraphQLSchema and resolvers then returns an updated GraphQLSchema.

apollo-server

It's also an abstraction on graphql-js. Uses the graphql-tools library for building GraphQL servers.

Introspection is disabled on production by default but on non-production it uses introspection to expose a /graphql playground.

Apollo Studio

Not really a library but I thought worth a mention.

Plugs into apollo-server and provides stats and information of your GraphQL server in realtime. Such as:

Monitoring operations
Tracking errors
Profiling resolvers (see example on below screen)

2: Building our own GraphQL executor

Here we will build our own GraphQL execute function that a parsed query and our schema can run against. It will ignore any validation. We will use GraphQL instance types to create a schema, ignoring the schema parsing step.

We saw earlier in the query lifecycle section that the graphql-js library contains the necessary functions and they can be imported individually for usage. e.g. parser(), validate() and execute(). In addition to those functions it also exports type instances which can be used to build up a graphql schema. While we WILL NOT be using any of the functions they provide, we WILL BE using the type intances they provide to build a valid schema.

We will be using 3 different scenarios to examine the mechanism in place.

Before we go any further here are some of the common terms mentioned.

Terms

Definitions
- Name for top-level statements in the document
- Most GraphQL types you define will be object types and they have name and describe fields
Operations
- The type of request i.e query/mutation
Selections
- They are definitions that appear at a single level of a query
- For example field references e.g a, fragment spreads e.g ...c and inline fragment spreads ...on Type { a }
- See in the spec
Execution Context
- Data that must be available at all points during the queries execution.
- Includes the Schema of the type system currently executing and fragments defined in the query document
- It is passed into the resolver as context
- Commonly used to represent an authenticated user or request-specific cache
Resolvers
- Functions to populate data for a field in your Schema
- Can return a value, a promise or an array of promises

This part is split into 2 steps.

Step 1 - Query and schema
Step 2 - Execute

Step 1 - Query and schema

Step 1 is to gather our query AST and schema instances using graphql-js, so they are ready for processing

These are the types which we will be importing from the graphql-js library. 2 scalars, 1 object and the schema type.

The query will look like below, it is close to how a normal graphql query would run, except usually there is an extra parse(query) step before execution.

// "document" = is our query in (hardcoded) AST form
const schema = new GraphQLSchema({ …<schema objects> })
// GraphQLSchema contains class instances of root-level query, fields, types, resolvers
ourExecute({ schema, document });
// execute query AST with schema

Normally this would be:

// schema = built from "schema.graphql" and "graphql-tool" loadSchemaSync/makeExecutableSchema/addResolversToSchema
const document = parse(query)
execute({ schema, document })

We will be looking at 3 scenarios. Each scenario below will contain 2 images.

The query AST - This is the query put into AST form
The schema - This is how the schema looks (see comment for pre-schema parsing form)

Scenario 1

A root query with resolver args.

Query: { test(aInt: -123) }

Query AST

Schema

Scenario 2

A root query with inner object and resolver args.

Query: { test(aInt: -123) { name } }

Query AST

Schema

Scenario 3

A different root type with inner object.

Query: { person { name } }

Query AST

Schema

Step 2 - Execute

Now that we have the setup complete for 3 scenarios we will look to build our actual execute function. The code is below and will be explained.

Starts on line 54. Here we call execute with our first definition as well as the types map. So we are limiting our POC to the first operation which we know is the query. In reality it would iterate over the definitions.

Inside execute (line 3) we grab the Query and rootTypes (line 6), then process the first selection in the selection set (line 8). Again hardcoded for this POC. We then grab the returnType (line 10) this would be used for validation against the query result, I have omitted that here. Line 11 creates an empty operations object to store any operation arguments on for later.

On line 13 we check for arguments and grab the argument name and value. We could also grab the type (line 16) which would be used for validation.

We define executeField on line 19, we will go through that next. Lastly we call executeFields with our current selection-set and an empty object. This will make up our response to return.

Now on to executeFields. It takes a selection-set, response object and field type as arguments. This is so it can call the query tree recursively and build the response data as we go along.

Line 20 we iterate over all selection-set selections (so this part works with multiple selections). We grap the field (line 21) and check for it on the root query level (line 24). For "Scenario 1" the field is user. So it checks Query._fields[user].resolve, which does exist for "Scenario 1".

If it exists (i.e. "Scenario 1") we call the resolver with the argument name and value, and then place the data returned from the resolver onto that field name (e.g. user) on the response (line 28).

Line 32 we then check if the return types field exists on the Schemas rootTypes object. If so we execute it with the parents resolver data and then write that to the field on the response (line 36). For "Scenario 1" the returnType is a String, as a scalar this does not have _fields so this would not trigger. For "Scenario 2" the returnType is a Person which has _fields, one of which is name but right now the field is test so would not trigger until recursed below.

Line 40 and 41 we check if this selection has got selection-sets itself and if so execute those. We hand the selection-set, the current response data for this field, and the current field name. So we are recursively calling executeFields with the deeper selection sets and building up a single response to return. "Scenario 2" will call the root query resolver first, then call executeFields again with the inner name field, where the fields resolver will be found and called with the existing resolver data.

Lastly we return the built response data on line 48, see userResp.

Its worth noting we have omitted any scalar checks, with the real graphql-js.execute the field type is checked as part of the validation to make sure that non-scalars should have sub selections. However line 32 (rootTypes?.[returnType]?._fields?.[field]?.resolve) is a kind of scalar check (scalars don't have _fields).

Checking results

In order to check this would work I wrote some unit tests found here. It includes some pure assertions as well as spies and I initially ran them against graphql-js.execute() to ensure they were written correctly. I then swapped to using my executor. The schema objects are those shown earlier, but this is all of it together.

Scenario 1 looks like this

Scenario 2 looks like this

Scenario 3 looks like this

A summary of the data assertions is here:

Here is the results of the test runner:

It works !!! Good job 👍💪

I encourage anyone that is interested to check out the code and play with the mechanism yourself.

What have we missed?

As mentioned there are many additional parts to the real graphql executor which we have omitted from our library. Some of those are:

Scalar type checks and scalar coercion
Processing multiple queries at once
Building response data not via pass-by-ref
Handling more complex scenarios
Validation of query (including errors)
Execution context - would be passed into the resolver.

Thanks so much for reading (or watching), I learnt a huge amount about GraphQL from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

4 techniques to improve as a software dev

Sun, 16 Aug 2020 00:00:00 GMT

I spent many years struggling to improve as an engineer. Learning how to use modern tools and frameworks is not the same as actual improvement and in the long run I had not found that it had any kind of meaningful impact on the kind of input I would have in my team and company.

This article is aimed at the software engineer who is wondering how to get to the next level in their learning. Perhaps trying to have a larger input and a more direct impact on all manner of interactions. This can range from reviewing Pull Requests to team meetings.

Today we will be looking at 4 techniques that I have utilised to help me improve as an engineer.

Experience more patterns and interesting solutions
Improve your general problem solving
Identify and learn from your current behaviour
Utilise common management principles

1. Experience more patterns and interesting solutions

There is a huge amount to learn from other people's code. Most are expected to review code written internally but what about code written externally?

The JavaScript ecosystem is a jungle of public open-source tools, many of which have different and interesting designs. I would recommend finding your favourite 2-3 tools/frameworks. Start with some basic questions and dig deeper and deeper into the code, identifying any patterns and solutions which you may not be familiar with.

This is something I have experience with. I began an "under-the-hood of" series for this very reason and feel I have learnt a tremendous amount by applying this practice to TypeScript, Webpack and many more. But I encourage you to do it yourself and see the results.

2. Improve your general problem solving

If you are somebody who has sometimes struggled to identify optimal solutions to a given problem, I recommend using a technique called "diverse problem solving". This is how it works.

To start with a problem is identified (e.g. "should our onboarding form have a back button")

Review the problem from as many key perspectives, for example:
- The happy path
- All possible error states
- Mobile/tablet/desktop
- Consistency with other products or the code
Come up with 2-3 different solutions
List the pros and cons of each
Pick the most optimal 1

This technique helps you practice identifying areas which a given problem will effect and therefore determine the best possible solution faster.

It is a not a quick process so is not practical to do for every problem. But what I found was that applying this once/twice a week really helped me improve my overall problem solving. For me, it helped to write the output of each step down but feel free to do it the way you are comfortable.

3. Identify and learn from your current behaviour

I would sometimes look back on Pull Request comments or even my input in certain meetings and think "hmm yeah I could have said X better" or "I said Y but really it was not that relevant". I started applying a technique to help me identify the gaps in my skillset and behaviour which were leading to these kinds of (not quite negative, but not positive) input.

The technique goes like this.

Keep a log of any input you give in a PR/meeting etc which you thought could have been better
At the end of the week look back on all the points, and for each one fill in this template

Date:
Event:
What I said/did:
What I could have said/done:
Lesson:

From this I have collated a list of "lessons" which are helping me plug up those gaps I had which I would not have previously been able to. Do not focus on the negativity, put your focus on the lessons.

Some of these lessons may well come from colleague or manager feedback, but its not always easy for them to identify behaviour which made you feel bad or actively want to improve.

4. Utilise common management principles

Learning the right way to look at a problem, solution or agenda item is crucial to maximising your impact.

There are a couple of management principles which I have found over the years have really helped me in a given situation, despite the fact I am not a manager.

For example the "Eisenhower Matrix". Its obviously completely subjective when I say "for managers" as this kind of knowledge can be useful to anyone at anytime. But I only discovered it by starting to read management books and the sections on "how to prioritise".

For anyone who is not familiar with the Eisenhower matrix, its idea is to help you categorise an item into 1 of

Important and urgent
Important but not urgent
Not important but urgent
Not important and not urgent

Its definitions I have always found very helpful:

Important = lead to us achieving our goals, or others theirs
Urgent = Immediate consequence if left

Inside a meeting or Pull Request I will always consider those 2 definitions and where the current issue fits in relation to them.

What I have found is that my reasoning has gotten stronger as my understanding on "why we would or would not do a task" has grown.

There are many other principles which can improve the way you look at a problem or agenda item.

I have spent the past couple years using these techniques and I have really found my input has improved. I can look back on my entries from 12 months ago and clearly see that I learnt the lessons.

The result of this is that I am much happier with my behaviour and overall impact within my team but also my company.

If you have any ideas for other techniques I would love to hear them, please email or tweet me. Thanks, Craig

Under-the-hood of VSCode auto formatters (e.g Prettier)

Thu, 02 Jul 2020 00:00:00 GMT

VSCode has become a staple for many developers local environment. One of the reasons is the powerful extensions which can be run from within the IDE itself, from type checking to code auto formatting. Here we will take a look at an overview of VSCode and then dig into how the auto formatting works. The final task will be to build a small version of Prettier (or eslint --fix) which can run as a VSCode extension. The goal is to understand the mechanics at work inside this kind of extension and some key differences with other types of extensions.

This is part of my "under-the-hood of" series:

A video for this post can be found here. A part of my "under-the-hood of" video series.

The article today will be broken down into 3 parts:

Electron
VSCode extensions
Our own Prettier extension

1. Electron

Under the hood VSCode is built with Electron. Electron is a software framework which allows for the development of desktop GUI applications using web technologies. It combines the Chromium rendering engine and the Node.js runtime. It was recently added to the OpenJS Foundation.

There are 2 types of processes inside Electron

Main process
- only 1 run at a time
- job to handle all the Renderers
Renderer process
- many running at once
- represents a page with UI
- job to run code

A basic example of creating a new node process (renderer) is a below.

let { fork } = require("child_process")
let serverProcess = fork(__dirname + "/server.js")

Messages are sent across processes via IPC (inter-process communication), this enables rendrer-to-renderer communication, bypassing the main process. The GUI's are Chromium web pages.

2. VSCode extensions

VSCode has extensions which introduce more advanced functionality. As it is built with Electron it must conform to the process rules.

In VSCode all extensions run in a single Renderer process called the "extension host process" (process name shared-process, in task manager they are under Code helper (renderer)).

There seem to be 2 main types of VSCode extensions:

The extension host proccess does all the work
The extension host process runs a "client" whose job is to spawn a "server" Renderer process, which will do most of the heavy lifting. The server communicates with the client.
- this is common for a Language Server
- it is worth noting that only the "extension host process" can update the GUI.
You can see which process is running what task via the VSCode Process Explorer (CMD Palette "Developer: Open Process Explorer")

Type 2 - Language Servers

Language Servers are background tasks spawned by an extension for a specific purpose.

TypeScript and Eslint are both examples of extensions which run Language Servers for their compilers. This is so the "server" can do the majority amount of the processing in the background, and only needs to relay messages (e.g. diagnostics) to the "client". They are often configured to re-run on save. This produces a more performant "client". In general it is good practice to run linters under a Language Server.

There are several strong pros of using Language Servers:

Persistent memory and caching - the server never dies so can maintain a cache in memory to perform faster compiles in future runs
Faster boot up times - the server is already running shrinking any boot-up wait time to practically 0.

Finally it is worth noting that VSCode comes with its own "Language Server Protocol" which makes the client/server communication easier. It is a JSON-RPC standard between the development tool and the language server.

See example architecture below.

There is a language server extension guide available here. An extension must make use of vscode-languageclient to implement a language client and server.

Type 1 - "One-and-dones"

For extensions which do not run a langauge server, a common term is "one-and-done".

Examples of this are Prettier and Mocha (non-watch mode). Where the extension/compiler runs once and then ends until it is called into action again, perhaps via a file save (assuming it respects your preference for editor.formatOnSave, which most extensions do).

They can be less performant than Langauge Server as no data or information is persisted and must run in its entirety each time. However:

there is no complex communication required between server and client.
for something like Prettier which reformats the entire file, it would require sending a huge amount of data between the client and server, so doing it all on the main extension process in the long run might be more performant.

3. Our own Prettier extension

We are going to build a VSCode extension which will be a stripped down version of Prettier. It will run for 2 different scenarios.

Scenario 1

Before

myFunction()

After

myFunction()

So for scenario 1 it will respect that no arguments are given and not change anything.

Scenario 2

Before

myFunction(one, two, three)

After

myFunction(
  one,
  two,
  three
);

So for scenario 2 it will recognise the arguments, format them correctly and update the code in our file.

Lets get started 💪

1. JSON setup

Add necessary setup to package.json

  "main": "./src/extension",
  "publisher": "craig-vscode-extension",
  "activationEvents": [
    "onCommand:extension.SexifyIt"
  ],
  "contributes": {
    "commands": [
      {
        "command": "extension.SexifyIt",
        "title": "Sexify it"
      }
    ]
  },

This will

tell VSCode the extension JavaScript can be found in ./src/extension
register our extension to run on activation (when it starts), see activationEvents
tell VSCode to title our extension under Sexify it inside the VSCode command palette.

Aswell as a simple launch file .vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "extensionHost",
      "request": "launch",
      "name": "Launch Client",
      "runtimeExecutable": "${execPath}",
      "args": ["--extensionDevelopmentPath=${workspaceRoot}"]
    }
  ]
}

This is so VSCode knows how to run our client.

2. Build a basic text replacement extension

Inside src/extension.js add the below.

Lets walk through what it is doing

line 2 register our extension "extension.SexifyIt"
line 6 grabs the current active editor
line 9 grabs all the document text
line 10 grabs the range details for the current selected text (it needs a range for the replace function)
line 13 calls our processing and reprinting function returning the new code, more on that in a moment
line 16/17 replaces the current selected range with our newly formatted code

3. AST processing and reprinting

Inside the same file we define some AST utilities.

line 2-5 are simple character utilities
line 6 is a small function for an "Expression" statement (e.g. myFunction())
the Statements object contains functions for each AST node type, with knowledge of how to "reprint" them i.e. turn AST back into JavaScript. It contains an ExpressionStatement and a CallExpression, which is what we need for our scenarios.

The bulk of the processing is on line 21-28.

grab number of arguments in function - line 21
if not greater than 0 (line 22) return as is
else we add a line break (line 23) and process each argument (line 26)
details of line 26:
- add an "indent" then the "argument name"
- check if it is last argument, if not add a coma followed by a line break, else just add line break
finally on line 29 we return an array of all the strings

Our function would output the following from the above:

["myFunction", "(", "\n", " one,\n", " two,\n", " three\n", ")"]

It contains all the necessary code including the updated formatting. Notice the line breaks and indents.

Finally the code to kick it off

line 2 we convert out function string into an AST object
- using a 3rd party tool called abstract-syntax-tree for this
line 4 we map over each item in the body
- the body is an array of each "block" (sometimes a line, sometimes an entire function)
we call the Statements function for that AST node type (i.e. here will be CallExpression)
lastly on line 5 and 6 we flatten the new array and rejoin it so it becomes a string again.

Checking it works

So by running our extension in debugger mode (i.e. F5) we can open a file with our JavaScript in (example in files/test.js), highlight the code, open the "Command Palette" and find our "Sexify it" extension.

It will turn...

myFunction(one, two, three)

Into this

myFunction(
  one,
  two,
  three
);

Confirming it works 👍. Converts our single-line function into multi-line with all the correct indents, exactly like Prettier does.

Thanks so much for reading or watching, I learnt a huge amount about VSCode and auto formatters from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Feature team vs Product team, which are you?

Sat, 13 Jun 2020 00:00:00 GMT

There are many types of teams and cultures that exist inside of businesses which utilise groups of software engineers/developers. Over the past 10 years I have experienced many of them so today I thought I would write about what are, in my opinion, the most common 2. Then finish with why 1 of them is my favourite.

To clarify these types of teams can be found in any company which uses technology, regardless of whether the end product is software-based (SASS) or if it is targeted directly at customers or businesses (i.e. B2B or B2C).

The Feature Team 🛠

Features teams typically consist of:

1 project manager
1 designer or UX specialist
2-10 developers or testers

They are given a roadmap laid out by senior project leaders and they take responsiblity for building it. There is freedom in how they execute the features and projects.

Their primary focus is to delivery features and projects.

The Product Team 📊

Product teams consist of similar members but with a slight difference

1 product owner -> note the difference
1 designer or UX specialist
2-10 developers or testers

If the team contains many more members than this then the team begins reaching critical mass.

Product teams are given a business problem to solve, not a roadmap. These problems usually come from a "Product Strategy"

They are measured by their business results, so there is a lot of pressure to ensure their chosen features and projects help the customers and the business.

Their primary focus is to help the business grow and have a bigger impact.

What kind of problems do Product Teams solve? 🤔

We are talking about very common and high-level businesss problems, such as:

How can we increase revenue?
How can we decrease churn rate?
How can we increase customer acquisition?
How can we increase customer retention?

The team is empowered with finding the best way to solve the problem at hand. It is not unusual for the team name to be a word which relates to these questions e.g. "Acqusition Team" or "Retention Team".

Phases of a Product Team

There are typically 2 phases that a product team can be found in at any given time and they are interchangeable.

Both are essential and require the entire team to give input.

1. Discovery phase 🔍

The team are collaborating and working through one of the problems with the goal of coming up with a physical feature or project. This is where real innovation comes in.

It is worth noting this is usually quite hard to do at home and relies on very good team relationships to do remotely.

2. Delivery phase 🚀

So once a feature or project has been chosen, this is the phase dedicated to writing the code and shipping it.

Unlike the discovery phase this functions very well remotely although good team communication is still vital to an efficient delivery.

The "Product Strategy" 📜

Dictating the problems a product team should solve is usually the "product strategy" (it is not essential though which is why companies often ignore it).

The "product strategy" takes the high level goals of the business (there might be alot) and helps FOCUS them into a couple of key points. Without this focus it is hard to know exactly which goal to spend resources on this quarter. Trying to hit 2 has a higher chance of succeeding than trying to hit 7, as the below quote I came across many years ago explains.

"The key to product is what you don't do"

Each strategy item usually has an OKR to help define what the objective and target is with it.

Types of engineers: Rockstars 🤘 vs Superstars 💫

Something I thought was worth touching on is my feelings on Rockstars vs Superstars. I generally find software engineers fall into one of those titles.

Superstars are usually good at most things and they like to try new things.

Rockstars are amazing at one thing in particular and usually do not like to try new things.

I generally find that Rockstars fit well with a Feature team, whereas Superstars fit better with a Product team. Although both can do equally well in either type, it all depends on whether the engineer enjoys being involved in solving problems rather than shipping features.

Summary

I am a big fan of the Product Team. Of having ownership over a commercial business goal and having to solve problems which will help move the business forward and ultimately exist outside of the tech world. I love writing code but I want the code I ship to have the maximum impact it can because developer time is not cheap (I do not mean that in an arrogant way but from a business resource perspective).

I think the question is not "Which type of team are you in?" but "Which type of team do you want to be working in??

I hope you found this article useful. Please email or tweet me if you have any thoughts about the Product vs Feature team topic.

Thanks

How the Feature Lead role finally helped me grow

Fri, 12 Jun 2020 00:00:00 GMT

I have been a senior engineer for around 7 years. I definitely was not ready for it when I was hired by the BBC, but it pushed me to grow into the role and I eventually did.

However the past 4 years or so I have really struggled to grow more. My knowledge has improved as the JS ecosystem around me was transformed from one built on jQuery to where we are now. But it has always been clear that this is not the same as genuine growth. By that I mean my role and impact in a given business, what I have to offer besides being told what code to write.

I started working at Nested almost 2 years ago and came across this idea of Feature Lead. It is not a new concept (having seen it first mentioned in 2012, but it’s likely older than that) but was the first time I had worked somewhere that was interested in using it.

At Nested we are impact-driven, by that I mean we aim for optimal ROI (return-on-investment). We try to quantifiably demonstrate that we are heading towards our goal by using KPIs (key performance indicators; low-level targets which help achieve a greater goal).

The result is that the leaders all the way up the chain have bought into this role, as the low level metrics now play a huge part in the overall success of the business. This has really helped push the Feature Lead role to be more prominent and rise within our company.

What is a Feature Lead? 🤔

For those who are not familiar here is my definition. Very early on in a feature’s life, literally when a high-level problem is being discussed, a Feature Lead is selected (or volunteered or whatever) to lead the project. I feel the purpose of the Feature Lead is to;

Plan and deliver the feature with a focus on (business) value

Essentially this involves doing more thinking up-front, and following up. Both are not crazy notions.

The 3 main activities it has to accomplish this are:

Plan and deliver the feature well i.e using the time and resources effectively
Consider the business impact (success criteria) and validate it (report on it via metrics)
Consider any next steps

The important aspect is that the Lead feels responsible for these activities and has ownership of them. They directly communicate and present on these to the stakeholders/product team/business.

It is less a process to follow and more a series of considerations to review and be mindful of (repeatedly at first).

Smaller features will not require as much following of the practices as larger features. However it’s often unclear beforehand how much value will come from the discovery, planning and thinking up-front.

What does this actually look like? 🎩

Let's break down what is involved and at what point of a project.

Planning 📝

Directly involved in early phases like inception, ideation/brainstorming, design etc. This should begin with discussing the problem at hand and potential solutions.
Deciding on a v1 i.e the smallest shippable thing bringing the most value (it doesn’t have to be small itself).
Exploring the unknowns as early as possible, validating assumptions before committing to pieces of work. This includes answering (or working with others to answer) any engineering architecture concerns which there might be.
Understand the business objectives the project has. What does success look like? Are you confident on how to report on its success (analytically using metrics if possible)? If not, spike to learn and discover. Ensure you will be able to report how the feature improves the business.

Managing Delivery 👷‍♀️

By that I mean 2 things:

Breaking down the feature into smaller parts i.e ticket spec’ing out. Being able to provide a high level estimate. I.e weeks vs months. If you have answered all the tough assumptions it should be relatively easy to get a rough idea about time-frame.
Mediating out the work. Communicating to managers and team members what needs to be done and in what order.

The likelihood is that you will work on at least a couple of the tickets yourself, so you will do some coding in amongst managing the delivery.

Validation ✔️

The after-care is also very important.

What I mean by this is ensuring we follow up and understand its impact.

For example “our new form didn’t do great on mobile, so we will iterate and try variations on mobile only”.

Days or weeks after a launch, review any metrics to learn how it is performing or if any hypothesis were proved.

Here we should consider how we want to surface these results and how long they need to last/be observed for.

Next steps ⏭️

This directly follows on from validation. Any learnings from the first version should be considered and applied to a potential second/next version. For instance:

If we added Y could we improve the engagement by 50%?
If we test different colour or text on the form could we get the conversion rate higher?

If enough value can be gauged from a second iteration of all the steps above, then it starts again.

Theoretically this process can repeat indefinitely as long as it is proving to add value.

So you as a Feature Lead are thinking about delivery and validation earlier, as well as having a more proactive part in decisions across the entire process.

This might also involve more managerial tasks such as:

Regularly reviewing the success criteria with stakeholders and communicating these to the team.
Collaborating with all disciplines across the business, from marketing to legal. Are the problems and your take on the solution all understood?
Communicating with stakeholders and (for the bigger features) sending out weekly update emails as well as release emails. This has proved very useful at sharing project information with the right people.

It’s worth noting you can be an engineer in 1 project, then a Feature Lead in another. It is a temporary role in that it will only last as long as the project.

However if it takes weeks or months to prove the success or learnings then it is the Lead’s responsibility to keep track of this. Essentially all the hard work was for this, so in many ways it is the most crucial part.

Caveats about this role 🙁

It is important to remember that it is not about the process, it is a series of considerations.

People have varying degrees of interest in following it which is fine, it should empower engineers not control them. We are not replacing one process with another. Encourage people to practice it and see the results themselves.

It has been very obvious how much having leaders who have bought into this has helped. They have pushed it from the top, which has had an incredibly positive effect on its adoption.

The results 🎇

Having done it several times now, I can honestly say it has changed my life (I’m not being over dramatic).

Before this started I had very little opinions about what was going on outside the code, but not anymore.

Each time I have led, I have found new challenges from which I learn a huge amount from. This is likely because they are not the typical problems facing an engineer. I am now aware of problems from well outside my normal domain. Not to mention knowing the first names of people outside engineering.

I now have a big passion for understanding and utilising analytics data and also it’s setup. The considerations for this might include:

Which tool can I use to answer our questions? Understanding what each tool does (e.g Google Analytics vs Mixpanel vs Server-side tool) and their limitations.
How long will this metric need to exist for? Do we need it to still be available in 6 months or can we get our answer via a 15 minute hack (e.g a spreadsheet) and then not worry about it ever again?

Focusing on the ROI of any work really changes how you view the work. You consider all the time spent on each part of it from a business perspective. No longer wanting to write code for the sake of it, you want it to have a purpose. I have a deeper appreciation for questioning and understanding the value a piece of work actually delivers. I think this might be the part which has helped change me more than any other. I might spend slightly less time coding but the time I do spend is much more productive.

"No longer wanting to write code for the sake of it, you want it to have a purpose"

My efficiency at helping to deliver a project has improved as I fully consider the heavier details at the beginning, so the delivery process is more streamlined. I never cease to appreciate the lack of stress had at the end of a project well planned. Not to mention the pleasure in meeting expectations.

I have learnt I can offer value to different areas of the company, as I have a better understanding how they work as well as their pain-points. This not only has a positive effect on myself but also on the business. Knowledge sharing has been a pleasant unexpected side-effect of this role.

For me personally it has helped identify (substantial) gaps in the business that I never would have been able to find previously. For example we had issues with analytics discrepancies (tools producing different results for reasons unknown). I was able to get involved fixing them, which was hugely valuable across the business. The result of this are learnings which are useful to me, to engineering but also to the wider business.

Summary 😎

I do my best to try and use the skills I have acquired to add value where I can, not just where I am Feature Leading. This typically surfaces in suggesting things and proactively seeking to ask or answer questions for any feature currently in progress. I am only able to do this as I am more knowledgeable in diverse areas. My actions also include spreading awareness and encouraging others to partake in this practice.

One of the best developers I ever knew said

"Engineers should deliver software not develop software"

At the time I did not know what this meant (should we focus on deployment perhaps?), but now I feel I have my own strong ideas. For me delivering software is more about the value it brings than the way it’s built.

I need to thank my company Nested for the culture that they have pushed to adopt. For empowering and not controlling their employees. Without this we would not have had space to experiment and learn, and I certainly would not have gone on this journey.

Please feel free to get in touch about your experiences, I would love to hear about them. Also comments or feedback are always welcomed and appreciated.

Thanks

Under-the-hood of test runners (e.g Mocha)

Mon, 08 Jun 2020 00:00:00 GMT

Test runners are a very important part to the modern JavaScript application. Without which we could not be able to run any tests at all. They are fairly straight forward to understand in terms of what they do, they run your tests and print output. However something which is not as straight forward is how they work and the mechanisms they utilise to enable the command-line interface that we are all so familiar with.

A video for this post can be found here. This is part of my "under-the-hood of" series:

I have been a member of the MochaJS core team for a little over 2 years so I feel I have a good understanding how the Mocha test runner works. This post aims to shine a light on how the Mocha test runner works and the codebase design, by walking through a slimmed down version of the tool. The real one is many thousands of lines of code so I have created a slimmed down version which can be found here. All of it has been taken from the real MochaJS codebase, however only the core functionality has been included. Hopefully this will help us focus on and learn about the individual aspects which are critical to its core functionality. It will be a mix of code snippets, links to mochas repository and links to my repository.

The article today will be broken down into 3 parts:

Introduction to our test runner
Building our runner: Parsing phase
Building our runner: Execution phase (the bulk of the article)

Before we move on here is some of the basic terminology for the core functionality of any test runner:

Test - Functions with an assertion/s inside
Suite - Collection of tests
Hook - Functions run at specific times in a test runners lifecycle
Reporter - Determines the presentation of the output
Interface - The methods that will be use inside each test e.g. describe and it (also known as UI)
Runner - A given run of a suite, tests, hooks, reporter etc (uses an instance of a Runnable)

1. Our test runner

Given that test runners can include a huge variation of functionality, not all of which is necessary for the basic job of the test runner, today we will be building our own test runner which includes:

Tests
Suites
Hooks
Interfaces
Reporters

This means we will be ignoring all other features, for example option validation, timeout management, slow test flagging, pending states etc.

Our runner output will look like this.

One test fails
Two tests in different suites pass
Second suite logs all hooks

Modules in MochaJS

It is necessary to understand the types of Modules that can be used in and are found in the MochaJS codebase, as we will be sticking to the same.

MochaJS uses CJS module inside the core, so it does not support ESM natively yet.
MochaJS accepts ESM for test files.

It is also worth noting that Mocha does not have a transpilation step (yet, plans are at work) so most of the code is ES5.

Our runners basic breakdown

There are 3 parts to our runner

1. The entities required for a test runner

This includes everything from above i.e Tests, Suites, Hooks, Reporters, Interfaces, Runner and Runnable.

2. The Parsing phase

This is the first of 2 steps for the runner. The goal with this phase is to build a coherent CLI for the user.

MochaJS makes use of yargs. You can see the starting point in lib/cli/cli.js. As it is ES5 it uses prototypes so it can use ES5 class instances.

3. Execution phase

The final part of the runner relies on a Mocha instance to be handed to it from the parsing phase. This part creates an instance of a Runnable and executes everything from the suites and tests to the hooks and reporters.

2. Parsing phase

Here we create a yars instance, it includes the basics required to have an intuitive command-line interface (e.g. asking for version information or help). We attach a commands object (see details below).

I have included in comments the location of the code in the real Mocha codebase.

// lib cli/cli.js main()
const argv = process.argv.slice(2)
// lib/cli/options loadOptions()
var args = yargsParser.detailed(argv).argv
args._ = Array.from(new Set(args._))

yargs()
  .scriptName("our_mocha")
  .command(commands.run)
  .fail((msg, err, yargs) => {
    yargs.showHelp()
    const message = msg || err.message
    console.error(`\nERROR: ${message}`)
    process.exit(1)
  })
  .help("help", "Show usage information & exit")
  .version("version", "Show version number & exit", "1.0")
  .epilog("Mocha Resources: ...")
  .config(args)
  .parse(args._)

Below details the commands object using a child yargs instance. It will set options and run validation checks (see methods options() and check())

Finally if it passes all these it will build a new Mocha instance (see new Mocha(argv)) and hand it to next phase (runMocha(mocha, argv))

const builder = yargs => {
  // cli/run.js builder()
  return yargs
    .options({
      config: {
        config: true,
        description: "Path to config file",
      },
      reporter: {
        default: defaults.reporter,
        description: "Specify reporter to use",
        requiresArg: true,
      },
      ui: {
        default: defaults.ui,
        description: "Specify user interface",
        requiresArg: true,
      },
    })
    .check(argv => {
      // lib/cli/run-helpers.js handleRequires
      // load --requires first, because it can impact "plugin" validation

      // lib/cli/run-helpers.js validatePlugin
      // validate `--reporter` and `--ui`.  Ensures there's only one, and asserts that it actually exists
      // Checks keys on Mocha.reporters + Mocha.interfaces

      return true
    })
}
const handler = async function(argv) {
  const mocha = new Mocha(argv)

  try {
    // NEXT PHASE
    await runMocha(mocha, argv)
  } catch (err) {
    console.error("\n" + (err.stack || `Error: ${err.message || err}`))
    process.exit(1)
  }
}
const commands = {
  run: {
    command: ["$0 [spec..]", "inspect"],
    describe: "Run tests with Our-Mocha",
    builder,
    handler,
  },
}

My code for the parser can be found here.

Building the Mocha instance

Let us look at what creating a new Mocha instance does by examining its constructor plus the ui and reporter instance methods.

function Mocha(options) {
  this.files = []
  this.options = options

  // lib/context.js. empty context
  function Context() {}
  // root suite
  this.suite = new Suite("", new Context(), true)

  this.ui(options.ui).reporter(options.reporter)
}
Mocha.prototype.ui = function(ui) {
  var bindInterface
  ui = ui || "bdd"
  bindInterface = Mocha.interfaces[ui]
  bindInterface(this.suite)

  return this
}
Mocha.prototype.reporter = function(reporter) {
  const builtinReporters = Mocha.reporters
  reporter = reporter || "spec"
  var _reporter
  // Try to load a built-in reporter.
  _reporter = builtinReporters[reporter]
  // Try to load reporters from process.cwd() and node_modules

  this._reporter = _reporter
  return this
}
Mocha.reporters = {
  base: Base, // imported from elsewhere
  spec: Spec,
}
Mocha.interfaces = {
  common: function() {},
  bdd: function() {},
}

So above:

Constructor: Creates instance of our Root Suite and attaches to this.suite (see this.suite = new Suite). Suite inherits pub/sub from EventEmitter.
ui method: Based on the interface we bind events, which attach interface methods, onto our suite context this.suite. See details on bindInterface below.
reporter method: Creates instance of a Reporter and attaches to this._reporter (see this._reporter = _reporter). An example of a reporter is the Spec function (see Mocha.reporters.spec)

How does the Interface work with our Mocha instance?

For our BDD interface it is set below, called from the ui() method (here) and initially the constructor.

bdd: function bddInterface(suite) {
  var suites = [suite]

  suite.on(EVENT_FILE_PRE_REQUIRE, function(context, file, mocha) {
    var common = Mocha.interfaces.common(suites, context, mocha)

    context.before = common.before
    context.after = common.after
    context.beforeEach = common.beforeEach
    context.afterEach = common.afterEach

    context.describe = function(title, fn) {
      return common.suite.create({
        title: title,
        file: file,
        fn: fn,
      })
    }

    context.it = function(title, fn) {
      var suite = suites[0]
      var test = new Test(title, fn)
      test.file = file
      suite.addTest(test)
      return test
    }
  })
}

Let us walk-through whats is going on.

Each time the event is fired (inside the execution phase coming later) the listener callback runs. The callback is given the params context, file and our mocha instance..

So based on an event (i.e. on("EVENT_FILE_PRE_REQUIRE") ) it attaches the interface methods, see context.after = ..., context.describe = ..., context.it = ..... Many of which come from a utility object called common.

Here the describe() method creates and returns a new Suite via suite.create(). The it() method creates and returns a new Test via new Test().

So thats it for the parsing phase, we now have our Mocha instance which has the reporter and the suite with the interface getting attached each time.

3. Execution phase

Now we have everything that we need to execute our test runner. We will be loading files and then running it.

1. Loading

Our parser calls runMocha inside the handler() (in parsing phase above).

async function runMocha(mocha, options) {
  const { spec = [] } = options

  // if options.watch watchRun()

  // singleRun
  // collectFiles and lookupFiles here
  mocha.files = spec
  await mocha.loadFilesAsync()
  return mocha.run(exitMochaLater)
}

We start by building an array of all the files to run. We pass them from the spec array.

We have a simplified version of this, but in the real mocha codebase there are many checks done against the spec items to check for globs, files and directories (see lib/cli/collect-files.js and lookupFiles).

Next we call loadFilesAsync. This loads the ESM and CJS files asynchronously, emits the "EVENT_FILE_PRE_REQUIRE" event and then runs the root suite.

Mocha.prototype.loadFilesAsync = async function() {
  var self = this
  var suite = this.suite

  for (let file of this.files) {
    // preload
    suite.emit(Suite.constants.EVENT_FILE_PRE_REQUIRE, global, file, self)
    // load
    file = path.resolve(file)
    const result = await require(file)
    // postload events
  }
}

The "EVENT_FILE_PRE_REQUIRE" event had the listener set in our bddInterface function above (here). So it is now that the interface methods are added to the suite context.

It is under esm-utils.js in the real mocha codebase.

Finally it runs the mocha.run() which moves us onto the running step.

2. Running

This phase is split into the following sections:

Mocha.run
Reporters
Runner.run
In-depth look at the code for runner.run
"Running step" summary

mocha.run

Mocha.prototype.run = function(fn) {
  var suite = this.suite
  var options = this.options
  options.files = this.files
  var runner = new Runner(suite, options.delay)
  createStatsCollector(runner)
  var reporter = new this._reporter(runner, options)

  const noop = () => ""
  function done(failures) {
    fn = fn || noop
    if (reporter.done) {
      reporter.done(failures, fn)
    } else {
      fn(failures)
    }
  }

  return runner.run(done)
}

Lets walk-through what it does

Create instance of a Runner (file here) and add stats collecting.

Similarly to the Suite, the Runner inherits from EventEmitter. This is for the publish and subscribe events functionality that Mocha relies on.

// lib/stats-collector.js
function createStatsCollector(runner) {
  var stats = {
    suites: 0,
    tests: 0,
    passes: 0,
    pending: 0,
    failures: 0,
  }

  if (!runner) {
    throw new TypeError("Missing runner argument")
  }

  runner.stats = stats

  runner.once(EVENT_RUN_BEGIN, function() {
    stats.start = new Date()
  })
  runner.on(EVENT_SUITE_BEGIN, function(suite) {
    suite.root || stats.suites++
  })
  runner.on(EVENT_TEST_PASS, function() {
    stats.passes++
  })
  runner.on(EVENT_TEST_FAIL, function() {
    stats.failures++
  })
  runner.on(EVENT_TEST_END, function() {
    stats.tests++
  })
  runner.once(EVENT_RUN_END, function() {
    stats.end = new Date()
    stats.duration = stats.end - stats.start
  })
}

Here is our stats collector. On pass/fail/end events increment pass/fail/end on the runner.stats object. In the real mocha codebase found lib/stats-collector.js.

Back to our Mocha.prototype.run. Next we create an instance of a Reporter using the Runner (see new this._reporter(runner, options))

Reporters

Inside the reporter we add spec-specific listeners onto the runner events. For example on EVENT_RUN_END even will run reporter.epilogue().

See below for a snippet from the spec reporter which includes events for

Run begin
Test pass
Test fail
Run end

function Spec(runner, options) {
  Base.call(this, runner, options)

  runner.on(EVENT_RUN_BEGIN, function() {
    Base.consoleLog()
  })

  // more ....

  runner.on(EVENT_TEST_PASS, function(test) {
    var fmt =
      indent() +
      color("checkmark", "  " + Base.symbols.ok) +
      color("pass", " %s") +
      color(test.speed, " (%dms)")
    Base.consoleLog(fmt, test.title, test.duration)
  })

  runner.on(EVENT_TEST_FAIL, function(test) {
    Base.consoleLog(indent() + color("fail", "  %d) %s"), ++n, test.title)
  })

  runner.once(EVENT_RUN_END, self.epilogue.bind(self))
}

So essentially as the runner is running our tests, the Spec reporter is printing output.

Below is the epilogue from the Base reporter. It processes the details on this.stats and once the tests are finished, outputs a final statement detailing the summary of the run.

Base.prototype.epilogue = function() {
  var stats = this.stats

  Base.consoleLog()

  // passes
  var fmt =
    color("bright pass", " ") +
    color("green", " %d passing") +
    color("light", " (%s)")

  Base.consoleLog(fmt, stats.passes || 0, milliseconds(stats.duration))

  // failures
  if (stats.failures) {
    fmt = color("fail", "  %d failing")

    Base.consoleLog(fmt, stats.failures)

    Base.list(this.failures)
  }
}

See the spec and base files for more.

Finally our mocha.run triggers return runner.run(done).

runner.run

Runner.prototype.run = function(fn) {
  var self = this
  var rootSuite = this.suite
  fn = fn || function() {}
  function start() {
    self.started = true
    self.emit(Runner.constants.EVENT_RUN_BEGIN)

    self.runSuite(rootSuite, function() {
      self.emit(Runner.constants.EVENT_RUN_END)
    })
  }
  // callback
  this.on(Runner.constants.EVENT_RUN_END, function() {
    fn(self.failures)
  })

  start()

  return this
}

So lets walk-through runner.run

Creates a listener for EVENT_RUN_END events
- It will run the fn callback with failure details
Emits an EVENT_RUN_BEGIN event
Executes each suite on Root suite, via runSuite
- This is very important; for each suite it will run all tests and hooks
- Once that is finished it emits anEVENT_RUN_END event

So not that crazy but we need to dig deeper into runSuite.

This is where all the suite, tests and hook are executed. To summarise how it works, starting with the Root Suite it:

Executes all before hooks for the suite
Then runs all tests for the suite
Then runs all after hooks for the suite
The moves onto the next suite.

If you are interested in more details, the next section looks at the code necessary to run the above. Alternatively skip to the summary section after this to see an overview.

In-depth look at the code for `runner.run`.

The total size of all the code required is quite large, so in an attempt to make it easier to follow the mechanics below I have included most of the required code for them.

The Runner methods we will cover are:

runSuite
hook
runTests
runTest
hooks

I will explain each one as we go along by doing both code comments and text outside the function. Use whichever you prefer.

Runner.runSuite

Runner.prototype.runSuite = function (suite, fn) {
  this.emit(EVENT_SUITE_BEGIN, (this.suite = suite))

  function done(errSuite) {
    // 6. run all afterAll hooks
    self.hook(HOOK_TYPE_AFTER_ALL, () => {
      self.emit(EVENT_SUITE_END, suite);
      fn(errSuite);
    });
  }

  function next() {
    curr = suite.suites[i++]; // grab next suite
    // 5. no suites left, done
    if (!curr) done();
    // 4. call suite
    runSuite(curr, next);
  }

  // 1. run all beforeAll hooks
  hook(HOOK_TYPE_BEFORE_ALL, () => {
    // 2. run tests + callback once complete
    runTests(suite, next)
  }
}

emit EVENT_SUITE_BEGIN
trigger the beforeAll hooks with a callback.

`hook` callback (after `beforeAll`)

trigger runTests handing suite and next callback.

`next` function

grab the next suite in the suites array
if no suite is left trigger done
else call itself via runSuite with the next suite.

`done` function

run all afterAll hooks
emit EVENT_SUITE_END event

Runner.hook

Runner.prototype.hook = function (name, fn) {
  // 1. for current suite, get hooks with name x
  var hooks = this.suite.getHooks(name);

  function next(i){
    // 3. grab current hook
    var hook = hooks[i];
    // 4. executed all hooks under name x for suite
    if (!hook) fn()

    // set hooks test context
    hook.ctx.currentTest = ...

    // 5. execute hook.
    hook.run(() => {
      // 6. end of hook trigger next hook
      next(++i)
    });
  }

  // 2. trigger start of hooks
  next(0);
}

Grab all hooks with the name given
trigger next callback with hook index

`next` function

using index grab the current hook
if no hook call fn
else set test context
lastly execute the hook (hook.run) with a callback

`hook.run` method

This comes from the Runnable.run method below as the hooks inherit Runnable.

Call the function with the current context.

Runnable.run(fn) {
  var ctx = this.ctx;
  fn.call(ctx);
}

`run` callback

increment the hook index and call next again (triggering the next hook under the given name)

Runner.runTests

Runner.prototype.runTests = function(callback) {
  function next() {
    // grab next test
    test = tests.shift()
    // no tests left, run callback running next suite
    if (!test) callback()
    // run beforeEach hooks
    hooks(HOOK_TYPE_BEFORE_EACH, () => {
      runTest(err => {
        if (err) {
          // test failure
          this.emit(EVENT_TEST_FAIL, test, err)
          self.emit(EVENT_TEST_END, test)
          // run after each hook
          return hooks(HOOK_TYPE_AFTER_EACH, next)
        }
        // test pass
        self.emit(EVENT_TEST_PASS, test)
        self.emit(EVENT_TEST_END, test)
        // run after each hook
        hooks(HOOK_TYPE_AFTER_EACH, next)
      })
    })
  }
  // run next test
  next()
}

trigger next callback

`next` function

grab the next test
if no tests are left trigger callback
else run beforeEach hooks

`hooks` callback

trigger runTest

`runTest` callback

if there is a test failure emit EVENT_TEST_FAIL event
else emit EVENT_TEST_PASS event
for both branches emit the EVENT_TEST_END event and run afterEach hook

Runner.runTest

Runner.prototype.runTest = function(fn) {
  var self = this
  var test = this.test

  if (!test) return

  try {
    // 1. run the test
    test.run(fn)
  } catch (err) {
    fn(err)
  }
}

early return if no test available
run the test via test.run
handle test errors

Runner.hooks

Runner.prototype.hooks = function(name, callback) {
  function next(suite) {
    if (!suite) callback()
    hook(name, () => {
      next(suites.pop()) // run hooks for next suite
    })
  }
  next()
}

call the next function

`next` function

if there are no suites left to run trigger the callback
trigger the hook for the given name
after that run hooks for the next suite

"Running step" summary

So essentially the core flow involved in the runner mentioned above is:

The Runner
- Attaches event listeners
- Updates stats property
- Triggers the Root suite
The Suite
- Executes the beforeAll hooks
- Begins execution of the tests
- Executes the afterAll hooks
- Executes the next suite
The Tests
- Executes the beforeEach hooks,
- Executes the tests
- Emits the test pass, fail and end events
- Handle any unexpected failures
- Executes the afterEach hooks
The Hooks
- Executes all the hooks under the given name
The Reporter
- Attaches event listeners
- Processes the stats
- Logs output to terminal/browser etc.

Thats it

So there you have it, the Mocha test runner broken down from the parsing to the execution.

Thanks so much for reading or watching. I hope you have found this article useful or at least interesting in some ways. You can find the repository for all this code here.

Thanks, Craig 😃

Introducing my under-the-hood of series

Thu, 28 May 2020 00:00:00 GMT

I am producing a blog and video series taking an indepth look at the mechanisms inside of different tools and libraries found in the JavaScript ecosystem. My approach is to start with an overview then build mini (~50 lines) working versions of them.

Topics covered so far

Future topics

npm/yarn - package managers
Blockchain
Ethereum
esbuild
styled-components
Linters (e.g Eslint)
Got a suggestion? Email me at craigtaub@gmail.com

Prefer watching a walk-through?

If you prefer watching a video walkthrough, I have an "Under-the-hood of" YouTube channel here

Under-the-hood of type systems (e.g. TypeScript)

Fri, 08 May 2020 00:00:00 GMT

I struggled to find an explanation from a low-level view of exactly how a "JavaScript type system compiler" is implemented. I understood many of the jobs of a type system but was unsure of the mechanisms involved and how they worked together.

This article looks to shine a light on some of the fundamentals at work under-the-hood. It is not possible to focus on everything in 1 article so here we will be looking at "type checks" specifically. Starting with an overview of type systems then building our own compiler which can run type checks and output sensible messages. For more help on transforms please see my article or talks on Web Bundlers or Source Maps.

This is part of my "under-the-hood of" series:

A video for this talk can be found here. Part of my "under-the-hood of" series here.

NOTE: Apologies as there is some repetition in this article with my previous article on Source Maps. But it is important to include all of the necessary information to understand the mechanisms we will be looking at today.

This article will be broken down into (click links to navigate):

PART A: Overview of type system compilers (including TypeScript) Syntax vs Semantics

What is AST?
Types of compilers
What does a language compiler do?
How does a language compiler work?
Type system compiler jobs
Advanced type checker features

PART B: Building our own type system compiler

The parser
The checker
Running our compiler
What have we missed?

Lets make a start 💪

PART A: Overview of type system compilers

Syntax vs Semantics

Something which is important to run over early is the difference between syntax and semantics.

Syntax

Is typically code which is native to JavaScript. Essentially asking if the given code is correct for the JavaScript runtime. For example the below is syntactically correct:

var foo: number = "not a number"

Semantics

This is code specific to the type system. Essentially asking if the given types attached to the code are correct. For example the above is syntactically correct BUT semantically wrong (defining the variable as a number but setting a string).

Next onto AST and compilers in the JavaScript ecosystem.

What is AST?

Before we go much further we need to take a quick look at one of the important mechanisms inside any JavaScript compiler, AST.

AST stands for "Abstract Syntax Tree", it is basically a tree of "Nodes" representing a Program of code. A "Node" is the smallest possible unit and is basically a POJO (i.e. plain old js object) with "type" and "location" properties. All Nodes have these 2 properties but based on the "type" they can have various other properties as well.

In AST form code is very easy to manipulate so operations like adding, removing or even replacing are do-able.

An example is the below code:

Would become the following AST:

There are websites such as https://astexplorer.net/ which are great at letting you write JavaScript code and immediately see its AST.

Types of compilers

There are 2 main types of compilers in the JavaScript ecosystem

1. Native compiler

A native compiler will convert code into a form that can be run by a server or computer (i.e. machine code). A compiler such as the one found in the Java ecosystem converts code into bytecode and then into native machine code.

2. Language compiler

A language compiler has a quite different role. The compilers for TypeScript and Flow both count in the category as language compilers as they output code into JavaScript. The main difference with native compilers is that they compile for tooling-sake (e.g. optimising code performance or adding additional features) not to produce machine code.

What does a language compiler do?

Let us start with the basics. A couple of the core jobs found in a type system compiler are:

1. Performing type checks

By this I mean the introduction of "types" (often via explicit annotations or implicit inference) and a way to check that 1 type matches another e.g. string vs number.

2. Running a language server

For a type system to work in a development environment it is best if it can run any type checks in an IDE and provide instant feedback for the user. Language servers connect a type system to an IDE, they can run the compiler in the background and re-run when a user saves a file. Popular languages such as TypeScript and Flow both contain a language server.

3. Transforming code

Many type systems contain code which is not supported in native Javascript (e.g. type annotations are not supported) so they must transform from unsupported JavaScript to supported JavaScript.

As mentioned at the very top we will be focusing on point (1) Performing type checks. If it seems valuable we could explore (2) language servers in the future. My articles on Web Bundlers and Source Maps go into more detail about (3) transforming code.

How does a language compiler work?

Next we will have a look at the steps required to perform all of the above jobs in an efficient and scalable way. There are 3 common stages to most compilers in some form or another.

1) Parse source code into AST

Lexical analysis -> Turn string of code into a stream (i.e. an array) of tokens.
Syntactic analysis -> Turn stream of tokens into its AST representation

Parsers check the "syntax" of given code. A type system will have to house its own parser, often containing thousands of lines of code.

The Babel parser contains 2,100 lines of code just to process code statements (see it here) which can understand the syntactical analysis of any compiler-specific code but also append additional information for types.

Hegel appends a typeAnnotation property to code which has a type annotation (you can see it doing it here).

TypeScript's parser is a whopping 8,600 lines of code (find where it begins traversing the tree here). It houses an entire superset of JavaScript which all requires the parser to understand.

2) Transform nodes on AST

Manipulate AST nodes

Here any transformations to apply to the AST are performed.

3) Generate source code

Turn AST into string of JavaScript source code

A type system has to map any non-JS compliant AST back to native JavaScript.

But how does a type system fit into that?

Type System compiler jobs

As well as the above steps, Type system compilers will usually include an additional step or 2 after "parsing" which will include the type-specific work.

On a side-note TypeScript actually has a total of 5 phases in its compiler, they are:

Language server pre-processor
Parser
Binder
Checker
Emitter

As you can see above the language server contains a pre-processor which triggers the type compiler to only run over the file/s which has changed. This will follow any "import" statements to determine what else could have changed and would need to be included in the next re-run. Additionally the compiler has the ability to only re-process the branch of the AST graph which has changed. More on "lazy compilation" below.

There are 2 common jobs to the type system compiler:

1. Inferring

Inferring is required for code which does not have an annotation. On that subject there is a really interesting blog post here on when to use type annotations and when to let the engine use Inference.

Using a pre-defined algorithm the engine will calculate what the type for a given variable/function is.

TypeScript uses the algorithm "best common type" inside of its Binding phase (the 1st of 2 semantic passes). It considers each candidate type and picks the type that is compatible with all the other candidates. Contextual typing comes into play here i.e. using the location in the inference. There is more help on this in the TypeScript spec here. TypeScript actually introduces the idea of "Symbols" (interface here) these are named declarations which connect declaration nodes in the AST to other declarations contributing to the same entity. They are the basic building block of the TypeScript Semantic system

2. Checking

Now that (1) is complete and types have been assigned the engine can run its type checks. They check the "semantics" of the given code. There are many flavours of these types of checks ranging from type mis-match to type non-existing.

For TypeScript this is the Checker (the 2nd semantic pass) and it is 20,000 lines of code long. I feel that gives a really strong idea of just how complicated and difficult it is to check so many different types across so many different scenarios.

The type checker is NOT dependent on calling code i.e. if the file executes any of its own code (i.e. at runtime) . The type checker will process each line in a given file itself and run the appropriate checks.

Advanced type checker features

A couple of additional concepts which we will not dig into any deeper today due to the complexity they entail:

Lazy compilation

A common feature for modern compilations is "lazy loading". They will not recalculate or re-compile a file or AST branch unless absolutely required.

TypeScripts pre-processor can use AST code which is stored in memory from a previous run. This has a massive performance boost as it can just focus on running over a small part of the program or node tree which has changed. TypeScript uses immutable read-only data-structures stored in what it terms "look-aside tables". This makes it easy to know what has/has not changed.

Soundness

There are certain operations which a compiler can not know is safe at compile-time and must wait for run-time. Each compiler must make difficult choices as to what will and will not be included. TypeScript has certain areas which are said to "not to be sound" (i.e. require run-time type checks).

We will not be addressing the above features in our compiler as they add additional complexity and not worth it for our small POC.

Onto more exciting stuff now, building one ourselves…

PART B: Building our own type system compiler

We are going to build a compiler which can run type checks for 3 different scenarios and throw a specific message for each. The reason we will limit it to 3 scenarios is so we can focus on the specific mechanisms in work around each one, and hopefully by the end have a really strong idea on how to introduce more complex type checks.

We will be working with a function declaration and an expression (calling that function) in our compiler.

The scenarios are:

(1) Issue with type matching a string vs a number

fn("craig-string") // throw with string vs number
function fn(a: number) {}

(2) Issue with using an unknown type which is not defined

fn("craig-string") // throw with string vs ?
function fn(a: made_up_type) {} // throw with bad type

(3) Issue with using a property name not found on the interface

interface Person {
  name: string;
}
fn({ nam: "craig" }) // throw with "nam" vs "name"
function fn(a: Person) {}

Onto our compiler, there are 2 parts to our compilers, the parser and the checker.

The Parser

As previously mentioned we won't be focusing on a parser today. We will be following the Hegel parsing approach of assuming a typeAnnotation object has been attached to all annotated AST nodes. I have hardcoded the AST objects.

Scenario 1 will use the below parser:

function parser(code) {
  // fn("craig-string");
  const expressionAst = {
    type: "ExpressionStatement",
    expression: {
      type: "CallExpression",
      callee: {
        type: "Identifier",
        name: "fn",
      },
      arguments: [
        {
          type: "StringLiteral", // Parser "Inference" for type.
          value: "craig-string",
        },
      ],
    },
  }

  // function fn(a: number) {}
  const declarationAst = {
    type: "FunctionDeclaration",
    id: {
      type: "Identifier",
      name: "fn",
    },
    params: [
      {
        type: "Identifier",
        name: "a",
        typeAnnotation: {
          // our only type annotation
          type: "TypeAnnotation",
          typeAnnotation: {
            type: "NumberTypeAnnotation",
          },
        },
      },
    ],
    body: {
      type: "BlockStatement",
      body: [], // "body" === block/line of code. Ours is empty
    },
  }

  const programAst = {
    type: "File",
    program: {
      type: "Program",
      body: [expressionAst, declarationAst],
    },
  }
  // normal AST except with typeAnnotations on
  return programAst
}

You can see the expressionAstblock for our top-line expression statement, and the declarationAst for where we have declared our function on the second line. We return a programAst which is a program with both AST blocks in.

Inside the AST you can see the typeAnnotation the param identifier "a", matching where it sits in the code.

Scenario 2 will use the below parser:

function parser(code) {
  // fn("craig-string");
  const expressionAst = {
    type: "ExpressionStatement",
    expression: {
      type: "CallExpression",
      callee: {
        type: "Identifier",
        name: "fn",
      },
      arguments: [
        {
          type: "StringLiteral", // Parser "Inference" for type.
          value: "craig-string",
        },
      ],
    },
  }

  // function fn(a: made_up_type) {}
  const declarationAst = {
    type: "FunctionDeclaration",
    id: {
      type: "Identifier",
      name: "fn",
    },
    params: [
      {
        type: "Identifier",
        name: "a",
        typeAnnotation: {
          // our only type annotation
          type: "TypeAnnotation",
          typeAnnotation: {
            type: "made_up_type", // BREAKS
          },
        },
      },
    ],
    body: {
      type: "BlockStatement",
      body: [], // "body" === block/line of code. Ours is empty
    },
  }

  const programAst = {
    type: "File",
    program: {
      type: "Program",
      body: [expressionAst, declarationAst],
    },
  }
  // normal AST except with typeAnnotations on
  return programAst
}

It is very similar to Scenario 1 with its expression, declaration and program AST blocks. However the difference is the typeAnnotation inside params is made_up_type instead of what scenario 1 has which is NumberTypeAnnotation.

Scenario 3 will use the below parser:

function parser(code) {
  // interface Person {
  //   name: string;
  // }
  const interfaceAst = {
    type: "InterfaceDeclaration",
    id: {
      type: "Identifier",
      name: "Person",
    },
    body: {
      type: "ObjectTypeAnnotation",
      properties: [
        {
          type: "ObjectTypeProperty",
          key: {
            type: "Identifier",
            name: "name",
          },
          kind: "init",
          method: false,
          value: {
            type: "StringTypeAnnotation",
          },
        },
      ],
    },
  }

  // fn({nam: "craig"});
  const expressionAst = {
    type: "ExpressionStatement",
    expression: {
      type: "CallExpression",
      callee: {
        type: "Identifier",
        name: "fn",
      },
      arguments: [
        {
          type: "ObjectExpression",
          properties: [
            {
              type: "ObjectProperty",
              method: false,
              key: {
                type: "Identifier",
                name: "nam",
              },
              value: {
                type: "StringLiteral",
                value: "craig",
              },
            },
          ],
        },
      ],
    },
  }

  // function fn(a: Person) {}
  const declarationAst = {
    type: "FunctionDeclaration",
    id: {
      type: "Identifier",
      name: "fn",
    },
    params: [
      {
        type: "Identifier",
        name: "a",
        typeAnnotation: {
          type: "TypeAnnotation",
          typeAnnotation: {
            type: "GenericTypeAnnotation",
            id: {
              type: "Identifier",
              name: "Person",
            },
          },
        },
      },
    ],
    body: {
      type: "BlockStatement",
      body: [], // Empty function
    },
  }

  const programAst = {
    type: "File",
    program: {
      type: "Program",
      body: [interfaceAst, expressionAst, declarationAst],
    },
  }
  // normal AST except with typeAnnotations on
  return programAst
}

As well as the expression, declaration and program AST blocks there is also an interfaceAst block which holds the AST for our InterfaceDeclaration. The declarationAst now has a GenericType on its annotation as it takes an object identifier i.e. Person. The programAst will return an array of those 3 objects for this scenario.

Similarities in the parsers

As you can see from above, the main area which holds the type annotation for all 3 scenarios is the declaration param. All 3 have that in common.

The Checker

Now onto the part of the compiler which does our type checks. It needs to iterate through all the program body AST objects and depending on the node type do the appropriate type checks. We will add any errors onto an array to be returned to the caller for printing.

Before we go any further, the basic logic we will work with for each type is:

Function declaration: check the types for the argument are valid, then check each statement in the block body
Expression: find the function declaration for the caller, grab the type on the declarations argument, lastly grab the type of the expressions caller argument and compare them.

The code

This snippet contains the typeChecks object (and errors array) which will be used to check our expression and a basic annotation check.

const errors = []

const ANNOTATED_TYPES = {
  NumberTypeAnnotation: "number",
  GenericTypeAnnotation: true,
}

// Logic for type checks
const typeChecks = {
  expression: (declarationFullType, callerFullArg) => {
    switch (declarationFullType.typeAnnotation.type) {
      case "NumberTypeAnnotation":
        return callerFullArg.type === "NumericLiteral"
      case "GenericTypeAnnotation": // non-native
        // If called with Object, check properties
        if (callerFullArg.type === "ObjectExpression") {
          // Get Interface
          const interfaceNode = ast.program.body.find(
            node => node.type === "InterfaceDeclaration"
          )
          // Get properties
          const properties = interfaceNode.body.properties

          // Check each property against caller
          properties.map((prop, index) => {
            const name = prop.key.name
            const associatedName = callerFullArg.properties[index].key.name
            if (name !== associatedName) {
              errors.push(
                `Property "${associatedName}" does not exist on interface "${interfaceNode.id.name}". Did you mean Property "${name}"?`
              )
            }
          })
        }
        return true // as already logged
    }
  },
  annotationCheck: arg => {
    return !!ANNOTATED_TYPES[arg]
  },
}

Let us walk through the code. Our expression has 2 types of checks:

For NumberTypeAnnotation; the caller type should be a NumericLiteral (i.e. if annotated as a number, the caller type should be a number). Scenario 1 would fail here but nothing is logged yet.
For GenericTypeAnnotation; if it is an object we search the tree for an InterfaceDeclaration and then check each property of the caller on that interface. Any issues get pushed onto the errors array, with a helpful message about what property name does exist and therefore what it could actually be. Scenario 3 would fail here and get this error.

Our processing is limited to this file, however most type checkers have the notion of "scope" so they would be able to determine if a declaration was anywhere in the runtime. Ours has an easier job as it is just a POC.

This snippet contains the processing of each node type in the program body. This is where the type check logic above is called from.

// Process program
ast.program.body.map(stnmt => {
  switch (stnmt.type) {
    case "FunctionDeclaration":
      stnmt.params.map(arg => {
        // Does arg has a type annotation?
        if (arg.typeAnnotation) {
          const argType = arg.typeAnnotation.typeAnnotation.type
          // Is type annotation valid
          const isValid = typeChecks.annotationCheck(argType)
          if (!isValid) {
            errors.push(
              `Type "${argType}" for argument "${arg.name}" does not exist`
            )
          }
        }
      })

      // Process function "block" code here
      stnmt.body.body.map(line => {
        // Ours has none
      })

      return
    case "ExpressionStatement":
      const functionCalled = stnmt.expression.callee.name
      const declationForName = ast.program.body.find(
        node =>
          node.type === "FunctionDeclaration" && node.id.name === functionCalled
      )

      // Get declaration
      if (!declationForName) {
        errors.push(`Function "${functionCalled}" does not exist`)
        return
      }

      // Array of arg-to-type. e.g. 0 = NumberTypeAnnotation
      const argTypeMap = declationForName.params.map(param => {
        if (param.typeAnnotation) {
          return param.typeAnnotation
        }
      })

      // Check exp caller "arg type" with declaration "arg type"
      stnmt.expression.arguments.map((arg, index) => {
        const declarationType = argTypeMap[index].typeAnnotation.type
        const callerType = arg.type
        const callerValue = arg.value

        // Declaration annotation more important here
        const isValid = typeChecks.expression(
          argTypeMap[index], // declaration details
          arg // caller details
        )

        if (!isValid) {
          const annotatedType = ANNOTATED_TYPES[declarationType]
          // Show values to user, more explanatory than types
          errors.push(
            `Type "${callerValue}" is incompatible with "${annotatedType}"`
          )
        }
      })

      return
  }
})

Let us walk through the code again, breaking it down by type.

FunctionDeclaration (i.e. function hello() { })

Start by processing the arguments/params. If you find a type annotation check if the type exists for the argument given i.e. argType. If it does not add an error to errors. Scenario 2 would get an error here.

Lastly we process the function body, however as we know there is no function body to process I have left it blank.

ExpressionStatement (i.e. hello())

First check the program body for the declaration of the function. This is where scope would apply to a real type checker. If no declaration is found add an error to the errors array.

Next we check each defined argument type against the caller argument type. If there is a type mismatch found then add an error onto the errors array. Both Scenario 1 and Scenario 2 will get this error.

Running our compiler

I have introduced a basic repository with a simple index file which processes all 3 AST node objects in 1 go and logs the errors. When I run it I get the below:

So to summarise:

Scenario 1

We defined the argument type of number, yet called it with a string

Scenario 2

We defined a type on the function argument which does not exist and then we called our function, so we get 2 errors (1 for bad type defined, 1 for type mismatch)

Scenario 3

We defined an interface but used a property called nam which was not on the object, we are asked if we meant to use name instead.

It works !!! Good job 👍💪

What have we missed?

As mentioned there are many additional parts to a type compiler which we have omitted from our compiler. Some of those are:

The parser: we manually wrote the AST blocks these would be generated on a real type compiler
Pre-processing/language compiler: A real compiler has mechanisms to plug into the IDE and re-run at appropriate times
Lazy compilation: No intelligence around what has changed or using memory
Transform: we have skipped the final part of the compiler which is where the native JavaScript code is generated.
Scope: as our POC is a single file it did not need to understand the notion of "scope" however real compiler have to always be aware of context.

Thanks so much for reading or watching, I learnt a huge amount about type systems from this research and I hope it was useful for you. You can find the repository for all this code here.

Thanks, Craig 😃

Source maps from top to bottom

Wed, 29 Apr 2020 00:00:00 GMT

This article covers a basic overview and an in-depth explanation of the mechanics of JS code generators and source maps. Building our own from scratch, hence why it is quite long.

Source Maps are something of a mystery to most people. They are found in most web-based compilation scenarios; from type system to web bundlers. But often the details of how they are actually built is not 100% transparent as their usage alone can be complex enough. Today we will start with a brief overview of what they are and how to use them. Then move onto walking through the lower-level mechanics; by building our own compiler which will generate some code and produce its own Source Map for browser consumption.

This is part of my "under-the-hood of" series:

A full video walkthrough for this post is here. Part of my “under-the-hood of” video series.

The plan for today:

Part A: Intro to source maps and compilers

What is a source map and why are they useful?
Using source maps with popular tools
What is AST?
Steps in transforming JavaScript
How compilers build source maps

Part B: Building our own compiler

Building a JavaScript code generator
What is Base64 VLQ?
Adding source map support
Testing our source map

Lets get started 💪

Part A: Intro to source maps and compilers

1. What is a source map and why are they useful?

First let us look at some reason why people write JavaScript which needs transpilation into native JavaScript:

Using a type system,
Using the latest ES 8–9–10 features,
Code optimising (e.g minification)
Bundle optimising (e.g. vendor vs app bundles)

The modern compiler architecture looks like this:

The problem is that in the process of turning your modules into assets, the code itself becomes un-readable for humans as its emphasis is put on the computer (usually a browser or server) to read.

Some example code which has been put through a compiler is below. It becomes near impossible to reason about and even harder to debug if there is an issue.

This is where Source Maps come in !!

A basic definition for a Source Map would be:

“A source map provides a way of mapping code within a compressed file back to it’s original position in a source file”

So fairly straight forward in its purpose. Modern browsers will parse the source map automatically and make it appear as though you’re running un-minified or uncombined files.

An example below shows debugging TypeScript in the browser, something which is only possible due to Source Maps.

With this you could now put a breakpoint in your code and inside the browser inspect the call stack, variables and any runtime state, all via the pre-compiled TypeScript code.

2. Using source maps with popular tools

Connecting the browser to the source map

There are 2 ways to notify the Browser there is a source map available.

At the footer of a JavaScript file add

//# sourceMappingURL=/path/to/file.js.map

In the “Header” of a JavaScript file add

X-SourceMap: /path/to/file.js.map

Couple of points to note

Chrome will only download a source map if DevTools is open (as they can be huge in size)
Source maps will not appear as a network request (inside the network tab)
Once you have your source map you can add breakpoints inside the “source” code (found under sources tab).

Source map spec

Current source maps must follow the latest version of the source map spec. That is version 3 and can be found in full here, it is written by mostly Mozilla and Google engineers. Version 3 comes with improvements to its overall size that will speed up its downloading and parsing.

The below shows an example source map, an important point is the “mappings”, these are Base64 VLQ strings which contain the actual mappings from source to generated code. More on this later as we will be producing our own.

Usage in popular tools:

Node.js

Via the flag — enable-source-maps

Source maps cached and used for stack traces when an exception occurs.

Babel

By default Babel will add a source map location to the bottom of every generated bundle e.g.

//# sourceMappingURL=file.map.js

But via the flag — source-maps — inline you can tell Babel to use an inline source map, which looks like below (i.e. base64 encode string of contents).

//# sourceMappingURL=data:application/json;charset=utf-8;base64,....

Webpack

Via the config property devtool: ‘source-map’

It is worth noting that because tools like Webpack often carry out multiple transformations using many processors at once (e.g. Babel and TypeScript) it can still produce a single source map. Each processor will produce its own source map but there are libraries available which can concatenate JavaScript files while consolidating corresponding source map files. An example is called mapcat.

3. What is AST?

Before we go much further we need to take a quick look at one of the important mechanisms inside any JavaScript compiler, AST.

AST stands for “Abstract Syntax Tree”, it is basically a tree of “Nodes” representing a Program of code. A “Node” is the smallest possible unit and is basically a POJO (i.e. plain old js object) with “type” and “location” properties. All Nodes have these 2 properties, but based on the “type” they can have various other properties as well.

In AST form code is very easy to manipulate, so operations like adding, removing or even replacing.

An example is the below code:

Would become the following AST:

There are websites such as https://astexplorer.net which are great at letting you write JavaScript code and immediately see its AST.

Tree traversal

The most important part of processing the AST is understanding that there are different approaches and each has pros and cons.

An example of a popular type (and the type we will use today) is called “Depth-first search” and it works by starting at the root and exploring as far left as possible along each branch before backtracking. So it would process a tree in the below order:

So if we had a block of code such as

2 + 3 * 1

It would produce the following tree:

4. Steps in transforming JavaScript

There are 3 steps in transforming JavaScript:

1) Parse source code into AST

Lexical analysis -> Turn string of code into a stream (i.e. an array) of tokens.
Syntactic analysis -> Turn stream of tokens into its AST representation

2) Transform nodes on AST

Manipulate AST nodes (any library plugins would operate here e.g. Babel)

3) Generate source code

Turn AST into string of JavaScript source code

TODAY we are going to focus on the jobs of a generator !!

Libraries differ between doing just step 1 to doing all 3 steps.

Examples of libraries which do all 3:

Examples of libraries which do just 1:

Esprima (does parsing)
ast-types (does AST node manipulation)
Escodegen (does generation)

5. How compilers build source maps

There are 3 parts to producing a source map, all of which a compiler has to do:

Transform code and note the new generated source location
Check for a difference in location between the original and generated code
Using these mapping build a source map

This is an over-simplification and we will get more into the nuts and bolts of it in Part B below.

Part B: Building our own compiler

1. Building a JavaScript code generator

We will be starting with the below architecture. With the aim to produce a transformed file (index.es5.js) and source map (index.es5.js.map) after compilation.

Our src/index.es6.js will look like this (a simple “add” function):

function add(number) {
  return number + 1
}
globalThis.add = add

So now we have our pre-compiled source code. We want to start looking at the compiler.

THE PROCESS

There are a couple of steps our compiler must perform:

1. Parse the code to AST

As this article is not focusing on parsing, we will use a basic 3rd party tool for this (esprima or escodegen)

2. Add a shallow clone of each node onto the AST

This idea was borrowed from recast. The idea is that each Node will hold itself as well as a clone of itself (i.e. the original). The clone is used to check if the Node has changed. More about this later.

3. Transformation

We will manually be doing this. We could have used a library such as ast-types or @babel/types as they have useful APIs.

4. Generate source code

Turn our AST into JavaScript.

5. Add source map support

4 and 5 are done at the same time as above. This will involve traversing the tree and detecting where the AST node has changed with its “original” property. For those instances store a mapping between the “original” and the “generated” code.

6. Write to build/

Finally write our generated source code and its source map to the appropriate file.

THE CODE

Let us look at these steps again, but this time in more detail.

1. Parse the code to AST

Using a basic 3rd party tool (I went for a simple one called ast), we grab our file contents and pass them into the libraries parser.

import fs from "fs"
import path from "path"
import ast from "abstract-syntax-tree"

const file = "./src/index.es6.js"
const fullPath = path.resolve(file)
const fileContents = fs.readFileSync(fullPath, "utf8")
const sourceAst = ast.parse(fileContents, { loc: true })

2. Add a shallow clone of each node onto the AST

First we define a function called “visit” with the job of traversing the tree and executing our callback function on every single Node.

export function visit(ast, callback) {
  callback(ast)

  const keys = Object.keys(ast)
  for (let i = 0; i < keys.length; i++) {
    const keyName = keys[i]
    const child = ast[keyName]
    if (keyName === "loc") return
    if (Array.isArray(child)) {
      for (let j = 0; j < child.length; j++) {
        visit(child[j], callback)
      }
    } else if (isNode(child)) {
      visit(child, callback)
    }
  }
}
function isNode(node) {
  return typeof node === "object" && node.type
}

Here we are doing a “depth-first search” as mentioned above. For a given Node it will:

Execute the callback
Check for the location property, if so return early
Check for any properties which are arrays, if so call itself with each child
Check For any properties which are AST Nodes, if so call itself with the node.

Next we move onto producing our clones.

export const cloneOriginalOnAst = ast => {
  visit(ast, node => {
    const clone = Object.assign({}, node)
    node.original = clone
  })
}

Our cloneOriginalAst function produces a clone of the Node and appends that onto the original.

For our cloning we use Object.assign so it is a shallow clone and copies the top-level properties. The nested props are still connected by pass-by-reference i.e. changing them will change the clone. We could have also used the spread operator here as that does the same thing. We will do our comparison using the top-level which is enough to compare 2 AST nodes and determine if the node has changed or not.

Overall our code here will return the same tree except with “original” property on every single Node.

3. Transformation

Next we will do our node manipulation. We will keep it simple so are going to just swap 2 nodes from our program. So we will start with:

number + 1

And will end with:

1 + number

Simple in theory right !

Our code to do the swap is below:

// Swap: "number + 1"
// - clone left node
const leftClone = Object.assign(
  {},
  sourceAst.body[0].body.body[0].argument.left
)
// - replace left node with right node
sourceAst.body[0].body.body[0].argument.left =
  sourceAst.body[0].body.body[0].argument.right
// - replace right node with left clone
sourceAst.body[0].body.body[0].argument.right = leftClone
// Now: "1 + number". Note: loc is wrong

We have not used a clean API to do this (which many libraries provide) as we have manually swapped the 2 nodes.

An example of using a library with a helpful API could look something like below, provided by the documentation on ast-types.

This way is certainly safer, easier to follow and faster to develop with. So in general I would recommend using it for any complex AST manipulation, most big name compilers do.

4. Generate source code

Code generators are typically housed in a single file and are several thousand lines long. For example escodegen’s compiler is 2,619 lines (see here). That is on the smaller side compared to others (crazy right!)

I have used much of the same code for our compiler (as most generators need very similar logic to process AST into JavaScript) EXCEPT only what is absolutely necessary for us to process the code from our “index.es6.js” file.

Below I have defined the 3 types of code we have inside our compiler.

a) Node processors and character utilities

These are general utility functions used to process AST nodes (depending on the type e.g. a function declaration will have an identifier) and build source code. It also includes some common character constants (e.g. a “space”). They are called from our code “type statements” in the next section.

I would not worry too much about the details here unless you plan on writing a compiler. This was largely borrowed from the generator in escodegen here.

// Common characters
const space = " "
const indent = space + space
const newline = "\n"
const semicolon = ";" // USUALLY flags on this

// Utility functions
function parenthesize(text, current, should) {
  if (current < should) {
    return ["(", text, ")"]
  }
  return text
}
const generateAssignment = (left, right, operator, precedence) => {
  const expression = [
    generateExpression(left),
    space + operator + space,
    generateExpression(right),
  ]
  return parenthesize(expression, 1, precedence).flat() // FLATTEN
}
const generateIdentifier = id => {
  return id.name
}
const generateFunctionParams = node => {
  const result = []
  result.push("(")
  result.push(node.params[0].name) // USUALLY lots of logic to grab param name
  result.push(")")
  return result
}
const generateStatement = node => {
  const result = Statements[node.type](node)
  return result
}
const generateFunctionBody = node => {
  const result = generateFunctionParams(node)
  return result.concat(generateStatement(node.body)) // if block generateStatement
}
const generateExpression = node => {
  const result = Statements[node.type](node)
  return result
}

b) Type statements

This is an object holding functions which are tied to an AST node type. Each contains the logic necessary to process that AST node type and produce source code. For example for a function declaration it contains all possible variations of arguments, identifiers, logic and return types. There is a level of recursion that is common here i.e. for a type statement to trigger another type statement which might trigger another etc.

Here we ONLY have the necessary statement functions to process our “index.es6.js” file, so it is fairly limited. You can see how much code is required just to process our AST tree of 3–4 lines of code (in addition to that of the above section).

Again this has borrowed from escodegen here so please feel free to ignore the details, unless you plan to write your own compiler.

const Statements = {
  FunctionDeclaration: function(node) {
    let id
    if (node.id) {
      id = generateIdentifier(node.id)
    } else {
      id = ""
    }
    const body = generateFunctionBody(node)
    return ["function", space, id].concat(body) // JOIN
  },
  BlockStatement: function(node) {
    let result = ["{", newline]
    // USUALLY withIndent OR for loop on body OR addIndent
    result = result.concat(generateStatement(node.body[0])).flat()
    result.push("}")
    result.push("\n")
    return result
  },
  ReturnStatement: function(node) {
    // USUALLY check for argument else return
    return [
      indent,
      "return",
      space,
      generateExpression(node.argument),
      semicolon,
      newline,
    ]
  },
  BinaryExpression: function(node) {
    const left = generateExpression(node.left)
    const right = generateExpression(node.right)
    return [left, space, node.operator, space, right]
  },
  Literal: function(node) {
    if (node.value === null) {
      return "null"
    }
    if (typeof node.value === "boolean") {
      return node.value ? "true" : "false"
    }
    return node.value
  },
  Identifier: function(node) {
    return generateIdentifier(node)
  },
  ExpressionStatement: function(node) {
    const result = generateExpression(node.expression) // was []
    result.push(";")
    return result
  },
  AssignmentExpression: function(node, precedence) {
    return generateAssignment(node.left, node.right, node.operator, precedence)
  },
  MemberExpression: function(node, precedence) {
    const result = [generateExpression(node.object)]
    result.push(".")
    result.push(generateIdentifier(node.property))
    return parenthesize(result, 19, precedence)
  },
}

c) Process code statements

Lastly we are going to iterate over the program body (i.e. each line of code) and start running our generator. This will now return an array called “code” which contains every line of our newly generated source code.

const code = ast.body.map(astBody => Statements[astBody.type](astBody)).flat()

6. Write to build/

We are going to skip step 5 for now and complete the core elements of our compiler. So for this step we will

Add a source map location to our generated code (we will build this in the next section)
Produce a bundle for the generated code (joining our array of code together), and copy the original code so that the browser can see it (this is only 1 approach to this).

// Add sourcemap location
code.push("\n")
code.push("//# sourceMappingURL=/static/index.es5.js.map")

// Write our generated and original
fs.writeFileSync(`./build/index.es5.js`, code.join(""), "utf8")
fs.writeFileSync(`./build/index.es6.js`, fileContents, "utf8")

5. Add source map support

There are 4 requirements when it comes to building a source map:

Store record of source file
Store record of generated file
Store mappings of line/columns
Display in Source Map file using spec version3

For a quick win we can use the library which almost every JavaScript code generator uses called source-map. It is from Mozilla and handles storing of points 1–3 as well as the processing the mappings into Base64 VLQ (step 4).

Little reminder what a source map looks like with mappings highlighted (from way above):

The mappings are Base64 VLQ, but what is that?

2. What is Base64 VLQ?

A brief overview on Base64 and VLQ first.

Base64

Solves the ASCII problem of handling languages which don’t have the full ASCII character set. Base64 has only a subset of ASCII, which is easier for processing in different languages.

VLQ (variable-length quantity)

Breaks a binary representation of an integer into a group of small blocks of variable bits.

Base64 VLQ

Optimised to make it easy to have mapping between big numbers and corresponding information in source files.

A line of code is represented in a series of “segments”. The number “1” would be: AAAA => 0000 Here is an example of how the numbers correlate to build a “Segment”:

Building a basic mapping in JavaScript would look something like this:

// .. define "item"
const sourceArray = []
sourceArray.push(item.generated.column)
sourceArray.push("file.es6.js")
sourceArray.push(item.source.line)
sourceArray.push(item.source.column)
const encoded = vlq.encode(sourceArray)

However this does not handle line and segment separation (which can be very tricky) so it is still more efficient to use Mozilla’s library.

3. Adding source map support

Back to our compiler !!

Using Mozillas SourceMapGenerator

To make full use of Mozillas library we will:

Create a sourceMap instance to hold and build our mappings
Initialise and store local mappings

So at times when a node has changed we will; build the location then add it to our local mappings and to the SourceMap instance. We keep a local instance so that we can keep a record of the current location start and end as that will be crucial to build out the next location.

// SourceMap instance
const mozillaMap = new SourceMapGenerator({
  file: "index.es5.js",
})

// Local mappings instance
const mappings = [
  {
    target: {
      start: { line: 1, column: 0 },
      end: { line: 1, column: 0 },
    },
    source: {
      start: { line: 1, column: 0 },
      end: { line: 1, column: 0 },
    },
    name: "START",
  },
]

We need a function which will actually handle updating these mapping instances. The below “buildLocation” function handles all the location generation logic. Most libraries have a similar function using offsets for column and line given by caller.

Its job is to work out the new start of line and column numbers as well as end of line and column numbers. It will only add a mapping if a node has changed, limiting what mappings we will store.

const buildLocation = ({
  colOffset = 0,
  lineOffset = 0,
  name,
  source,
  node,
}) => {
  let endColumn, startColumn, startLine
  const lastGenerated = mappings[mappings.length - 1].target
  const endLine = lastGenerated.end.line + lineOffset
  if (lineOffset) {
    endColumn = colOffset
    startColumn = 0 // If new line reset column
    startLine = lastGenerated.end.line + lineOffset
  } else {
    endColumn = lastGenerated.end.column + colOffset
    startColumn = lastGenerated.end.column
    startLine = lastGenerated.end.line
  }

  const target = {
    start: {
      line: startLine,
      column: startColumn,
    },
    end: {
      line: endLine,
      column: endColumn,
    },
  }
  node.loc = target // Update node with new location

  const clonedNode = Object.assign({}, node)
  delete clonedNode.original // Only useful for check against original
  const original = node.original
  if (JSON.stringify(clonedNode) !== JSON.stringify(original)) {
    // Push to real mapping. Just START. END is for me managing state
    mozillaMap.addMapping({
      generated: {
        line: target.start.line,
        column: target.start.column,
      },
      source: sourceFile,
      original: source.start,
      name,
    })
  }

  return { target }
}

So now we have our “buildLocation”, we have to go about introducing it across the code. A couple of examples are below. For the “generateIdentifier” processor utility and “Literal” AST type statements you can see how we incorporate “buildLocation”.

// Processor utility
const generateIdentifier = id => {
  mappings.push(
    buildLocation({
      name: `_identifier_ name ${id.name}`,
      colOffset: String(id.name).length,
      source: id.original.loc,
      node: id
    })
  );
  return id.name;
};

// AST type statement function (part of "Statements" object)
Literal: function(node) {
  mappings.push(
    buildLocation({
      name: `_literal_ value ${node.value}`,
      colOffset: String(node.value).length,
      source: node.original.loc,
      node
    })
  );

  if (node.value === null) {
    return "null";
  }
  if (typeof node.value === "boolean") {
    return node.value ? "true" : "false";
  }
  return node.value;
};

We need to apply this throughout our code generator (i.e. all node processors and AST type statement functions).

I found this tricky as the node-to-character mapping is not always 1–2–1. For example a function could have brackets either side of its argument which has to be accounted for when it comes to character line positions. So:

(one) =>

Has different character positions to:

one =>

What most libraries do is introduce logic and defensive checks using information on the AST nodes, so all scenarios can all be covered. I would have followed the same practice except I was only adding code for what was absolutely necessary for our “index.es6.js”.

For full usage please see the code for my generator in the repository here. It is missing large pieces but it does the job and is the building blocks for a real code generator.

The final part is writing our source map contents into the source map file. This is surprisingly easy with Mozillas library as they expose a “toString()” method which will handle the Base64 VLQ encoding and build all the mappings into a v3 spec compliant file. NICE !!

// From our Mozilla SourceMap instance
fs.writeFileSync(`./build/index.es5.js.map`, mozillaMap.toString(), "utf8")

Now our reference to “./build/index.es5.js” earlier will have a file that exists.

OUR COMPILER IS NOW DONE !!!! 🤩

Thats it for the compiler, the last part is now getting confirmation that is has worked.

If we compile the code it should produce our build folder with 3 files inside.

npm run compile

That is the original, the generated and the source map.

4. Testing our source map

There is a fantastic website at https://sokra.github.io/source-map-visualization/ which lets you visualise source map mappings.

The page starts like this:

By dropping our 3 files into it we can now see this:

Very colourful eh !

It contains the original code, the generated code and the mappings decoded (at the bottom).

A little reminder of our transformation from earlier:

// Swap: "number + 1"
// - clone left node
const leftClone = Object.assign(
  {},
  sourceAst.body[0].body.body[0].argument.left
)
// - replace left node with right node
sourceAst.body[0].body.body[0].argument.left =
  sourceAst.body[0].body.body[0].argument.right
// - replace right node with left clone
sourceAst.body[0].body.body[0].argument.right = leftClone
// Now: "1 + number". Note: loc is wrong

We have swapped:

number + 1

Into:

1 + number

Can we confirm the mapping has worked?

If we hover over the character or mapping, it will highlight the mapping and its corresponding locations in the generated and original locations.

This screenshot shows what happens when I hover over the literal number “1” character. It clearly shows there is a mapping.

This screenshot shows what happens when I hover over the variable identifier “number” word. It clearly shows there is a mapping.

SUCCESS !! 💪

What did we miss?

So what are the limitations of building a compiler like this?

Not all JavaScript statements are covered (only what our file needed)
Current it only works with 1 file. Web bundlers will follow the app building a dependency graph and applying transformations on those files (see my “under-the-hood of web bundlers” article for more on this).
Output file vs bundle. Web bundlers will produce bundles with code runnable in specific JavaScript environments, ours was very limited (again see my “under-the-hood of web bundlers” for more).
Basic transform. Performing additional optimisations would not really be easy to do without a lot of new code.

Thanks so much for reading. This topic was very far reaching and I learnt a huge amount in researching it. I really hope this serves as useful in shining a light on how JavaScript compilers and source maps work together including the mechanisms involved.

The source code for this can be found at craigtaub/our-own-babel-sourcemap.

Thanks, Craig 😃

Under-the-hood of React Hooks

Mon, 20 Apr 2020 00:00:00 GMT

React is filled with interesting design patterns, when Hooks were introduced it cleaned up many of the issues people had with the more arguably frustrating features such as classes and lifecycle methods. Today we will have a look at building our own React engine with Hooks, so we can understand the mechanisms at work for useState and useEffect.

We will not be covering Fibre, reconciliation or the change detection mechanisms.

This is part of my "under-the-hood of" series:

A video for this post can be found here. A part of my "under-the-hood of" video series.

React overview

We should start with a basic bare-bones version of React. To do that we need to touch on some of the core design concepts which React follows, such as:

JSX

Elements are POJO’s (plain-old-javascript-objects)
Components are functions

Pure rendering

A instance of a render follows: Render = Component(Data)

State machine

One-directional data flow
Data change detection forces updates for all children who need it

We are going to build our own (super simple) version of React and then add Hooks, to do that we need a basic version of the React engine which should render (log) a component with some props and return the component (for interacting with later).

const React = {
  render: Component => {
    const exampleProps = {
      unit: "likes",
    }
    const compo = Component(exampleProps)
    console.log("Render: ", compo.inner)
    return compo
  },
}

Now we have a basic engine we need an accompanying component which returns the React element (object) during the render:

const Component = props => {
  return {
    type: "div",
    inner: props.unit,
  }
}

We can now use our React and Component together and see its output (below each line of code):

let App = React.render(Component) // render
// logs: render 'likes'
App = React.render(Component) // re-render
// logs: render 'likes'

So we now have a basic Component which is rendered by our React engine.

React Hooks

A brief reminder about Hooks.

"Hooks let you use state and other React features inside a pure component, without writing a class"

Classes would often confuse people and add bloat to your code, however there are other strengths to using Hooks some of which are:

Easier to build and re-use stateful logic
Easier to split components into relatable pieces -Removes confusion over the many lifecycle methods (i.e. what correlates to what event)
No fundamental changes to your React knowledge, just a more direct way to use features you already know
Easy to add types, as Hooks are just functions (unlike high-order components and most of the render props pattern)
Easy to test with "react-dom/test-utils"

So it is easy to see why Hooks are such a powerful mechanism. So let’s examine how they work and build our own.

The useState Hook

There are a couple of core concepts with useState:

Use state in pure component.
Use any primitive (string/array/object etc.)

According to the docs they always return:

a reducer state
an action dispatcher

However this can be simplified into a basic property value and a setter. There is a simple rule which is very important to Hooks. That is that the Hooks are called at the top. That means outside of any conditionals or nesting blocks. This is crucial to the Hooks mechanism and we will examine why.

So let us expand out React engine to include the useState Hook. We will need a "state index" counter, as well as "state store" for values.

const React = {
  index: 0, // state index
  state: [], // state store
  useState: defaultProp => {
    const cachedIndex = React.index
    if (!React.state[cachedIndex]) {
      React.state[cachedIndex] = defaultProp
    }

    const currentState = React.state[cachedIndex]
    const currentSetter = newValue => {
      React.state[cachedIndex] = newValue
    }
    React.index++
    return [currentState, currentSetter]
  },
  render: Component => {
    const exampleProps = {
      unit: "likes",
    }
    const compo = Component(exampleProps)
    console.log("Render: ", compo.inner)
    React.index = 0 // reset index
    return compo
  },
}

We have also added a useState method. It works by:

Checking if the current index has a value in the state, if not set the default
Then build the setter for the current state index value
Finally increment the index (for the next Hook) and return the setter and current value.

Now we will make use of our useState in our Component.

const Component = props => {
  const [count, setCount] = React.useState(0)
  const [name, setName] = React.useState("Steve")

  return {
    type: "div",
    inner: `${count} ${props.unit} for ${name}`,
    click: () => setCount(count + 1),
    personArrived: person => setName(person),
  }
}

See the "count" and "name" usage above. Also we have utilised both inside the "inner" property so that we can spy on it in the terminal. Additionally we have added some manual methods to update the state values. They are manual as we are not worrying about the change detection or reconciliation in this post. We will be manually triggering the returned functions as well as the render method (following how we used it in our React engine section earlier).

So on running the below, we log the rendered output. We can see with each "click() "we are updating the count, and with "personArrived()" we are updating person (terminal output below line of code).

let App = React.render(Component) // render
// render '0 likes for Steve'
App = React.render(Component) // re-render
// render '0 likes for Steve'
App.click()
App = React.render(Component) // re-render
// render '1 likes for steve'
App.click()
App.personArrived("Peter")
App = React.render(Component) // re-render
// render '2 likes for Peter'

There is 1 main issue in our usage. That is that this would NOT work for many components at the same time. It is very tightly coupled to our single component. The state should be held in module scope or a single store namespaced to the component.

The useEffect Hook

A short quote from the docs:

"A function which runs after the initial render and after every update"

Similarly to useState, the rule about calling at the top-level applies. It is required for our implementation as well.

Couple of concepts for effects:

Created during render
Run in definition order (like useState)
Run after a given lifecycle event (first render and/or when a dependency has changed. i.e. mount and update)
Returns a function (some effects require cleaning e.g. unsubscribing)

Our useEffect Hook mechanism is below:

useEffect: (callback, dependencies) => {
  const cachedIndex = React.index;
  const hasChanged = dependencies !== React.state[cachedIndex];
  if (dependencies === undefined || hasChanged) {
    callback();
    React.state[cachedIndex] = dependencies;
  }
  React.index++;
  return () => console.log("unsubscribed effect");
},

A run-through of the code:

Cache the store index
Check if the dependency has changed in state (this is our variable tracking mechanism)
If there is no dependency given or its changed, run our effect and update store index value with our dependency value
Then increment the store index for future Hooks to use their own index
Finally return a function

You can see the way that our effect Hooks also depend on the "state index". Similarly to useState, it relies on the deterministic order of values in the state array.

All state and effect Hooks must be called in the same order, so their state index can always be found in the same location in the state array.

Now lets update our component to use the effect Hook

const Component = props => {
  const [count, setCount] = React.useState(0)
  const [name, setName] = React.useState("Steve")

  const exitThis = React.useEffect(() => {
    console.log("Effect ran")
  }, name)

  return {
    type: "div",
    inner: `${count} ${props.unit} for ${name}`,
    click: () => setCount(count + 1),
    personArrived: person => setName(person),
    unsubscribe: () => exitThis(),
  }
}

The above Hook will look for changes on the "name" property, also we have exposed an "unsubscribe" function for our effect Hook.

So now onto the running order during rendering, I have logged the state array with each render to help us follow (see below screenshot for the detailed explanation).

let App = React.render(Component) // render
// state: [ 0, 'Steve' ]
// Effect ran
App = React.render(Component) // re-render
// state: [ 0, 'Steve', 'Steve' ]
// Effect does not run
App.click()
App = React.render(Component) // re-render
// state: [ 1, 'Steve', 'Steve' ]
// Effect does not run
App.click()
App.personArrived("Peter")
App = React.render(Component) // re-render
// state: [ 2, 'Peter', 'Steve' ]
// Effect ran

App.unsubscribe()
// "unsubscribed effect"

The below details what is happening above for our effect.

First render

After 2 state runs the "cachedIndex" is now 2 inside of useEffect. The state value is "undefined" as it does not exist for that index (yet). So we will run our callback and write a new entity into state (under index 2), which is our dependency value (i.e. currently "Steve").

SO: state now has local state values and effect values. It is in a deterministic order based on the “state index” (i.e. where it is called in the component). This means it is reliable as this order should not change. Remember Rule #1.

Re-renders

We will check the 3rd value in our state array and the effect will run only if it changes. See above our Hook runs again when we change the name to "Peter". Lastly we unsubscribe from our Hook.

There are 2 main issues with the above (among others):

1. Only works with 1 dependency, not an array.

It is easy to expand our code so we could use an array. By using the same store index and storing an array of dependencies on the state, not just single value. Then iterate over that dependency array to check for changes.

2. Ours is run before the render, not after

This is a more complex issue to fix and would require fundamental changes to our React engine. As this is a small POC example it is not worth fixing.

I hope this was useful or at the very least a little interesting. I very much enjoyed learning about Hooks and have a deeper appreciation for them as a feature which has helped move the front-end developer experience forward. You can find a gist of the code here Thanks, Craig 😃

Under-the-hood of Web bundlers (e.g. Webpack)

Sun, 19 Apr 2020 00:00:00 GMT

Webpack is somewhat of a black box for most developers. Tools like "create-react-app" abstract most of the bundler functionality away. I did some research into it and began building my own light-weight web bundler to understand more about what it entails.

There will be 3 parts to this article:

What is a "web bundler"
Building a compiler for a "web bundler"
Using the output with an application

This is part of my "under-the-hood of" series:

A full video walkthrough for this post can be found here. A part of my "under-the-hood of" video series.

1. What is a "web bundler"

We should first ask the question "Its 2020, why bundle in the first place?". There are many answers to this question:

Performance: 3rd party code is expensive, we can use static code analysis to optimise it (things like cherry picking and tree shaking). We can also simplify what is shipped by turning 100 files into 1, limiting the data and resource expense on the user
Support: the web has so many different environments and you want your code to run in as many as possible, while only writing it once (e.g. adding Polyfills where necessary)
User experience: Utilise browser caching with separate bundles (e.g. vendor for all your libraries and app for your application itself) Separate concerns: Manage how you serve fonts, css, images as well as JS.

The basic architecture of a web bundler is:

Basically we put modules through a compiler to produce assets.

There are many concepts involved in the compiler. It is one of the reasons why I feel it is such an interesting topic, as there is so much in such a small amount of space.

These concepts are:

IIFE
Pass by ref
Dependency graphs (as we traverse our application files)
Defining custom import/export system (which can run on any environment)
Recursive functions
AST parsing and generation (turning source code into its tokenized form)
Hashing
Native ESM (ESM manages cyclic dependencies well due to its compile-time checks)

We will be ignoring non-js assets in our compiler; so no fonts, css or images.

2. Building a compiler for a "web bundler"

This will be a massive oversimplification of how Webpack works, as there are many different ways to solve the problem, hopefully this way will offer some insight into the mechanisms involved.

The overview of a compiler is below, we will be breaking down each phase.

Our application:

Our application consists of 4 files. Its job is to get a datetime, then hand that to a logDate, whose job is to add text to the date and send it to a logger. It is very simple.

Our application tree is thus:

PHASE 1

Using a 3rd party tool for AST parsing we (see code below):

Determine files full path (very important so its clear if we are dealing with the same file again)
Grab files contents
Parse into AST
Store both contents and AST onto a "module" object.
Process the dependencies inside the contents (using the AST "ImportDeclaration" value), recursively calling this function with the value
Finally add that function to the depsArray, so we can build up our tree with the first file appearing last (this is important)

SO our tree now looks like the below right array:

PHASE 2

A compilers job is to "Execute code which will produce executable code". This means we will have 2 levels of code so we will review them 1 at a time. First we will review what the compiler builds, then review the built/outputted code (run by the browser).

First the built code

Templates:

Module template: Its job is to convert a given module into a module our compiler can use.

We hand it the module code and an index (Webpack also does this with the index).

We want the code to be as compatible in as many environments as possible. ES6 modules support strict mode natively, but ES5 modules do not so we explicitly define strict mode in our module templates.

In NodeJS all ES modules are internally wrapped in a function attaching runtime details (i.e. exports), here we are using the same. Again Webpack does this.

Runtime template: Its job is to load our modules and give a id of the starting module.

We will review this more later, once we have the modules code inside it.

Custom import/export:

With our import statement we will be replacing the instance of "importing" with our own. It will look like the middle comment.

Our export will do something similar to the import, except replace any "exports" with our own. See bottom comment.

It is worth noting Webpack stores dependency IDs on the module earlier. It has its own "dependency template" which replaces the imports and exports usage with custom variables. Mine swaps just the import itself (theirs swaps the entire line and all usages of it). One of MANY things which aren’t exactly the same as the real Webpack.

Transform

Our transform function iterates through the dependencies. Replaces each import and export it finds with our own. Then turns the AST back into source code and builds a module string. Finally we join all the module strings together and hand them into the runtime template, and give the index location of the last item in the dependency array as this is our "entry point".

Now the code outputted from the compiler:

The left hand side is our runtime, the right hand side shows all the "modules" which are loaded. You can see they are the modules we started with at the beginning.

What is going on?

The runtime template IIFE runs immediately handing the modules array as an argument. We define a cache (installedModules) and our import function (ourrequire_). Its job is to execute the module runtime and return the exports for a given module id (the ID correlates to its location in the modules array). The exports are set on the parent module, utilising pass-by-ref, and the module is then stored in cache for easier re-use.. Finally we execute the import function for our entry point which will start the application as it does not require calling an export itself. All imports inside our modules will now utilise our custom method.

3. Using the output with an application

Now we have an updated "vendorString" we want to use it (the above code). So we:

Create a hash of the contents which is to be used in the bundle filename and stored in the manifest
Write the vendorString into our new bundle

Lastly we run a small express server application which pulls the bundle name from the manifest and exposes the built code (/build) under a /static route.

If we now run:

> npm run compile
> npm run start

Our application will run and we can see our bundle and its contents in the "network" tab.

Lastly we can confirm it worked by checking the "console". Good job 👍

Not covered

You might be wondering "so what else does Webpack do which ours does not?"

Handles non-js assets (css/images/fonts)
Dev and HMR: this is built into Webpack
Chunks: Webpack can put different modules into different chunks, and each can have a slightly different runtime and polyfills if necessary. i.e. vendor, dynamic imports
Multiple exports: Ours could do this but needs a defensive check on the module type so its not worth it for this mess.
Further optimisations (e.g. minification/code splitting/cherry picking/tree shaking/polyfills)
Source maps: Webpack uses a mix of preprocessor which all generate their own maps. Webpack manages merging them all together.
Making it extensible or configurable (e.g. loaders, plugins or lifecycle). Webpack is 80% plugins even internally i.e. the compiler fires hooks on lifecycle events (e.g. "pre-process file") and the loaders listen out for this event and run when appropriate. Additionally we could extend our compiler to support lifecycle events, perhaps using NodeJS event emitter, but again not worth it for this mess.

Thats it

I hope this was useful to you as I certainly learnt a lot from my time on it. There is a repository for anyone interested found at craigtaub/our-own-webpack

Thanks, Craig 😃

craigtaub.dev

Subscribe

The RSS Feed

Email list

Other ways

Thank you!

NodeJS vs Golang - an AWS costing exercise

Introducing CompanyX

CompanyX - January AWS bill

What can we ignore?

What is the savings?

EC2

Fargate

Lambda

The savings

Overall

Under-the-hood of GraphQL DataLoader

1: Overview

Batching

VS your ORM

Batch function

Caching

2. Building our own GraphQL Dataloader

Under-the-hood of Docker

1: Overview

Client

Registry

Host

Daemon

Example - build command

Build cache

Images

Image layers

Containers

Container layer

Storage/volumes

Storage driver

Copy-on-Write strategy

OverlayFS

Volumes

Networking

2: Building our own Docker

Daemon

Existing images

images/lowest-layer contains:

images/middle-layer contains:

CLI

Commands

Build

Run

Running the app

Lets go 💪

Run middle-layer image

Build highest-layer image

Run highest-layer image

Under-the-hood of NPM

1: Overview

CLI tool

npm init

npm install

Lockfile

Module resolution

Cache

npm version

npm pack

npm publish

Registry

Tarballs

API

Querying

NPM links

Website

2. Module vs Package

3: Building our own NodeJS package manager

Registry

CLI

Install script

Publish script

Website

Testing it works

`npm init`

`npm install`

`npm version`

`npm pack`

`npm publish`

`example-lib`

`example-app`

1. Parses "schema notation" (usually found in a `schema.graphql` file) into AST