Source Map-Aware Code Generation

Lately I’ve been working on a code generation feature. Specifically, making a code generation process to be source map aware—so it will preserve the original source locations. This is an interesting thing because most people don’t handle source maps in their day to day, and it’s just a critical piece for great developer experience in the JavaScript world.

To do that, I did a couple of iterations and had a few insights and built some tools for the job I think is worth sharing.

When I first got the task, I realized I first need to replace my input files in the test with files that have source maps. That means, they have to be different than the original version. Some examples of tools that can do it are bundlers like webpack, esbuild and rollup, and transpilers like TypeScript and Babel.

None of these tools supports my use case, which is building and wrapping the user code with my own code. Essentially, what I’m aiming is as if I were building one of these bundlers!

In order to achieve my goal, I played around with magic-string. Magic String is an incredible library by Rich Harris of Svelte that has a great interface of applying string manipulations with source map generation.

This library is so good and so fun to work with. I found myself in need for a playground utility to have a quick way of generating source maps that are also shareable using URLs.

So I built the magic-string playground. It’s not a fancy app, but more a fun experiment. I don’t really expect it to be super useful for many people but it was to me. So definitely check it out if you are in need to create source maps by hand: https://magic-string-playground.vercel.app

The magic string library is very good. Unfortunately, it seems that it does not support a feature that I needed. I needed to preserve input source maps. I tried to play with some libraries to pair with magic string but then I found webpack-sources. It’s like the answer to magic-string from the Webpack community.

This library provides high level manipulation for sources, which can have input source maps—or not, using different Source interfaces. SourceMapSource is a structure that has source maps, and OriginalSource for instance, can be used for files we don’t have input sources for. Then, we can use ConcatSource to concat multiple sources into a single one, preserving the input source maps!

This Source composition makes the code easy to handle because every code generation function can change its signature from (input: string) => string to (input: Source) => Source and source maps and transformations are just a byproduct, handled by the library.

Let’s say we had the following code-generation source, which wraps the user code with a function that accepts a random variable:

function wrapUserCode(userCode: string): string {
  return `
    (function(random) {
      ${userCode}
    })
  `;
}

That is easy to read and comprehend. However, if we want to preserve source maps, we can start using ConcatSource from webpack-sources. It will look a bit different:

function wrapUserCode(userCode: Source): Source {
  const concat = new ConcatSource();
  concat.add(`(function(random) {\n`);
  concat.add(userCode);
  concat.add(`\n})`);
  return concat;
}

By doing so, we successfully preserve source maps (which is a big win). However, the code is harder to understand visually. The template literals helped us a lot by having visual cues where we integrate user code by some kind of indentation (even if the output indentation was off).

We should favor correctness over aesthetics. But I wondered, how we can get the best of both worlds?

We can use tagged template literals to make everything work like a magic. Our end goal would be to have the following API:

function wrapUserCode(userCode: Source): Source {
  return sourcemapped`
    (function(random) {
      ${userCode}
    })
  `;
}

... so what’s happening here? 🤔

Tagged Template Literals are a JavaScript feature that allows to call a function without parenthesis and instead use the template literal syntax. An array of all the plain strings will be the first argument of the function call, and all interpolated variables will also be sent as arguments. You might saw it previously with a couple of tagged template literals, like gql for GraphQL queries, or styled for Styled Components.

Practically, the example we just saw would be almost the same as writing the following:

function wrapUserCode(userCode: Source): Source {
  return sourcemapped([`(function(random) {\n`, `\n})`], userCode);
}

So now, it’s time to write our sourcemapped template tag. The implementation itself is rather easy, which is a good thing.

function sourcemapped(
  strings: TemplateStringsArray,
  ...sources: Source[]
): Source {
  const concat = new ConcatSource();

  for (let i = 0; i < strings.length; i++) {
    const string = strings[i];
    const source = sources[i];
    if (string) concat.add(string);
    if (source) concat.add(source);
  }

  return concat;
}

... that’s it. So simple. The ratio of impact to lines of code is so good. These 12 lines allow us to build much more maintainable software that enables developers to have much much better experience.

Isn’t it cool?