Prebuild Tasks

There are two "prebuild" tasks, linting and type checking that need to be run before any kind of building takes place.

The buildtools call both ESLint and tsc in parallel since they are not dependent on each others' outputs.

Running ESLint from the Command Line

ESLint provides documentation detailing how to use its Node API. Below is the code that does just that:

export async function runEslint(
  input: InputAsset,
  { fix, stats, concurrency }: LintOptions = defaultLintOptions
): Promise<LintResult> {
  const linter = new ESLint({
    fix,
    stats,
    concurrency,
    cwd: gitRoot,
  });

  try {
    const linterResults = await linter.lintFiles(input.directory);
    if (fix) {
      await ESLint.outputFixes(linterResults);
    }

    if (stats) {
      const lintstatsDir = pathlib.join(outDir, 'lintstats');
      await fs.mkdir(lintstatsDir, { recursive: true });

      const csvFormatter = await linter.loadFormatter(pathlib.join(import.meta.dirname, '../../lintplugin/dist/formatter.js'));
      const csvFormatted = await csvFormatter.format(linterResults);
      await fs.mkdir(outDir, { recursive: true });
      await fs.writeFile(pathlib.join(lintstatsDir, `${input.type}-${input.name}.csv`), csvFormatted);
    }

    const outputFormatter = await linter.loadFormatter('stylish');
    const formatted = await outputFormatter.format(linterResults);

    const severity = findSeverity(linterResults, each => severityFinder(each, fix));

    return {
      formatted,
      severity,
      input
    };
  } catch (error) {
    return {
      severity: 'error',
      formatted: `${error}`,
      input
    };
  }
}

Because the configuration file for the repository is located at the root of the repository, we need to set the cwd to the path to the root of the repository when initializing the ESLint instance. This allows ESLint to resolve the configuration correctly.

Linting warnings and errors come in two types, fixable and non-fixable. Fixable errors don't cause a non-zero exit code when ESLint is run with --fix, while non-fixable errors always cause a non-zero exit code.

ESLint provides several formatters for processing the results objects it returns. To produce the human readable output that is printed to the command line, the stylish formatter is loaded and used.

Inspecting the Linting Config

The entire repository's linting configuration is located at the root of the repository within eslint.config.js. If you want to view the view what rules are being applied to which files you can use the config inspector, which can be started using yarn lint:inspect

When run with the --stats option, the collected stats will be written to the output directory under the lintstats folder.

Out of Memory Error

As of the time of writing, spinning up a single ESLint instance to lint all the files in the repository at once seems to cause ESLint/NodeJS to run out of memory. A lot of the tooling has been designed to seamlessly bypass this issue by linting each bundle and tab with a separate instance of ESLint, but this does mean that we also need specific tooling for linting the rest of the repository.

That's why there's a lint command and also a lintglobal command.

Calling Typescript from Node

Most of the code for running Typescript functionality from Node was taken from this Github issue.

import fs from 'fs/promises';
import pathlib from 'path';
import type { InputAsset, Severity } from '@sourceacademy/modules-repotools/types';
import { findSeverity } from '@sourceacademy/modules-repotools/utils';
import chalk from 'chalk';
import ts from 'typescript';

type TsconfigResult = {
  severity: 'error';
  error: any;
} | {
  severity: 'error';
  results: ts.Diagnostic[];
} | {
  severity: 'success';
  results: ts.CompilerOptions;
  fileNames: string[];
};

export type TscResult = {
  input: InputAsset;
} & ({
  severity: 'error';
  error: any;
} | {
  severity: Severity;
  results: ts.Diagnostic[];
});

async function getTsconfig(srcDir: string): Promise<TsconfigResult> {
  // Step 1: Read the text from tsconfig.json
  const tsconfigLocation = pathlib.join(srcDir, 'tsconfig.json');
  try {
    const configText = await fs.readFile(tsconfigLocation, 'utf-8');

    // Step 2: Parse the raw text into a json object
    const { error: configJsonError, config: configJson } = ts.parseConfigFileTextToJson(tsconfigLocation, configText);
    if (configJsonError) {
      return {
        severity: 'error',
        results: [configJsonError]
      };
    }

    // Step 3: Parse the json object into a config object for use by tsc
    const { errors: parseErrors, options: tsconfig, fileNames } = ts.parseJsonConfigFileContent(configJson, ts.sys, srcDir);
    if (parseErrors.length > 0) {
      return {
        severity: 'error',
        results: parseErrors
      };
    }

    return {
      severity: 'success',
      results: tsconfig,
      fileNames
    };
  } catch (error) {
    return {
      severity: 'error',
      error
    };
  }
}

export async function runTsc(input: InputAsset, noEmit: boolean): Promise<TscResult> {
  const tsconfigRes = await getTsconfig(input.directory);
  if (tsconfigRes.severity === 'error') {
    return {
      ...tsconfigRes,
      input
    };
  }

  const { results: tsconfig, fileNames } = tsconfigRes;

  try {
    // tsc instance that only does typechecking
    // Type checking for both tests and source code is performed
    const typecheckProgram = ts.createProgram({
      rootNames: fileNames,
      options: {
        ...tsconfig,
        noEmit: true
      }
    });
    const results = typecheckProgram.emit();
    const diagnostics = ts.getPreEmitDiagnostics(typecheckProgram)
      .concat(results.diagnostics);

    const severity = findSeverity(diagnostics, ({ category }) => {
      switch (category) {
        case ts.DiagnosticCategory.Error:
          return 'error';
        case ts.DiagnosticCategory.Warning:
          return 'warn';
        default:
          return 'success';
      }
    });

    if (input.type === 'bundle' && severity !== 'error' && !noEmit) {
      // If noEmit isn't specified, then run tsc again without including test
      // files and actually output the files
      const filesWithoutTests = fileNames.filter(p => {
        const segments = p.split(pathlib.posix.sep);
        return !segments.includes('__tests__');
      });
      // tsc instance that does compilation
      // only compiles non test files
      const compileProgram = ts.createProgram({
        rootNames: filesWithoutTests,
        options: {
          ...tsconfig,
          noEmit: false
        },
        oldProgram: typecheckProgram
      });
      compileProgram.emit();
    }

    return {
      severity,
      results: diagnostics,
      input
    };
  } catch (error) {
    return {
      severity: 'error',
      input,
      error
    };
  }
}

export function formatTscResult(tscResult: TscResult): string {
  const prefix = chalk.cyanBright('tsc completed');

  if (tscResult.severity === 'error' && 'error' in tscResult) {
    return `${prefix} ${chalk.cyanBright('with')} ${chalk.redBright('errors')}: ${tscResult.error}`;
  }

  const diagStr = ts.formatDiagnosticsWithColorAndContext(tscResult.results, {
    getNewLine: () => '\n',
    getCurrentDirectory: () => process.cwd(),
    getCanonicalFileName: name => pathlib.basename(name)
  });

  switch (tscResult.severity) {
    case 'error':
      return `${prefix} ${chalk.cyanBright('with')} ${chalk.redBright('errors')}\n${diagStr}`;
    case 'warn':
      return `${prefix} ${chalk.cyanBright('with')} ${chalk.yellowBright('warnings')}\n${diagStr}`;
    case 'success':
      return `${prefix} ${chalk.greenBright('successfully')}`;
  }
}

The high level overview of this process is as follows:

Read the raw text from the tsconfig.json
Parse the tsconfig.json into a JSON object using ts.parseConfigFileTextToJson
Parse the JSON object into actual compiler options using ts.parseJsonConfigFileContent. This also returns an array of file names for parsing.
Use ts.createProgram to get the preliminary program for type checking only.
Call typecheckProgram.emit() to produce the typechecking results.
Combine the results with ts.getPreEmitDiagonstics.
If there were no typechecking errors use ts.createProgram again with the typecheck program to perform compilation and declaration file emission excluding test files by manually setting noEmit to false.
Format the diagnostic objects using ts.formatDiagnosticsWithColorAndContext

Reading and Parsing `tsconfig.json`

The first three steps in the process involve reading the raw text from the tsconfig.json and then parsing it. At the end of it, ts.parseJsonConfigFileContent resolves all the inherited options and produces the compiler options in use, as well as the file paths to the files that are to be processed.

Type Checking

At step 4, ts.createProgram is called for the first time. It is called with every single file as returned from ts.parseJsonConfigFileContent. However, it is called with noEmit: true. This prevents any Javascript and Typescript declaration files from being written. This is important because we want test files to be type checked, but we don't want them to be compiled into Javascript and exported with the rest of the code. If they were included and the tsconfig was configured to produce outputs, the test files would end up being written to the outDir. typecheckProgram.emit is called to perform the type checking.

If there are no errors and the tsconfig was configured to produce outputs, ts.createProgram is called again. This time, test files are filtered out. ts.createProgram has a parameter for passing in the previous program object, allowing it to reuse an existing program so it doesn't have to reinitialize the entire object again. program.emit is called to produce any compiled Javascript and Typescript declaration files.

Prebuild Tasks ​

Running ESLint from the Command Line ​

Calling Typescript from Node ​

Reading and Parsing tsconfig.json ​

Type Checking ​

Prebuild Tasks

Running ESLint from the Command Line

Calling Typescript from Node

Reading and Parsing `tsconfig.json`

Type Checking