fix(babel): memory consumption (#1331)

* chore: changeset

* feat(babel): new cache system

* fix(babel): invalidate cache whenever it's possible

* fix(babel): fix tests for module.ts

* chore: apply eslint member-ordering

* fix(babel): leaked AbortError

* fix(babel): crashes during concurrent runs

* feat(babel): softErrors feature flag and docs for globalCache and happyDOM

* fix(babel): several massive memory leaks

* fix(babel): quick fix for `Unexpected token 'export'`

* fix(babel): test for superseded entrypoint in eval

* fix(babel): bug with concurrent processing

* fix(babel): more logs for uncaught AbortError

* fix(babel): Unhandled AbortError

* fix(babel): an attempt to break the loop in processing

* feat(babel): cache for getExports

* fix(babel): endless loop if an error happens in generator

.changeset/thirty-otters-hammer.md

@@ -0,0 +1,5 @@
+---
+'@linaria/babel-preset': patch
+---
+
+Optimized memory consumption.

.eslintrc.js

@@ -237,6 +237,7 @@ module.exports = {
           'error',
           { prefer: 'type-imports' },
         ],
+        '@typescript-eslint/member-ordering': memberOrder,
 
         // TODO
         '@typescript-eslint/ban-ts-comment': 0,
@@ -251,9 +252,7 @@ module.exports = {
     },
     {
       files: ['**/processors/**/*.ts'],
-      rules: {
-        '@typescript-eslint/member-ordering': memberOrder,
-      },
+      rules: {},
     },
     {
       files: ['*.js', '*.jsx'],

.gitignore

@@ -73,3 +73,6 @@ build/
 tsconfig.tsbuildinfo
 
 .linaria-cache
+
+# debug
+*.debug.ts

docs/FEATURE_FLAGS.md

@@ -10,6 +10,7 @@ Feature flags are used to enable or disable specific features provided. The `fea
 - `["glob1", "glob2"]`: Enables the feature for files matching any of the specified glob patterns.
 - `["glob1", "!glob2"]`: Enables the feature for files matching `glob1` but excludes files that match `glob2`.
 
+
 # `dangerousCodeRemover` Feature
 
 The `dangerousCodeRemover` is a flag that is enabled by default. It is designed to enhance the static evaluation of values that are interpolated in styles and to optimize the processing of styled-wrapped components during the build stage. This optimization is crucial for maintaining a stable and fast build process. It is important to note that the `dangerousCodeRemover` does not impact the runtime code; it solely focuses on the code used during the build.
@@ -45,3 +46,18 @@ Suppose you have a file named `specialComponent.js` that contains code that shou
 ```
 
 You are instructing Linaria to exclude the `specialComponent.js` file from the removal process. As a result, any code within this file that would have been removed by the `dangerousCodeRemover` will be retained in the build output.
+
+
+# `globalCache` Feature
+
+The `globalCache` is enabled by default. Linaria uses two levels of caching to improve the performance of the build process. The first level is used to cache transformation and evaluation results for each `transform` call, usually a single call of Webpack's loader or Vite's transform hook. The second level is used to cache the results of the entire build process. The `globalCache` feature controls the second level of caching. Turning off this feature will result in a slower build process but decreased memory usage.
+
+
+# `happyDOM` Feature
+
+The `happyDOM` is enabled by default. This feature enables usage of https://github.com/capricorn86/happy-dom to emulate a browser environment during the build process. Typically, the `dangerousCodeRemover` feature should remove all browser-related code. However, some libraries may still contain browser-specific code that cannot be statically evaluated. In such cases, the `happyDOM` feature can be used to emulate a browser environment during the build process. This allows Linaria to evaluate the code without encountering errors caused by missing browser APIs.
+
+
+# `softErrors` Feature
+
+The `softErrors` is disabled by default. It is designed to provide a more lenient evaluation of styles and values that are interpolated in styles. This flag is useful for debugging and prevents the build from failing even if some files cannot be processed with Linaria.

packages/babel/package.json

@@ -47,7 +47,8 @@
     "cosmiconfig": "^8.0.0",
     "happy-dom": "10.8.0",
     "source-map": "^0.7.3",
-    "stylis": "^3.5.4"
+    "stylis": "^3.5.4",
+    "ts-invariant": "^0.10.3"
   },
   "devDependencies": {
     "@types/babel__core": "^7.1.19",

packages/babel/src/HOW_IT_WORKS.md

@@ -54,7 +54,7 @@ This function identifies the names of all exports in the file. If the file conta
 
 ### transform
 
-Prepares the file for execution: identifies the utilized Linaria processors, invokes eval-time substitutions for them, removes unnecessary code, and calls the `evaluator`. From the remaining code, surviving imports are extracted, wrapped in `resolveImports`, and queued. After imports are resolved, the `processImports` task is set to handle them. The final step is `addToCodeCache`, which stores all the gathered information in the cache for later utilization in `evalFile`.
+Prepares the file for execution: identifies the utilized Linaria processors, invokes eval-time substitutions for them, removes unnecessary code, and calls the `evaluator`. From the remaining code, surviving imports are extracted, wrapped in `resolveImports`, and queued. After imports are resolved, the `processImports` task is set to handle them.
 
 
 ### resolveImports
@@ -67,11 +67,6 @@ This function exists in two variants: synchronous for strictly synchronous envir
 Invokes `createEntrypoint` for each import. At this stage, it might return `"ignored"` if a loop is detected. In this case, the specific import is skipped. For the remaining imports, `processEntrypoint` will be enqueued without the parent's `AbortSignal`.
 
 
-### addToCodeCache
-
-Simply adds the result to the cache. While it could be done directly in `transform`, this approach provides clearer logging.
-
-
 ### evalFile
 
 Executes the code prepared in previous steps within the current `Entrypoint`. Returns all exports that were requested in `only`.

packages/babel/src/cache.ts

@@ -1,52 +1,25 @@
 import { createHash } from 'crypto';
 
-import type { File } from '@babel/types';
-
 import { linariaLogger } from '@linaria/logger';
+import { getFileIdx } from '@linaria/utils';
 
-import type { IBaseEntrypoint, IModule, ITransformFileResult } from './types';
+import type { Entrypoint } from './transform/Entrypoint';
+import type { IEvaluatedEntrypoint } from './transform/EvaluatedEntrypoint';
 
 function hashContent(content: string) {
   return createHash('sha256').update(content).digest('hex');
 }
 
 interface ICaches {
-  entrypoints: Map<string, IBaseEntrypoint>;
-  ignored: Map<string, true>;
-  resolve: Map<string, string>;
-  resolveTask: Map<
-    string,
-    Promise<{
-      importedFile: string;
-      importsOnly: string[];
-      resolved: string | null;
-    }>
-  >;
-  code: Map<
-    string,
-    {
-      imports: Map<string, string[]> | null;
-      only: string[];
-      result: ITransformFileResult;
-    }
-  >;
-  eval: Map<string, IModule>;
-  originalAST: Map<string, File>;
+  entrypoints: Map<string, Entrypoint | IEvaluatedEntrypoint>;
+  exports: Map<string, string[]>;
 }
 
 type MapValue<T> = T extends Map<string, infer V> ? V : never;
 
 const cacheLogger = linariaLogger.extend('cache');
 
-const cacheNames = [
-  'entrypoints',
-  'ignored',
-  'resolve',
-  'resolveTask',
-  'code',
-  'eval',
-  'originalAST',
-] as const;
+const cacheNames = ['entrypoints', 'exports'] as const;
 type CacheNames = (typeof cacheNames)[number];
 
 const loggers = cacheNames.reduce(
@@ -58,76 +31,57 @@ const loggers = cacheNames.reduce(
 );
 
 export class TransformCacheCollection {
-  private contentHashes = new Map<string, string>();
-
-  protected readonly entrypoints: Map<string, IBaseEntrypoint>;
-
-  protected readonly ignored: Map<string, true>;
-
-  protected readonly resolve: Map<string, string>;
-
-  protected readonly resolveTask: Map<string, Promise<string>>;
-
-  protected readonly code: Map<
-    string,
-    {
-      imports: Map<string, string[]> | null;
-      only: string[];
-      result: ITransformFileResult;
-    }
-  >;
+  public readonly entrypoints: Map<string, Entrypoint | IEvaluatedEntrypoint>;
 
-  protected readonly eval: Map<string, IModule>;
+  public readonly exports: Map<string, string[]>;
 
-  protected readonly originalAST: Map<string, File>;
+  private contentHashes = new Map<string, string>();
 
   constructor(caches: Partial<ICaches> = {}) {
     this.entrypoints = caches.entrypoints || new Map();
-    this.ignored = caches.ignored || new Map();
-    this.resolve = caches.resolve || new Map();
-    this.resolveTask = caches.resolveTask || new Map();
-    this.code = caches.code || new Map();
-    this.eval = caches.eval || new Map();
-    this.originalAST = caches.originalAST || new Map();
-  }
-
-  public invalidateForFile(filename: string) {
-    cacheNames.forEach((cacheName) => {
-      this.invalidate(cacheName, filename);
-    });
-  }
-
-  public invalidateIfChanged(filename: string, content: string) {
-    const hash = this.contentHashes.get(filename);
-    const newHash = hashContent(content);
-
-    if (hash !== newHash) {
-      cacheLogger('content has changed, invalidate all for %s', filename);
-      this.contentHashes.set(filename, newHash);
-      this.invalidateForFile(filename);
-
-      return true;
-    }
-
-    return false;
+    this.exports = caches.exports || new Map();
   }
 
   public add<
     TCache extends CacheNames,
     TValue extends MapValue<ICaches[TCache]>,
   >(cacheName: TCache, key: string, value: TValue): void {
     const cache = this[cacheName] as Map<string, TValue>;
-    loggers[cacheName]('add %s %f', key, () => {
-      if (!cache.has(key)) {
-        return 'added';
+    loggers[cacheName](
+      '%s:add %s %f',
+      getFileIdx(key).toString().padStart(5, '0'),
+      key,
+      () => {
+        if (!cache.has(key)) {
+          return 'added';
+        }
+
+        return cache.get(key) === value ? 'unchanged' : 'updated';
       }
-
-      return cache.get(key) === value ? 'unchanged' : 'updated';
-    });
+    );
 
     cache.set(key, value);
   }
 
+  public clear(cacheName: CacheNames | 'all'): void {
+    if (cacheName === 'all') {
+      cacheNames.forEach((name) => {
+        this.clear(name);
+      });
+
+      return;
+    }
+
+    loggers[cacheName]('clear');
+    const cache = this[cacheName] as Map<string, unknown>;
+
+    cache.clear();
+  }
+
+  public delete(cacheName: CacheNames, key: string): void {
+    this.invalidate(cacheName, key);
+  }
+
   public get<
     TCache extends CacheNames,
     TValue extends MapValue<ICaches[TCache]>,
@@ -158,10 +112,24 @@ export class TransformCacheCollection {
     cache.delete(key);
   }
 
-  public clear(cacheName: CacheNames): void {
-    loggers[cacheName]('clear');
-    const cache = this[cacheName] as Map<string, unknown>;
+  public invalidateForFile(filename: string) {
+    cacheNames.forEach((cacheName) => {
+      this.invalidate(cacheName, filename);
+    });
+  }
 
-    cache.clear();
+  public invalidateIfChanged(filename: string, content: string) {
+    const hash = this.contentHashes.get(filename);
+    const newHash = hashContent(content);
+
+    if (hash !== newHash) {
+      cacheLogger('content has changed, invalidate all for %s', filename);
+      this.contentHashes.set(filename, newHash);
+      this.invalidateForFile(filename);
+
+      return true;
+    }
+
+    return false;
   }
 }

packages/babel/src/evaluators/index.ts

@@ -2,30 +2,21 @@
  * This file is an entry point for module evaluation for getting lazy dependencies.
  */
 
-import type { StrictOptions } from '@linaria/utils';
-
 import type { TransformCacheCollection } from '../cache';
 import Module from '../module';
+import type { Entrypoint } from '../transform/Entrypoint';
 
 export default function evaluate(
   cache: TransformCacheCollection,
-  code: string,
-  pluginOptions: StrictOptions,
-  filename: string
+  entrypoint: Entrypoint
 ) {
-  const m = new Module(
-    filename ?? 'unknown',
-    '__linariaPreval',
-    pluginOptions,
-    cache
-  );
+  const m = new Module(entrypoint, cache);
 
-  m.dependencies = [];
-  m.evaluate(code);
+  m.evaluate();
+  m.dispose();
 
   return {
-    value: m.exports,
+    value: entrypoint.exports,
     dependencies: m.dependencies,
-    processors: m.tagProcessors,
   };
 }

packages/babel/src/index.ts

@@ -13,11 +13,17 @@ import type { PluginOptions } from './types';
 export { slugify } from '@linaria/utils';
 
 export { default as preeval } from './plugins/preeval';
-export { default as withLinariaMetadata } from './utils/withLinariaMetadata';
+export {
+  default as withLinariaMetadata,
+  getLinariaMetadata,
+} from './utils/withLinariaMetadata';
 export { default as Module, DefaultModuleImplementation } from './module';
 export { default as transform } from './transform';
 export * from './types';
+export { EvaluatedEntrypoint } from './transform/EvaluatedEntrypoint';
+export type { IEvaluatedEntrypoint } from './transform/EvaluatedEntrypoint';
 export { parseFile } from './transform/Entrypoint.helpers';
+export type { LoadAndParseFn } from './transform/Entrypoint.types';
 export { baseHandlers } from './transform/generators';
 export { prepareCode } from './transform/generators/transform';
 export { Entrypoint } from './transform/Entrypoint';
@@ -28,8 +34,9 @@ export {
 } from './transform/generators/resolveImports';
 export { default as loadLinariaOptions } from './transform/helpers/loadLinariaOptions';
 export { withDefaultServices } from './transform/helpers/withDefaultServices';
+export type { Services } from './transform/types';
 export { default as isNode } from './utils/isNode';
-export { default as getTagProcessor } from './utils/getTagProcessor';
+export { getTagProcessor } from './utils/getTagProcessor';
 export { default as getVisitorKeys } from './utils/getVisitorKeys';
 export type { VisitorKeys } from './utils/getVisitorKeys';
 export { default as peek } from './utils/peek';