Memory consumption

[Anber]

Motivation

Memory management is far from optimal. Let's fix it.

Summary

How it started

With enabled Linaria:

{
  rss: 2658156544,
  heapTotal: 2518843392,
  heapUsed: 1744077720,
  external: 4438696,
  arrayBuffers: 1480808
}

✓ 4846 modules transformed.

✓ built in 39.06s

With disabled Linaria:

{
  rss: 1270349824,
  heapTotal: 1151057920,
  heapUsed: 923673784,
  external: 4455888,
  arrayBuffers: 1498000
}

✓ 4349 modules transformed.

✓ built in 16.58s

How it ended

{
  rss: 1758117888,
  heapTotal: 1624358912,
  heapUsed: 1517192880,
  external: 4878561,
  arrayBuffers: 1920673
}

✓ 4846 modules transformed.

✓ built in 34.18s

[changeset-bot]

🦋 Changeset detected

Latest commit: 21183fe

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 15 packages

Name	Type
@linaria/babel-preset	Patch
@linaria/cli	Patch
@linaria/esbuild	Patch
linaria	Patch
@linaria/rollup	Patch
@linaria/stylelint	Patch
@linaria/testkit	Patch
@linaria/vite	Patch
@linaria/webpack4-loader	Patch
@linaria/webpack5-loader	Patch
esbuild-example	Patch
rollup-example	Patch
vite-example	Patch
vpssr-linaria-solid-example	Patch
@linaria/webpack-loader	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[PierreGUI]

Do you need any logs from a failing build (heap out of memory)?

[Anber]

@PierreGUI thank you, but I don't need it now. I'll finish cache optimisation tomorrow, and then we will see what to do.

[Anber]

@PierreGUI @layershifter I hope, it's ready for testing.

[PierreGUI]

Hey @Anber thanks! I tried to build and got this error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Looks different from the one before, albeit still a "out of memory" 😢

[Anber]

@PierreGUI, let's try two things:

1. Add to your Linaria config the following fragment:

features: {
  globalCache: false,
  happyDOM: false,
}

2. Run build with DEBUG=linaria:*,-*:source*

In the second case, I don't need a log itself — just the size of it. You can forward it to a file with something like DEBUG=linaria:*,-*:source* pnpm build 2>log.txt

[Anber]

@PierreGUI Have you tried to increase the memory limit? https://geekflare.com/fix-javascript-heap-out-of-memory-error/ I'm still not sure if it is a general problem with memory consumption or some infinite loop inside the processing. If increasing won't help, then it's an infinite loop somewhere, and it's kind of good news.

[PierreGUI]

Hey @Anber,
Yes I have tried increasing max-old-space-size but it doesn't help.

1. Running with no globalCache nor happyDOM
   It takes a very long time and spawns this error a lot (I stopped at 21% building 0/2 entries 4095/4200 dependencies 1014/1208 modules, ~90 errors :

Error during transform of /src/components/clients/campaigns/NotificationBanner.view.tsx: RangeError: Maximum call stack size exceeded
    at StackOfMaps.join (/Users/pierre/code/linaria/packages/babel/lib/transform/helpers/StackOfMaps.js:70:16)
    at new Entrypoint (/Users/pierre/code/linaria/packages/babel/lib/transform/Entrypoint.js:36:19)
    ...

2. Running with DEBUG flags (but no extra config)
   It creates a file that is 1282240B (~1.3mB). It stops because out of memory.

So the hypothesis of an infinite loops grows stronger 🤔

[Anber]

@PierreGUI have you tried with the last version? We have found and fixed one infinity loop yesterday.

[Anber]

I see that you tried with the old version. @layershifter had the same StackOfMaps errors but with the last version everything is ok, however for such amount of modules it may require around 8Gb heap size. I will try to figure out how to reduce memory consumption, possibly at the expense of performance.

[PierreGUI]

Ehm, I pulled this branch's HEAD, ... and then forgot to prepare 😮‍💨
Ok trying again.

[PierreGUI]

Using the latest version:

I build with only one webpack entry point (previously 2, out of 15 total). Although the build goes through for some entrypoint, this one entrypoint still throws an out of memory error.

1. Running with no globalCache nor happyDOM
   I managed to build one entrypoint (out of 15) in 317358 ms 🎉

2. Running with DEBUG flags (but no extra config)
   linaria-debug-memory is 493895B (500kB). Assuming because running the build with only one entry point (instead of 2).

[Anber]

Looks like we have the new champion by the size of the codebase :) Anyway, the challenge accepted.

[Anber]

At that point, I've done everything possible. The next step is to improve the debug experience so it will be easy to understand what should be fixed next.

[PierreGUI]

I've tried this version and for my largest entrypoint status remains the same: the build gets stuck at some point (early 10% webpack progress), then it slowly grows in memory (I've tried up to 16gB) until it reaches the limit and blows up. In the meantime no new progress is displayed.
Let me know how I can help you finding that loop/leak and patch it!

[Anber]

@PierreGUI I'm going to add more debug info, and then we will try to figure out what happens in your project.

[PierreGUI]

Ok thanks 👍
FYI I managed to get an out of memory crash also with globalCache: false when I enable multiple webpack entrypoints (max 16gB RAM).

[Anber]

Hi @PierreGUI!

Have you checked the last published version?

[PierreGUI]

Amazing work! Thanks. The build no longer crashes and seems quite fast. There are a bunch of errors now, mostly types, I'll have a look and report them if necessary