Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIO compiles everything in MBED framework when RTOS is enabled #288

Closed
SuperBoss9 opened this issue Sep 24, 2019 · 19 comments
Closed

PIO compiles everything in MBED framework when RTOS is enabled #288

SuperBoss9 opened this issue Sep 24, 2019 · 19 comments

Comments

@SuperBoss9
Copy link

Hello everyone!

The problem is in mbed support in PIO. If I enable RTOS support via recommended macro then PIO recompiles everything that is in MBED framework (all features, all drives and everything). This operation takes a lot of resources even at a very powerful PC.
If PIO recompiles everything only once it is possible to survive. But it recompiles everything at the following situations:

  1. I add some more include predicates into my code.
  2. I'd like to debug.

That makes work non possible because every compilation takes from 400 (i7) sec to 20 minutes (i3).

One more thing that mbed supports called .mbedignore that can switch off non necessary features is not supported in PIO.

@pfeerick
Copy link

@valeros
Copy link
Member

valeros commented Sep 24, 2019

Hi @SuperBoss9 ! Thanks for reporting. I'm afraid that's default build logic for mbed framework, all suitable source files are collected based on the target configuration. Don't mbed tools (mbed-cli or mbed IDE) also compile the entire framework?

  • What do you mean? The project will be rebuilt if your platformio.ini file is changed, no workaround here. If you add includes only to individual files, only they should be recompiled.
  • Since the debug session requires additional debug flags, the entire project will be recompiled.

Regarding .mbedignore file, it should work with PlatformIO as well. Are you sure you're correctly using it? How did you specify paths to the folders you want to ignore? Please share here contents of this file and the location where you placed it.
Thanks!

@SuperBoss9
Copy link
Author

Hi @SuperBoss9 ! Thanks for reporting. I'm afraid that's default build logic for mbed framework, all suitable source files are collected based on the target configuration. Don't mbed tools (mbed-cli or mbed IDE) also compile the entire framework?

  • What do you mean? The project will be rebuilt if your platformio.ini file is changed, no workaround here. If you add includes only to individual files, only they should be recompiled.
  • Since the debug session requires additional debug flags, the entire project will be recompiled.

Regarding .mbedignore file, it should work with PlatformIO as well. Are you sure you're correctly using it? How did you specify paths to the folders you want to ignore? Please share here contents of this file and the location where you placed it.

Hello valeros!
Regarding to your questions:

  1. If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.
  2. Mbed studio doesn't rebuild mbed core files if I'd like to debug. PIO does.

PIO with VScode is a great thing but it is not usable with mbed RTOS functions due to the compilation issue.
Mbed Studio doesn't recompile everything until I ask it to do that clearly. Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

Regarding to .mbedignore it works with mbed Studio, where I can just put the file into a project root dir. In PIO I tested any projects dirs without any effect (and with any combinations of pathes).

mbed ignore:
mbed-os/features/lorawan/*
mbed-os/features/nfc/*
mbed-os/features/cryptocell/*

Unfortunately I won't be able to provide PIO .mbedignore example until end of this week.

@valeros
Copy link
Member

valeros commented Sep 24, 2019

If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.

Couldn't reproduce this behavior. In my case PlatformIO recompiles only changed files.

Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

The reason why you need to specify a special flag is because we support both variations of mbed framework: mbed2 (framework without rtos library) and mbed-os. If you don't switch between mbed2 and mbed-os what is the problem to specify -DPIO_FRAMEWORK_MBED_RTOS_PRESENT once?

I'm not familiar with Mbed studio, but you can always build your project in debug configuration or even start the debug session without compiling your project?

mbed ignore:
mbed-os/features/lorawan/*
mbed-os/features/nfc/*
mbed-os/features/cryptocell/*

Where did you place this file? The paths are relative and we don't even have mbed-os folder in our framework package, so this won't work with PlatformIO as in our case framework is located in a special folder.

@SuperBoss9
Copy link
Author

If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.

Couldn't reproduce this behavior. In my case PlatformIO recompiles only changed files.

Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

The reason why you need to specify a special flag is because we support both variations of mbed framework: mbed2 (framework without rtos library) and mbed-os. If you don't switch between mbed2 and mbed-os what is the problem to specify -DPIO_FRAMEWORK_MBED_RTOS_PRESENT once?

I've double check the behaviour at my i7:

  1. 1st compiling with RTOS enabled:
    [SUCCESS] Took 136.24 seconds
  2. 2st compiling with RTOS enables
    [SUCCESS] Took 31.22 seconds
  3. 3rd compiling with RTOS enabled and added include of my own class located in the project
    [SUCCESS] Took 130.97 seconds

I didn't change platformio.ini just add include of my own class to main and that caused mandatory clean build.

I'm not familiar with Mbed studio, but you can always build your project in debug configuration or even start the debug session without compiling your project?

mbed ignore:
mbed-os/features/lorawan/*
mbed-os/features/nfc/*
mbed-os/features/cryptocell/*

Where did you place this file? The paths are relative and we don't even have mbed-os folder in our framework package, so this won't work with PlatformIO as in our case framework is located in a special folder.

.mbedignore from above is from Mbed studio project where it works well.
Closer to the end of the week I'll post the file I used in PIO (tryed to put it at any folder I have in my project). But anyway .mbedignore is not a complete workaround: it is not possible to switch off all the features due to dependences between them.

PS. I've checked mbed versions:
PIO: PACKAGES: toolchain-gccarmnoneeabi 1.70201.0 (7.2.1), framework-mbed 5.51204.190701 (5.12.4)
MBED Studio: Running mbed-os 5.14.0

@pfeerick
Copy link

pfeerick commented Sep 25, 2019

.mbedignore from above is from Mbed studio project where it works well.

That might be the case, but since the paths used are relative (not absolute), if not placed relative to the mbed-os folder, it's not going to work, is it? And as valeros pointed out that there is no mbed-os folder in the platformio framework... so the likelihood of it working are ... zilch, nada, nope.

I noticed in another forum thread someone suggested that upgrading from python 2.7 to python 3 fixed what they referred to as 'random rebuilds'... maybe worth checking if that's the culprit? Maybe it was just a coincidence?

@valeros
Copy link
Member

valeros commented Sep 25, 2019

@SuperBoss9 Could you please provide a minimal project to reproduce the spurious recompilation?

@SuperBoss9
Copy link
Author

@SuperBoss9 Could you please provide a minimal project to reproduce the spurious recompilation?

Yes, sure
https://github.com/SuperBoss9/MBEDRecompileIssue.git

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

Recompilation 1
#include <mbed.h>
//#include <MyClass.h>
//#include <MyClass2.h>

Recompilation 2
#include <mbed.h>
#include <MyClass.h>
//#include <MyClass2.h>

Recompilation 3
#include <mbed.h>
//#include <MyClass.h>
#include <MyClass2.h>

No Recompilation
#include <mbed.h>
#include <MyClass.h>
//#include <MyClass2.h>

@valeros
Copy link
Member

valeros commented Sep 25, 2019

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

I tried your project (Python 2&3) and didn't notice this issue. What is your OS, version of Python, version of PlatformIO?

@SuperBoss9
Copy link
Author

SuperBoss9 commented Sep 25, 2019

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

I tried your project (Python 2&3) and didn't notice this issue. What is your OS, version of Python, version of PlatformIO?

I have the same issue at at least 3 of my computers. All of them under Win10Prox64
PIO Home 2.3.3·Core 4.0.3 (PIO IDE with VSCode)
Python Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:30:55) [MSC v.1500 32 bit (Intel)] on win32

@valeros
Copy link
Member

valeros commented Sep 25, 2019

Compiled your project on 2 different machines with Win10 (Python 2&3) and still can't reproduce the issue. Could you try another python version?

@SuperBoss9
Copy link
Author

I think that you need to create a fresh class after 1st build.
Here is a video of the process https://youtu.be/_Gl8Fm_d7WE

Also will try the same at my other 2 PCs. Check Python version.

@valeros
Copy link
Member

valeros commented Sep 25, 2019

@SuperBoss9 Thanks for the video, it helped me understand what we are talking about. So when you add new file it rebuilds the whole environment and at the moment this is expected behavior. As a workaround your can try build-cache-dir option, it should help you avoid recompilation of the entire framework.

@SuperBoss9
Copy link
Author

Thank you

@DesktopMan
Copy link

If I build my project, then click build again without changing anything that second build randomly takes either 15 seconds (OK) or 3 minutes (not OK).

This can't possibly be expected behavior? Core 4.3.1, Home 3.1.1, ST STM32 6.0.0.

[env:disco_l475vg_iot01a]
platform = ststm32
board = disco_l475vg_iot01a
framework = mbed
build_flags = -D PIO_FRAMEWORK_MBED_RTOS_PRESENT

This is with the rtos_basic example.

@valeros
Copy link
Member

valeros commented Apr 3, 2020

Hi @DesktopMan ! Could you please try to run pio platform update ststm32 and try to compile your project a few times in a row? Does it work better now?

@DesktopMan
Copy link

DesktopMan commented Apr 3, 2020

I did a fresh Windows install and all the tools in a virtual machine to make sure the test is correct. Building the empty Mbed project twice takes 20 seconds each time, even without changes. It compiles the HAL and platform source every time.

If I enable the OS with build_flags = -D PIO_FRAMEWORK_MBED_RTOS_PRESENT the initial build took 5 minutes and the next two 6 minutes, so same result there.

Since this happens on a clean install and doesn't require hardware to test it should be possible to replicate.

@Mixerito
Copy link

This issue should be reopened, since the problem is not solved by any of proposed workarounds.

build_cache_dir does not help.

.mbedignore could help to speed up the build as it could at least omit unused components and features (still does not help with repeatedly built components that we use and were not changed)
BUT we are not able to ignore some components for one project and use them in another project, since the .mbedignore file must be placed inside ".platformio/packages/framework-mbed/" folder.
Still unresolved issue: platformio/builder-framework-mbed#13

With -D PIO_FRAMEWORK_MBED_RTOS_PRESENT every build (without any change)
compiles all mbed libraries.

It takes :

  • cca 4 minutes in Win7/VSCode/Platformio/mbed
  • cca 2 minutes in Win7/CLion/Platformio/mbed

The development with these timing constraints is very uneffective.

@Mixerito
Copy link

With a big help of Valeros, I found a problem in my project.

Firstly in file
c:\Users\YOUR_USER.platformio\penv\Lib\site-packages\platformio\platform_run.py (Win)
or
/home/YOUR_USER/.platformio/penv/Lib/site-packages/platformio/platform/_run.py (Linux)
activate the --debug=explain option.

        args = [
            proc.get_pythonexe_path(),
            script_path,
            "-Q",      
            "--debug=explain",
            "--warn=no-no-parallel-support",
            "--jobs",
            str(jobs),
            "--sconstruct",
            os.path.join(fs.get_source_dir(), "builder", "main.py"),
        ]

This will reveal you the reason of recompile (during build).

In my case it was:

  • rebuilding <something.o> because the contents of the build action changed

There are some reasons for build action changed:

  • some .py pre-scripts modifying the build flags or include libs
  • some dynamic directives in platformio.ini build_flags (my case: -D COMPILE_TIME=$UNIX_TIME)

Hope this helps someone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants