Support bundling libraries with a sketch

[hoblins]

Describe the request

🙂 It will be easier for sketch developers to share a sketch as a package that includes library dependencies, avoiding the need for each user to install the library dependencies.
🙂 Library dependencies of the sketch can be controlled, allowing them to be pinned to the specific versions the sketch was intended to be used with, independent from the version of any globally installations of the library. As an example there are major changes to the API of the very popular "ArduinoJson" library in version 6 and sketches only work with one or the other major version series of that dependency.

Describe the current behavior

The common approach to bundling libraries with a sketch is to dump the library code files into the sketch root folder.

Arduino CLI version

a58d5ad

Operating system

All

Operating system version

All

Additional context

1. As a workaround in the preferences the sketchbook location (directories.user) can be set to the sketch folder. But this has to be done manually and changed every time another program is handled. So there could be some option for this to be set in the .ino file.
2. In the build folder a file build.options.json is generated automatically.
   There are some options where additional library folders could possibly be added?
   E.g.:
   • builtInLibrariesFolders
   • otherLibrariesFolders

Additional Requests

• Add advanced mode from Pro IDE arduino-ide#150 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Support bundling libraries with a sketch #1255 (comment)
• Detect libraries in Sketch Folder #2763
• https://forum.arduino.cc/t/how-do-i-include-a-library-in-a-sketch-directory/1079812
• https://forum.arduino.cc/t/arduino-ide-cant-find-local-custom-libraries-in-relative-path/669421
• https://forum.arduino.cc/t/suggestion-improve-library-manager/847666
• https://forum.arduino.cc/t/using-generated-headers/912691

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the latest nightly build
• My request contains all necessary details

[matthijskooijman]

I believe there are open issues about this already, did you look for them?

In any case, I'd really like to see the IDE support a libraries folder (as well as a hardware folder) inside the sketch directory, as you suggest here. That really helps to distribute a self-contained sketch, which is useful in some cases.

[per1234]

I looked for other open issues and found a couple related things, though none that would make this a duplicate:

• Add support for zipped sketch with embedded libraries arduino-builder#223 - Adds the requested feature, but only for zipped sketches.
• Support .inoz sketches Arduino#7555 - Doesn't seem to add anything of value beyond the information that was already in the Add support for zipped sketch with embedded libraries arduino-builder#223 thread.

---

The addition of recursive compilation of the src subfolder of the sketch folder does give us this feature to some extent. However, I suspect @hoblins is already aware of that option from their browsing of the forum discussions and found it to be unsatisfying.

The problem with bundling libraries in the src subfolder is that all use of the #include <foo.h> syntax in the bundled libraries for files in the src subfolder, which would work fine if the libraries were installed normally, must be changed to use relative paths to those files. Best practices would be for libraries to not use the #include <foo.h> syntax for internal dependencies anyway so I don't much mind changing those, but when you have dependencies between bundled libraries it gets ugly (e.g. #include <foo.h> -> #include "../../foo/src/foo.h").

[alandsidel]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck. I have maybe half a dozen projects that need to #include "../../includes/somefile.h" and there doesn't seem to be any way to convince the arduino compiler to do this. Even just reverting to #include "somefile.h" and then making that file a symlink to the real file doesn't work.

[matthijskooijman]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck.

Huh? But how would the feature requested by this issue help here? This issue requests that a copy of a library inside the sketch folder can be used just like a normal shared library. For your usecase, it sounds like you just need to make your "core private headers" into a library, install it into your sketchook "libraries" directory along with the other libraries, and then you can just include these from all sketches that need them?

[alandsidel]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck.

Huh? But how would the feature requested by this issue help here? This issue requests that a copy of a library inside the sketch folder can be used just like a normal shared library. For your usecase, it sounds like you just need to make your "core private headers" into a library, install it into your sketchook "libraries" directory along with the other libraries, and then you can just include these from all sketches that need them?

You're right, the issue as it stands wouldn't be helpful. I was referring to per1234's message a few up where he mentions e.g. #include "../../foo/src/foo.h" which itself wouldn't work either.

My personal goal is to get away from using the Arduino IDE and to VS Code for a variety of reasons, so adding to some IDE specific library manager doesn't seem like a good path forward. The forums have indicated to me that including with relative paths simply doesn't work in the Arduino IDE because the files are copied somewhere else prior to compilation, and it seems like the VS Code arduino plugin seems to do the same thing (I guess?) before calling avr-gcc, or however it enters the ecosystem's toolchain.

[per1234]

I was referring to per1234's message a few up where he mentions e.g. #include "../../foo/src/foo.h" which itself wouldn't work either.

I should clarify that when I mentioned #include "../../foo/src/foo.h" in my previous reply, that was referring to the #include directives in a bundled library that has a dependency on another bundled library, not to #include directives in a sketch.

I'll provide a more detailed explanation:

Let's say you have a sketch with a structure like this:

MySketch
|_ MySketch.ino
|_ src
     |_ bar
     |    |_library.properties
     |    |_src
     |        |_ bar.h
     |_ foo
          |_ library.properties
          |_ src
               |_ foo.h

With a normal library installation to the libraries subfolder of the sketchbook, if the "bar" library has a dependency on the "foo" library, it can just use #include <foo.h> and the Arduino library dependency resolution process automatically finds the "foo" library. However, if I want to bundle the "bar" and "foo" libraries with my sketch, as in my example structure above, then I must edit the "bar" library and modify #include <foo.h> to provide the relative path to foo.h: #include "../../foo/src/foo.h".

The forums have indicated to me...

Here's the forum thread @alandsidel mentioned: https://forum.arduino.cc/index.php?topic=699636

[Steve132]

I implemented this feature and made a pull-request arduino/Arduino#11110