SPIFFS and LFS-free Fast Lua Source Loader

[TerryE]

NewFeature / Technique

I am not sure whether is an application note or an example of an add-on project but I thought that I should raise this issue to get feedback from the other contributors

Justification

At them moment we need a filesystem in place and some files downloaded to be able to bootstrap up and provision a new module. Whilst this can be done using esptool and other downloaders, these are not project maintained and so for many developers who are tooled up with python, spiffsimg, etc. getting your project bootstrapped can involve some hurdles.

I have been experimenting with an alternative bootstrapping approach that might have some merit (I am actually using it on my automated test harness.) In essence I have taken a set of modules, run them through LuaSrcDiet and then convert the output compressed Lua source a C string and included them into a C module, so that fast.load('name') returns an albeit RAM-based module name:

static int fast_load (lua_State *L) {
  static const char *const opts[] = {"HTTPloadfile", "telnet", "hello", NULL};
  static const int optsnum[]      = {0,              1,        2 };
  static const char *const src[]  = {HTTPLOADFILE,   TELNET,   HELLO};
  int i = luaL_checkoption(L, 1, "HTTPloadfile", opts);
  int status = luaL_loadbuffer(L, src[i], strlen(src[i]), opts[i]);
  if (status != LUA_OK) lua_error(L);
  return 1;
}

My current fast module adds about 4Kb to the image. So:

• You can use fast modules without LFS or a file system set up.
• You would typically start one up interactively or use the node.startupcommand() to do this automatically after boot. So fast.load'telnet'(ssid,pwd) will start up a telnet server, for example.
• The module only uses RAM resources if you fast load it.

Squashed Lua is incredibly dense (the C string containing the telnet server is under 1Kb Flash).

At this moment my ideas are very much in a formative stage, but I have floated this for feedback and comment. See my gist Fash Lua module loader.

I am also going to look at adding an FTP implementation, but the current version is really designed to run out of LFS, I need to tweak this to do some lazy loading to reduce the RAM footprint for RAM only running.

[vsky279]

@TerryE Terry, why not to store directly the compiled Lua code in src?

[KT819GM]

From my point view of this looks incredible useful feature for something like OTA (or startup failure recovery) implementation where you could do full format of spiffs, download init.lua (or whatever startup type you use), download lfs.img, do restart and reload *.img.

[TerryE]

@vsky279, Funny you should mention that, as I was going through the same thoughts myself. There are two advantages of using the source:

• It is quite easy to code Lua that will run happily in Lua 5.1 and 5.3 so the module will work with both variants. This isn't the same with compiled LC formats, since these aren't compatible across versions. I would need to include two versions and use a version test to decide which to load.
• The source form (at least if compressed using LuaSrcDiet) is maybe 30-40% smaller.

However, the disadvantage of the source is that it needs to be compiled on the ESP to load, and compilation has quite a RAM overhead, limiting the maximum source file that I can load. The current FTP server was really optimised for LFS use, and so is a single 500 line file. It will load and run from RAM but only if it has been first compiled using luac.cross with the debug stripped.

I've been going through the code looking to see the simplest way to partition it, but at moment it looks as if this is doable and I'll update my gist later today.

[HHHartmann]

I've also been thinking about something like this.
Why not have luac.cross compile the sources as needed (5.1 or 5.3 / integer or not) before including them in the firmware.
This could also help in hybrid modules as planned in #2819 to tie the Lua part better to the C part.
If you include the compiled Lua this could be executed out of flash just like LFS I guess. Or am I missing something?

[TerryE]

@HHHartmann see my last comments. Also the dump format is not the same as the internal binary format. The file format is byte encoded and a lot more dense. Yes there is nothing stopping you putting the binary format into a byte array and directly loading it, but as you say the module would also need the 3 different variants (5.1-float, 5.1-int and 5.3) for the module to be runnable across al 3 platforms.

Whatever happens the executing code will still run out of RAM, and so will eat into the ~44kB heap space. If you need a large app then LFS is the only way to achieve this.

I am thinking of a different sweet-spot: someone has just download a bare cloudbuilder image and flashed it to an ESP8266 module with no SPIFFS or LFS set up and they want to start FTP'ing or a terminal session into it, etc. In practice this would be in a <restart>, <do something> cycle so as long each "something" fits in RAM it is doable.

[pjsg]

Something like https://github.com/moononournation/nodemcu-webide might be a good candidate -- I haven't tried it, but it would be amazing to bootstrap into it.

[HHHartmann]

@TerryE Terry, why do you need all different variants. For one specific firmware you need one matching variant which can be built on the fly using luac.cross. No need to embed all three variants in a specific firmware. A bit tricky to convert a binary file in a byte array but its not impossible.

I also haven't gotten the part about the different internal byte format. luac.cross can build it as LFS image, so why not as single file format. Or just LFS images for each module. The resulting string might need to be aligned around some word boundaries then.

[TerryE]

Why do you need all different variants?

Why way the Lua C API works is that there is a high commonality between the APIs if you stick as per the guidelines to lua.h and lauxlib.h. Because it is quite straightforward to use the same source code for both LUA=51 (int and float) and LUA=53 builds, this is how all modules are now compiled. In my suggested approach the fast module is just stored in the git repo as a separate module in its own right and gets included using the current make rule, and so we could for example add it simply to cloud builder.

A bit tricky to convert a binary file in a byte array...

Not really. That's what xxd -i does for example. No, IMO the trickier bit is how and when to automated the creation of the fast.c file and still keep the base Lua files usable as buildable into LFS images. I suggest than only the maintainer faces these issues and only updates the app/modules source if any of the dependent Lua source files have changed.

I also haven't gotten the part about the different internal byte format.

I changed the LFS format for Lua 5.3, partly because the LC format itself changed between versions but also to facilitate in-ESP LFS building which I still plan to roll out for 5.3 versions, so LC byte-stream code is 5.1-int, 5.1-float and 5.3 specific. If we changed the make process to build this one module during the make, then maybe we could get away with this, but I for one am not going to make my life difficult by attempting this. Perhaps you can take a look at my implementation and try your own variant.

For the avoidance of doubt, there is no readily feasible way that we could include executable Proto + TValue + TString + strt data in a module. Achieving this would be more complex than a fresh LFS implementation.

The resulting string might need to be aligned around some word boundaries then.

I am not sure why you say this. The LC format is just a btye stream that lundump.c loads into RAM. The only of undump vs compile is that undump is faster and doesn't require the temporary RAM for compilation. (This last is module-size dependent but is < 10Kb for typical smallish source files.)

Extending the make process to dynamically build modules is a risky path to go down and this increases the complexity of makes for every developer. Bad idea, IMO.

[HHHartmann]

Terry, thanks for your answer. I will try and understand why this is not going to be an easy task. I seem to have a quite incomplete view of the LFS implementation.

I think adding the complexity to dynamically build such modules might well be worth it and not every developer has to deal with it.
Besides there are some other not so obvious mechanisms in the build process already which I had do figure out at some point. But I only had to do this after some time and could live without for quite some time. So yes, it would be another mechanism, but as long as it runs it does not disturb anybody.

For the time to be (which might be rather longer or infinite than shorter if I want to get this fixed) your approach seems perfectly fitting and would allow for a startup system which is not dependent on LFS OTA success.

[TerryE]

The ideal of this is that if you have the fast module loaded then even if you have an empty FS and LFS, then you can still do a fast.ftp:open(....) after restart at the interactive prompt to be able to set up your FS. Ditto telnet

[TerryE]

A quick progress update. See my gists fast.c and the conversion utility lua-to-c-include.lua. The conversion utility is documented in code, and the overhead of running it is quite good.

This version of the FTP server includes the --[[SPLIT xxx]] markers parsed by the conversion utility. This mean that it can either be compiled as a single module for including in an LFS or it is split into separate overlays for fast loading using the lazy loader:

if FAST then 
  return setmetatable(FTP,{__index=function(self, name) return fast[name] end})
end

Note that the local FAST is set to 1 on all fast loaded source, so the module omits the __index meta-method when loaded into LFS. In this case I've split all of the FTP server main functions and these are loaded lazily as ephemeral functions. The runtime overhead is small enough not to need be too clever:

> l={'ftp','ftp-open','ftp-close','ftp-dataServer','ftp-ftpDataOpen','ftp-createServer',
  'ftp-processCommand', 'ftp-processBareCmds','ftp-processDataCmds','ftp-processSimpleCmds'}
> for _,m in pairs(l) do a=fast.load(m) print(node.heap()) a=nil collectgarbage() end
Module ftp compiled in 9 mSec
37248
Module ftp-open compiled in 11 mSec
37120
Module ftp-close compiled in 11 mSec
37352
Module ftp-dataServer compiled in 9 mSec
38360
Module ftp-ftpDataOpen compiled in 42 mSec
31928
Module ftp-createServer compiled in 35 mSec
32288
Module ftp-processCommand compiled in 9 mSec
38656
Module ftp-processBareCmds compiled in 15 mSec
36736
Module ftp-processDataCmds compiled in 37 mSec
32584
Module ftp-processSimpleCmds compiled in 22 mSec
36752
> =node.heap()
40616
> =fast.load'doesnotexist'
nil

A few more i's to dot, and testing shake down before I make the PR.

[TerryE]

If you want to try this out, then my gists include the updated version of ftpserver.lua, the fast.c module and the host utility lua-to-c-include.lua. If you want to use this last then you will also need LuaSrcDiet installed on your PC. I had to restructure the ftpserver a little:

• The source now includes --[[SPLIT XXXX]] directive to tell lua-to-c-include.lua where to break the code into source chunks.
• Each of main FTP methods is a separate lua source taking the FTP table as its object parameter. the FTP table has an __index method which loads the method on demand. These don't persist in RAM and have to be reloaded on each use. (This load is reasonably fast (10-30 mSec) so this isn't an issue). This overlaying leads about 20Kb headroom on the heap.
• In the original LFS version, a lot of shared variables were upvals to module locals. any with scope across methods have now been moved to object fields, so references to user now become this.user, and so on. Note that the module will still compile happily as a single source file for running out of LFS.
• I use luac -l -l and luacheck to make sure that I've got my scoping tight and everything is GCed as I go along, So doing an FTP:close(), for example results in everything being returned to available heap.

So in my case restarting the ESP and doing this works fine

terry@ellison8:~/nodemcu-firmware$ miniterm /dev/ttyUSB0  115200
--- Miniterm on /dev/ttyUSB0  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
fast.ftp():open('terry','terry','ssid','ssidpwd')
> .......Welcome to NodeMCU world	39448	192.168.1.200	255.255.255.0	192.168.1.1
=FTP
table: 0x3fff0ba8
> -- lots of FTP use
> FTP:close()
> =node.heap()
40504

@nwf @HHHartmann , I would appreciate you guys taking another look and giving me your reactions. Anyone else welcome :)

[TerryE]

It also struck me that most new developers aren't familiar with the more esoteric features such as using node.getpartitiontable(),node.setpartitiontable(),node.node_startupcommand() or even node.flashreload(). Maybe we should wrap all of these up plus setting default ssid and ssiddpwd, etc. into a single Lua script so that beginner can just load a cloud builder image and type fast.config() to get a simple menu system to configure a blank ESP, start up an ftpservice, (load some files), then reload the LFS.

[TerryE]

I've been playing with this a bit more, and I realise that this fast utility could be extended to be really useful for Lua developers. For example:

• fast.get('ftp/ftpserver'). fast.get() loads the latest version of a lua_modules file into SPIFFS.
• fast.split(module). Even through large modules like ftpserver are optimised for LFS use, I tweaked the source (about a 5% size overhead) to make the loading of all of the methods optionally dynamic using --[[SPLIT xx]] comments. These are ignored for compiling into LFS, but this fast.split() will split and compile the methods into separate LC files. This allows non-LFS developers to use services such FTP from SPIFFS as well as embedded into fast.
• fast.dir(pattern). A synonym for for k,v in pairs(file.list(pattern) do print (k,v) end. Something I do a lot and this uses less typing.
• Any other suggestions welcome.

[TerryE]

I am inclined to generate a PR for this one simply because it is such a bloody useful bootstrap for provisioning the ESP: include the module and build the firmware. You can enable ftp with a single typed line, and do all of the provisioning drag and drop from the PC.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TerryE]

leave open