stat, fstat or f_stat?

[diyelectromusic]

I'm struggling to work out which of stat, fstat or f_stat (or indeed something else) to use to get the size of a file on an SD card. The file is opened using fopen and that seems to work fine. I see that the library is provided by circle-stdlib/newlib, but with all the architectural options in these external projects, I'm struggling to work out what is supported in circle on the RPi.

I've tried using sys/stat,h and stat,() but that seems to be returning an error for me at the moment. Before I get too far into debugging it, I just wondered if there was any advice on the preferred method to use.

Thanks,
Kevin

[rsta2]

Hi Kevin, when you use circle-stdlib, you should use stat(), otherwise f_stat() for plain Circle with FatFs library.

Rene

[diyelectromusic]

Ok, many thanks. I tried stat but it always seemed to return an error. Now I know that was correct I'll take a closer look and try to work out what was going on!

Thanks,
Kevin

[diyelectromusic]

I'm still digging around with filesystems and trying to get to grips with the different options. On MiniDexed we're finding that it can be a little slow to load in lots of small files (which is sort of expected), but curiously it also appears to be slower for files that are indexed later in the directory. Some discussion here: probonopd/MiniDexed#456 (reply in thread)

Is this expected behaviour? i.e. is this a symptom of serially accessing the SD card or FAT data structures for example, or could there be something resetting that shouldn't need resetting (like re-seeking from the start of a file/structure every time)?

I'm digging around in all this, but to be honest, don't really know at this stage exactly what file system code is used in MiniDexed. Some was using the fatfs/ff.h interface, but some other code is using fopen/opendir too. It is the latter code showing the weird timing and I have tried rewriting it to use f_open/f_findfirst and so on and it still seems to do the same.

Thanks for any thoughts and pointers!

Kevin

[rsta2]

As far as I remember, MiniDexed uses both functions from circle-stdlib/newlib and FatFs functions directly. I think, I uses the first, everywhere where it is possible and the second only, where libraries are used, which are provided with Circle itself (e.g. addon/Properties), where circle-stdlib access is not available. When implementing features dedicated to MiniDexed I would always use circle-stdlib/newlib functions. They access FatFs functions by itself without much overhead.

The access to files and directories is implemented in the FatFs FAT Generic Filesystem Module and this module is responsible for the timing behavior, beside the SD card driver of course. Unfortunately personally I don't know the internal implementation details of FatFs and cannot explain, why the behavior, you have detected, occurs. What I know is, that for performance it is better to access a few large files, with larger read buffers, than many small files.

Some pointers:

• The FatFs FAT Generic Filesystem Module
• Performance Effective File Access in FatFs
• FatFs in Circle
• readdir() in circle-newlib

Rene

[diyelectromusic]

Brilliant - many thanks. I'll do some reading :)

Kevin

[diyelectromusic]

Ok, weirdly, if I split the files across subdirectories it is orders of magnitude faster - like keeping to a 100 files per directory means they all take seconds to load rather than minutes... There is definitely something odd about iterating over directory entries with large numbers (1000s) of files, but I can't find my way round any of that code to try to see what it might be!

More in the MiniDexed discussion :)

Kevin

[rsta2]

FatFs does not have a buffer cache. So when a file is opened, it has to read the directory from the beginning, until it has found the file name. For 1000 files this takes time.