on-demand random access

[remifontan]

Having in mind a texture engine such as OpenImageIO, I was looking through the documentation for a way that would let me on-demand load part of an exr . Basically, out-of-core access to a mipmapped exr.

I Looked in the exr::block and saw the AllChunksReader and FilteredChunksReader structs. If I understand correctly those two do consume exr reader when being constructed. And I suppose it means an exr can only be iterated once. To reload some other tiles, another FilteredChunksReader would have to be created, which would mean reopening and parsing metadata and headers.

Would there be a way to achieve on-demand loading of part of the an exr while avoiding the overhead of reopening the file and recreating a block::Reader ?

Regards.

[johannesvollmer]

Hi! Your observations seem to be correct.

It was planned to support this exactly as you described it, so I'm confident it's possible to to so. Unfortunately, the library does not yet offer a nice abstraction for your use case. But the basic building blocks are there, so you should be able to do it yourself.

About the block::Reader:
We can make the reader fields public. In that case, you can construct a reader from an existing meta data struct. That way you can construct a new block reader without reading meta data from the file again.

Unfortunately, that might not be enough though. Because with this approach, it will still construct a brand new image each time. You can try to find a way around that, but I'm not sure.

I'll describe the "correct" way to do it, it will be a lot of work though. You might be able to find a simpler way.
If you encounter some members or functions that are private right now, let me know, and we make them public. I'll help you with any questions.

To understand how loading arbitrary pixel tiles on demand could be achieved, let's look at the structure of an exr file.

An exr file consists of these three components, stored sequentially, right after another, in the file:

1. fancy meta data, including layer names, channels, layer positions, mip map count, and more
2. offset tables, in increasing-y order (indicating what byte index a given pixel chunk starts)
3. all the pixel data chunks, in arbitrary order

You need to prepare the following state (see block::Reader):

1. Read MetaData from the byte stream (using MetaData::read_validated_from_buffered_peekable ) (see here)
2. Read the offset tables from the byte stream (using MetaData::read_offset_tables) (see here)
3. Create your empty image based on the meta data

Keep those. Keep the byte stream open. Then, you are ready for loading any chunk you desire:

1. Based on your needs, compute which pixel chunk you want to load from the file (see metadata::Header for some helper functions)
2. In the offset tables, lookup where the chunk is located in the file (by computing the index of your block in increasing-y order)
3. Seek the byte reader to that position
4. Load that chunk from the byte stream (using Chunk::read) (see here)
5. Decompress the chunk using UncompressedBlock::decompress_chunk (see here)
6. Insert the chunk into your image.
   Now, the algorithms that know how to put the block into the image are in the image::read modules. You can pass your blocks into the same image reader that the library uses, see here. However, I think it's not possible to access the temporary partial image inside of the reader, because it's probably private. You might need to do your own. In that case:
7. Read the pixels from your decompressed block (see here).
   A block stores scanlines, and each scanlines contains all the samples of the line for each channel.

Calculating the correct indices get complicated if the file contains mip maps, or subsampling, or data windows and layer positions. That would be the main benefit of having an abstraction.

I'm interested to add a nice abstraction for this to the library, but it will require some non trivial architectural work... so it's unclear whether I can do it in the near future. Do you want to attempt it yourself?

[remifontan]

Thanks for your prompt reply. I'll take a closer look at exrs and at your suggestions.

[johannesvollmer]

I'm interested in your use case. Do you need multiple layers? Only rgb(a) or also custom channels? What shape should the image be in the end, some kind of RGB(A) GPU texture buffer? What structs will you use in the end?

[remifontan]

This is very much exploratory at the moment.

I have in mind experimenting with sparse virtual texture on GPU, and perhaps on-demand texture streaming for off-line renderer. Something like OIIO's texture cache is providing.

I would like to retrieve a single tile or a set of tiles from an exr, uncompressed, and store them in a tile based texture cache. Therefore I'm not sure if I need to construct an Image struct. Perhaps an API to get access to uncompressed tile data is enough.

Regarding layers and channels, I haven't thought about that too much. So I suppose RGBA is probably enough for a start.

[johannesvollmer]

Thanks for the insights :)

[johannesvollmer]

Playing around with the idea of having an abstraction for this. Maybe like this?

Opening a file and loading pixel subrects on demand. This assumes the full texture is allocated in memory in full resolution. Is this useful?

// this opens the file. it does not reading any pixels yet. the api is the same as the other api. it contains the full resolution image, but it will be empty at first.
let mut lazy_loader = exrs::image::lazy_open_rgba(
    BufRead::new(File::open("large-file.exr")),
    
    // how to construct your image. 
    |size, _channels| MyGpuImage::empty(size), // question: can you always immediately create a buffer? maybe gpu needs async support?
    
    // how to insert a pixel into your image
    |my_gpu_image, xy, (r,g,b,a)| my_gpu_image.set_pixel(xy, r,g,b,a),
);

// later ... 
my_app.on_view_changed(|view_rectangle_bounds|{
    lazy_loader.load_section(Bounds::DisplayWindow(view_rectangle_bounds)); // todo: this should be async, as exrs needs to decompress the block, which can take some time.

    let indermediate_image: &MyGpuImage = &lazy_loader.intermediate_image().layer.data;
    gpu_update_texture_buffer(main_texture, indermediate_image);
})


// dropping the lazy loader closes the file.

Maybe a low-level abstraction would be better (is more flexible and requires not too much code):

let mut seekable_bytes = File::open("large-file.exr");
let lazy_reader = exrs::image::lazy_read(&mut BufRead::new(seekable_bytes));

let mut my_gpu_image = MyGpuImage::empty(0, 0); // could also pass block_info.size already, but it would be too large


// later ... 
my_app.on_view_changed(move |view_rectangle_bounds, mip_level|{
    my_gpu_image.allocate_section(view_rectangle_bounds, mip_level);

    lazy_reader
        .read_pixel_section_rgba(
              &mut seekable_bytes,              
              LazyLoad::OnlyNewBlocks, // or LazyLoad::NewAndExistingBlocks

              // todo: can the following two simply be specified by passing a block index?
              Bounds::DisplayWindow(view_rectangle_bounds),
              Level::MipMap(mip_level),

              // pixel closure is called for each pixel in the section (might come from multiple exr blocks)
              move |x,y, (r,g,b,a)| my_gpu_image.set_pixel(x,y, r,g,b,a) // is capturing this mut reference possible with async?
        )
        .await
        .unwrap(); // returns Err when pixel section is not inside the image

    my_gpu_image.send_to_gpu();
});

Or even lower level:

let mut seekable_bytes = File::open("large-file.exr");

// this reader remembers meta data and offset tables. so it knows which blocks to load for any given pixel section. 
// it can remember which blocks are already loaded. it does not decompress the blocks itself.
let lazy_reader = exrs::image::lazy_read(&mut BufRead::new(seekable_bytes));

let mut my_sparse_texture = SparseTexture::empty(); // could also pass block_info.size already, but it would be too large

// later ... 
my_app.on_view_changed(move |view_rectangle_bounds, mip_level|{
    let decompressed_blocks: AsyncStream<UncompressedBlock> = lazy_reader
        .async_pixel_section_blocks(
              &mut seekable_bytes,              
              LazyLoad::OnlyNewBlocks, // or LazyLoad::NewAndExistingBlocks

              // todo: can the following two simply be specified by passing a block index?
              Bounds::DisplayWindow(view_rectangle_bounds),
              Level::MipMap(mip_level),
         );

    for block in decompressed_blocks {
         let unpacked_rgba_buffer = block.read_all_pixels_rgba(PixelVec::<[f32,f32,f32]>::new(), PixelVec::set_pixel).unwrap(); // todo: reuse pixelvec allocation
         my_sparse_texture.insert_rgba_buffer_section(view_rectangle_bounds, mip_level, unpacked_rgba_buffer);
    }
});

Most certainly, those two abstractions can coexist. The first on builds on the more modular second one.

Would any of this work for your use case?

[johannesvollmer]

The current version allows anyone to use async if they want, by providing an iterator over chunks (they are read from the file every time you call
.next())

[remifontan]

It looks good. I'm going to test it over the next few days. Thanks very much for spending the time to write this prototype.

I'll let you know how it goes.

[johannesvollmer]

I have not tested this at all yet, I'm certain there will be bugs still, I will add tests. But we can already test and experiment with the usability of the API.

For example, the lazy reader right now consumes the file reader - maybe we don't want that, and instead keep ownership of the file, as a user of the library? Maybe we can simplify the meta data borrowing, also by not keeping ownership?

[remifontan]

For example, the lazy reader right now consumes the file reader ...

that's an interesting idea.

Thinking of a multi-threaded environment. I was wondering how it would look like to get multiple threads to load different tiles from the same exr.

With your suggestion, the file reader is simply borrowed by the lazy_reader. Could the lazy_reader be made Send+Sync so that it could be shared across threads?

I suppose it means using an immutable borrow for load_all_chunks_for_display_space_section(&self, ...) ?

[johannesvollmer]

Yes. The only reason this is mutable, is because it reads the file, which mutates the file reader. Pulling the file out of the lazy_reder would mean more housekeeping for the user, but also more flexibility, because then it can be shared across threads easily. However, it's unlikely that multithreaded disk access speeds up the reading process. The decompression is the CPU-bound step, which can already be multithreaded right now.

I've testet the implementation, it's not correct. but it's not panicking either. Also added some tests for the rectangles, and fixed the bug you found. Edit: my test was wrong, seems like it works.

[remifontan]

It's been a few days, I thought I would post some feedbacks.

I've started to use your branch on my personal project, I can load pixels from exr on demand, it seems to work as expected, nice work! ✨

I realised, the API to select which channel to load was not as useful as I hoped. While I can understand it is very handy when we know which channel to load up front, and by name, but in some cases, it feels there's a need for an additional way to get to the channels. What about following use cases.

• user already knows the channels indices, no need to pass them by name anymore?
• user wants to load every channel

At the moment I am loading all channels 4 by 4, and using ReadOptionalChannel structs.
What about a way to iterate every channel of an UncompressedBlock?

Note, I am making an assumption here. I am assuming that when a block is uncompressed, all channels in that block are uncompressed. Even if the channel extractor is only selecting a subset of the channels.

For example, I might choose to extract channel "R", I am working on an ImageViewer and the user wants to see channel "R".
Later on, the user chooses channel "B". I suppose that block will have to be loaded again and uncompressed so that channel B can be extracted...
I was hoping to avoid such overhead by always loading all channels from a given block, which I could then cache on my side, in my TileCache.

I hope that made sense. :-)

[johannesvollmer]

Thanks for your feedback! I appreciate it.

Note, I am making an assumption here. I am assuming that when a block is uncompressed, all channels in that block are uncompressed. Even if the channel extractor is only selecting a subset of the channels.

That is correct.

user already knows the channels indices, no need to pass them by name anymore or
user wants to load every channel

I agree, this must definitely be possible. The code for this is already in the repository, but some minor changes might be necessary to make it nice to use.

For example, one important function is Uncompressed block::lines for a helper method that shows you all the lines in a block. However, this is untyped data, and you would still have to cast that into F32 or F16, depending on the channel type. And it doesn't convert the data to f32 if you need that, and it doesn't do subsampling.

I'll have a look into how we can improve it :)

[johannesvollmer]

I've pushed another example to the branch. It shows the use of a new function: block.unpack_channels. It simplifies de-interleaving the block, extracting one mini image per block, with strong typing. Would this help your use case?

[remifontan]

Hi,

It's been a while since I checked your branch. I resumed my experiment and noticed something about function load_all_chunks_for_display_space_section(...) that I'm very sure.

This is in the context of accessing a mip level which is not level 0.

The current block filtering logic is comparing the requested pixel_section - with the result of get_block_display_window_pixel_coordinates(...).

Checking what get_block_display_window_pixel_coordinates does. it seems to take the block size for givel level (``get_absolute_block_pixel_coordinates(...)) and translate it at the display_window origin. see get_block_display_window_pixel_coordinates(...)`.

This does not seem to take into consideration the mip level.

pub fn get_block_display_window_pixel_coordinates(&self, tile: TileCoordinates) -> Result<IntegerBounds> {
        let data_coords = self.get_absolute_block_pixel_coordinates(tile)?;
        Ok(self.get_display_window_from_data_window(data_coords))
    }

This makes me realised that I"m not sure how to use function load_all_chunks_for_display_space_section(header, (level, level), pixel_section) in the context of accessing mip levels.

example:

• we have a 64x64 image - mipmapped with tile size 32x32
• user wants to access the pixel from level 1 that corresponds to pixel (32,32) in level 0

I believe the user would expect to receive the single block stored at level 1. Conceptually, that block covers the whole image in this example. But in current implementation, get_block_display_window_pixel_coordinates(...) will return its display_window bounds as being {(0,0), (32, 32)}. And therefore the block filtering logic is going to miss it and return no block.

• IntergerBounds{pos(0,0), width(32, 32)} does not intersect IntergerBounds{pos(32,32), width(1,1)}

What would be the pixel_section that we need to pass to load_all_chunks_for_display_space_section() ? Should we express pixel_section in the level 1 space? meaning it would have been scaled down (taking rounding mode in consideration)?
that would give me a pixel_section = (0,0), (16,16).

Or should pixel_section be defined in the level 0 space? Then, isn't get_block_display_window_pixel_coordinates() incorrectly handling mip level? not that I know what "correctly handling mip level" would be in practice.

cheers.

[remifontan]

I just noticed this function: load_all_chunks_for_layer_space_section(). that's perhaps what I was looking for, I will experiment with it.