Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support retrieving resources as flows #4500

Open
wants to merge 12 commits into
base: master
Choose a base branch
from

Conversation

paxbun
Copy link
Contributor

@paxbun paxbun commented Mar 17, 2024

Changes

  • Added fun getResourceAsFlow(String, Int): Flow<ByteArray>, which retrieves the resource as a flow of byte array chunks.
    • On Android & JVM targets, the flow uses InputStream.
    • On Apple targets, the flow uses NSInputStream.
    • On web targets, the flow uses ReadableStreamBYOBReader.
  • Added ResourceIOException, which wraps internal IO exceptions thrown while reading resources.
  • Modified the Gradle plugin so the generated Res class contains getAsFlow(String, Int), which invokes getResourceAsFlow.
  • Added a section using Res.getAsFlow and kotlinx.io in the files tab of the resource demo app.
  • The default requested size of chunks is 8192, which is the value of Segment.SIZE of kotlinx.io or okio.

Motivation

I am making an app that uses deep learning. Although the actual app downloads the DL model from our server, we embed the DL model into our app as a resource for internal testing. I have been handling this by implementing resource reading with platform-specific APIs. I tried refactoring this part using the new resource API, but it didn't work because the app crashed with an OOM error when I used Res.readBytes.

To prevent such an error, I need to read resources as chunks, so I need a proper IO API here. Since kotlinx.io already provides platform-agonistic IO, I tried to implement this with kotlinx.io, but kotlinx.io only provides synchronous IO and is incompatible with web targets, so I failed.

So, I came up with the idea of using Flow for this, as Flow is the only first-party Kotlin API to do something sequentially and asynchronously, as far as I know.

Example

LaunchedEffect(Unit) {
  withContext(Dispatchers.IO) {
    // copy the resource content to someFileStream
    someFileStream.use { stream ->
      Res.getAsFlow("files/bigFile.bin").collect { chunk ->
        stream.write(chunk);
      }
    }
  }
}

@eymar eymar requested a review from terrakok May 21, 2024 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant