Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repository structure overhaul #776

Closed
eitsupi opened this issue Mar 24, 2024 · 5 comments
Closed

Repository structure overhaul #776

eitsupi opened this issue Mar 24, 2024 · 5 comments
Assignees
Labels
CI documentation Improvements or additions to documentation enhancement New feature or request pre-built images Related to pre-built images rocker scripts Related to rocker scripts
Milestone

Comments

@eitsupi
Copy link
Member

eitsupi commented Mar 24, 2024

I will be working over the next few weeks on the repository overhaul that was discussed in several issues.

The current repository structure was originally designed to be updated manually, and I forced it to be changed to be updated automatically several years ago (#181), so it is not optimized for automatic updates. (Files that should be updated manually and files that are automatically generated are not completely separated.)
And, because Dockerfile is generated from json and R scripts, it is not possible to generate complex Dockerfiles that incorporate multi-stage builds.

Therefore, rewrite the structure assuming automation as follows:

  • Dockerfile and bake files are generated by pouring variables into manually written templates with a template engine (The glue R pakcage).
  • Variables are stored in TSV and JSON files and are automatically generated daily by CI.
  • Completely rewrite the Dockerfiles to share layers instead of hierarchy.
    Now:
    FROM rocker/r-ver:4.3.3
    RUN ...
    Future:
    FROM ubuntu:jammy
    COPY rocker_scripts/install_R.sh /rocker_scripts/install_R.sh
    RUN /rocker_scripts/install_R.sh  # <- same as `rocker/r-ver`
    RUN ...
    COPY rocker_scripts/ /rocker_scripts/

PoC can be found here.
https://github.com/eitsupi/rocker-versioned-next

@eitsupi eitsupi added the CI label Mar 24, 2024
@eitsupi eitsupi self-assigned this Mar 24, 2024
@eitsupi eitsupi pinned this issue Mar 24, 2024
@eitsupi eitsupi added documentation Improvements or additions to documentation enhancement New feature or request rocker scripts Related to rocker scripts pre-built images Related to pre-built images labels Mar 24, 2024
@eitsupi eitsupi mentioned this issue Mar 24, 2024
@eitsupi eitsupi added this to the R 4.4.0 milestone Mar 24, 2024
eitsupi added a commit that referenced this issue Apr 7, 2024
Related to #776, #614, #779.

List images which the definition file has already been deleted in a
separate table.
@eitsupi
Copy link
Member Author

eitsupi commented Apr 14, 2024

Sorry for the slow work. I have not given up on completing this this week and here is the work in progress.
eitsupi/rocker-versioned-next#1

I am thinking of making the following changes:

  • rocker/binder and cuda images are excluded from the central bakefile.
  • latest-daily images are being removed as they have not been able to build recently.

@cboettig
Copy link
Member

All for removing latest-daily. why exclude rocker/binder though?

@eitsupi
Copy link
Member Author

eitsupi commented Apr 15, 2024

why exclude rocker/binder though?

As I wrote here, I think the current build strategy is not ideal for rocker/binder.
#614 (comment)

@cboettig
Copy link
Member

💯 right, sorry I got confused between the two threads. I've wondered if we should be pinning python versions in rocker/binder, but the rationale there given the recent security patch etc makes sense, and easier to do this with the build strategy you suggest in #614 (comment) !

@eitsupi
Copy link
Member Author

eitsupi commented Oct 13, 2024

I forgot to close it.

@eitsupi eitsupi closed this as completed Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI documentation Improvements or additions to documentation enhancement New feature or request pre-built images Related to pre-built images rocker scripts Related to rocker scripts
Projects
None yet
Development

No branches or pull requests

2 participants