command line tool to create "Default partition schema" on fresh disk build in distribution default

[ketonik]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Refer issue detailed here:
clearlinux/clr-boot-manager#168

Also asked question on community forums, no solutions for this yet:
https://community.clearlinux.org/t/what-command-line-needed-to-create-default-partition-schema-on-a-disk/1401

Describe the solution you'd like
A clear and concise description of what you want to happen.

I'd like a command line tool to create "Default partition schema" on a disk, providing the options and producing the same configuration the clr-installer would create. The intent is to prepare a secondary disk as a backup of the boot disk and maintain synced data backup.

To make this simpler as detailed below will burn your brains, basically a simple command tool (built into distribution by default by installer) to prepare disk that is bootable and can be used as a backup, ideally an option to later copy the contents of the root partition from another disk to this newly prepared disk, command tool name example:

clr-diskutil
Options:

-a Auto Partition (Default partition schema)
-d Destructive (Destroy the contents of the target device)
-e Encryption (Enable Disk Encryption)
rootfs=xfs
rootfs=ext4

Command Example
clr-diskutil -ade rootfs=xfs /dev/sdb

This would result in a fresh disk with Default partition schema:
VFAT(FAT32) | boot | /boot | 150MB
linux-swap | swap |   | 256MB
ext4 or XFS | root | / | remaining disk space

Maybe an Expert mode with additional options to manually define the partition schema

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Option 1-Tested and successfully working.
1-Connect to Dedicated Server Host Providers VPN.
2-Login to the IPMI web GUI from web browser to launch the JavaApplet into my IPMI Server.
3-Load the Console Redirection
4-Select Virtual Media Storage
5-Select clear-30800-live-server.iso and Plugin click OK
6-Reboot System and Press when the BIOS bootup option
7-Select the boot device to “Virtual CDROM”
8-System boots the Clear Linux image
9-Log in as root and run dd if=/dev/zero of=/dev/sdb bs=16384 status=progress
10-Run clr-installer and install to the blank /dev/sdb
11-Reboot, repeat step 5, plug in gparted-live-1.0.0-3-amd64.iso repeat steps 6-7
12-Once booted to GParted, select /dev/sdb, click on (root) partition sdb3, format as XFS or ext4 (or same as source /dev/sda3 (root)
13-Select /dev/sda, click to select the sda3 root partition right click and Copy.
14-Select /dev/sdb, click to select the sdb3 root partition, right click Paste.
Wait for data to be copied and confirmed.
15-Select the sdb3 root partition, right click select New UUID apply confirm.
16-Quit GParted, Reboot system so it boots to source /dev/sda.
17-Deploy cron jobs to run scheduled Rsync for incremental backup of /dev/sda3 to /devsdb3

Option 2-Unable to change reconfigure VFAT boot UUID resulting in:
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(0,0)

1-ssh to server run dd if=/dev/sda of=/dev/sdb bs=16384 status=progress
NOTE this creates Identical cloned Disk Identifier, UUID, PartUUIDs as detailed:
clearlinux/clr-boot-manager#168

Attempted remedy:
1-CHANGE Disk Identifier & PARTUUID

gdisk /dev/sdb

x (Expert menu)

f (Randomize the disk's GUID and all partitions' unique GUIDs)

w (write changes)

REBOOT

2-CHANGE root Label UUID

e2fsck -f /dev/sdb3 (check integrity of partition first)

tune2fs -U random /dev/sdb3 (Generate & apply LABLE UUID to the partition)

REBOOT

3-CHANGE swap UUID

uuidgen (create UUID)

swaplabel -U (UUIDGEN output) /dev/sdb2

result:
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(0,0)

Apparently efibootmgr tool has some functionality that may resolve this, but as of now unable to figure it out.

**Option 3- Clonezilla- Same resulting situation as dd cloning ie: creates Identical cloned Disk Identifier, UUID, PartUUIDs.

Additional context
Add any other context or screenshots about the feature request here.

[mdhorn]

@ketonik Thank for your request and details.

As you can see, we have quite a backlog of features and bug fixes; this won't be high on our list for a while.

But this an open-source project, so we do always try to prioritize reviewing pull requests submitted by others.

The command line already supports some of what you are requesting as we use the command line version for generating all of the published images. It might only require one new option to use the defaults sizes for /boot and swap, and then use post-install hooks for the rest.

My one concern I would be the usable model for this feature. Clear Linux* OS is a rolling release. Each day there is generally a new release, so the idea of having a "hot ready" OS image would very quickly fail far behind. I'm not sure it would provide the try of failover or recovery you might be hoping for.

[ketonik]

Hi Mark,

I understand from reading docs on https://github.com/clearlinux/clr-installer/tree/master/storage there are command line tools the installer is using, i’ve not yet established exactly which ones they are.

I appreciate this won’t be a high priority request.

You mention "Clear Linux* OS is a rolling release”

This is of particular concern and interest in understanding a little more of configuration possibilities to ensure a system is inline and automatically updated.

I decided to deploy Clear Linux to run a website, using Nginx-Mainline Web Server, PHP-Basics and MariaDB bundles my LEMP Stack.

Initially I had “swupd autoupdate”. I’ve noticed several times a week my website become inaccessible. To investigate I ssh into the machine and discover swupd autoupdates several services but some need be manually restarted with “systemctl”.

I’ve had to change policy and run "swupd autoupdate --disable” to avoid the automatic and un-notified inconvenience and unacceptable interruptions to the website’s production operational availability. This disables unfortunately the value and benefits of the rolling release model, which I particular do not want to sacrifice.

I’m familiarising myself with “clr-service-restart” https://github.com/clearlinux/clr-service-restart to try to figure out if I can configure a solution particular to my system LEMP stack so as to maintain 99.99% website uptime.

I’d need to identify what services running on my system will be subject to manual “service restarts” and have those additional services configured to be safely ‘allowed to automatically restart’ referenced by https://github.com/clearlinux/clr-service-restart#configuration

I’m new to LEMP stacks and Clear Linux and other linux; I use macOS personally. I'd need some help in identifying valid services for configuration to be allowed in the whitelist of automatic clr-service-restart.

If you know of anyone who’ll have the knowledge to assist in this i’d appreciate it. Perhaps Tutorial guide written specifically for Clear Linux, Nginx-Manline, MariaDB, PHP, that details potential issues with swupd autoupdate with indications and references to clr-service-restart configuration.

The was an LAMP guide Set up a LAMP web server on Clear Linux OS* link broken or gone
http://clearlinux.org/documentation/clear-linux/tutorials/web-server-install

Google web cache page is still available
http://webcache.googleusercontent.com/search?q=cache:z940M28R4pQJ:clearlinux.org/documentation/clear-linux/tutorials/web-server-install+&cd=1&hl=en&ct=clnk&gl=au&client=safari

ACTUALLY i Find Similar updated doc:
https://docs.01.org/clearlinux/latest/tutorials/wordpress/web-server-install.html#web-server-install

Considering the Clear Linux Website or at least https://docs.01.org/clearlinux/ is served by Nginx
I thought it appropriate for a Set up a LEMP web server on Clear Linux OS* Tutorial.

Currently my the system is in development stage, (we need to upload the Symfony App to the server and a few other components) .

Within 2 weeks I hope to go live operational, so just prior to that stage i’d want to identify and configure the system to operate uninterrupted and safely with “swupd autoupdate --enable”.

It would be awesome if someone with intimate Clear Linux LEMP familiarity could be arranged tp ssh into the machine to investigate and configure system services allowed auto restart to remain in the rolling release policy.

I can’t imagine Intel.com or Apple.com websites running an Operating System and Web server that would be subject to online uptime interruptions because of on demand automatic updates with manual system services requiring restarting.

I’m ok with a system restart once a month for kernel update but not services in operation.

So any resource, or a point of contact that you may recommend would be appreciated.

[mdhorn]

@ketonik It is my understanding that you are discussing the rolling release concerns with @ahkok offline.

[ketonik]

@ketonik It is my understanding that you are discussing the rolling release concerns with @ahkok offline.

Hi mdhorn, Yes indeed I emailed ahkok with specific concerns about clr-service-restart related to rolling release as he's the author of https://github.com/clearlinux/clr-service-restart

The intent was to have him take a look at these specific concerns.

My apologies for isolating you from the email discussion. My reason was because I made the mistake of replying in detail to your last where comment you mentioned "Clear Linux* OS is a rolling release”.

and

Considering the core objective of this thread was purely about a command line tool to create "Default partition schema" on fresh disk build in distribution default

I, in retrospect consider the clr-service-restart related to rolling release a completely seperate issue which ideally I should have cut this out into another specific issue at https://github.com/clearlinux/clr-service-restart

In some way however, the rolling release you mention does have a presence in the discussion of the command line tool to create "Default partition schema" because I'm also trying to establish a bootable backup solution. The command disk partitioning tool requested is the first part in preparation of a disk ultimately leading to a full bootable backup strategy solution with in the boundaries of rolling release. Im sure with Rsync incremental backup of the boot disk will be possible within this scope.

So to break it down.
A full bootable backup solution strategy is the goal
1-Prepare the back up media with default partition schema the same as the boot disk, without having to go through the bootable clr-installer and actually installing to the backup media hence the command line tool that could be used ssh'ing into remote server.

2-Command line tool also allow copy exact data from root partition of source to root partition of destination media (SSD). This strategy would completely avid any cloned Disk Identifier, UUID,s and PartUUID that otherwise render the backup disk inaccessible to the BIOS boot selection.

3-sudo swupd autoupdate --enable on source boot disk, and Run a scheduled Rsync script to keep root data of backup destination disk in sync.

Lets think, if swupd autoupdate --enable updated up the source boot disk and something broke and the Scheduled Rsync has yet to be been run on the bootable backup disk, we have a window of opportunity to remote to the machine, switch the BIOS Boot Priority not yet swup'd backup and bring the system back online, then later figure out what went wrong with the source disk after the latest swupd was applied.

I have two Identical Samsung 860 Pro SSD's int he remote server, surely it it common practice to have the second SSD redundantly backed up in case of issues. How this is achieved with Clear Linux is completely unknown with absolutely no documentation or discussions on strategies for this.

I mean is it typically done with mdadm to create RAID1, is so how is this done and at what stage can this be done after the initially installation of Clear Linux that is on a singe SSD in production and online?

So hope all these details give more understanding of my intended scope and help clarify any confusion made with these single isolated fragmented steps that we detailed.

phew, thats a lot of info, i'm sorry.

[mdhorn]

I'm going to move this discussion over to the main Distribution GitHub.
If we boil this down to some requirements for changes to the installer, then we can move it back here for implementation.

[bryteise]

So I didn't completely read everything here but it looks like if you want to replicate a raid1 backup, maybe what should be done is enable the use of software raid in Clear Linux (mdadm, btrfs are likely the two I'd start with).

[ketonik]

So I didn't completely read everything here but it looks like if you want to replicate a raid1 backup, maybe what should be done is enable the use of software raid in Clear Linux (mdadm, btrfs are likely the two I'd start with).

Hi,

The request is for a dedicated media partitioning tool. Just take out the media installation component of the installer and make a dedicated tool. We want the tool to do have the same functionality as we have in the installer. We want to prepare media for other purposes, not always for a new installation. Prepare the media so we can rsync data to a new storage device that will be bootable.

So it must let us do a destructive default partitioning schema and also advanced partitioning setup.

We have already a dedicated Network interface configure tool in nmtui, so do the same with the disk partitioning. The installer says it is using cgdisk so pull out that tool with all the scripts and make a dedicated tool.

What you read about the replicated raid1 is only a suggestion by others in response to the request to prepare bootable media for rsync. I don't believe raid on SSDs these days is suitable. Raid1 on SSD is very slow and unnecessary wearing of the second disk constant writes to it, it will probably fail at the same time as the source.

For now I do daily rsync of root partition BEFORE scheduled swupd, if swupd causes an issue on the running system I have the opportunity to reboot to the backup media. Provide the team with details of the issue and wait for a fix.

[bryteise]

@mdhorn I think having the installer create a commandline utility for this makes the most sense. It is an extra work item so prioritize accordingly.

[mdhorn]

I"m still not sold that this is a worthwhile tool or why this is needed.
There are recommendations on the partitions, but there are no "standard partitions" that won't change over time.

[ketonik]

This tool would allow us to prepare a fresh SSD and rsync the root partition to the new SSD.
How I resolved was to remotely via Supermicro IPMI interface, boot a Virtual USB Clear Linux install and install it to a new SSD in the remote host, just to get new UUID's on the boot, swap root partitions.

Then I needed to remotely boot to a GParted virtual disk, delete the root partition on the destination SSD, (the fresh install of CL), then copy and paste the source SSD root partition to the destination SSD then change the UUID of the root partition then reboot.

This took many many hours and complications connecting on slower VPN connection and back then the CL installer required net connection to download bundles.

With a dedicated partitioning tool bundle, I could have just ssh’d into remote host, run the this partitioning tool and make fresh boot, swap, root partitions with unique UUIDs on the new SSD. This would have taken 2 minutes tops

Then i’d run rsync to copy root data of source to root on destination at that would have taken 10 minutes.

This process could not be done with dd as dd cloned all the UUID’s and took enormous amount of effort changing them and in the end still was unable to change the boot EFI UUIDs the result was this new SSD was unable to be identified in Supermicro UEFI as a unique bootable disk, even though the UEFI shows this device on the SATA port and selected, the machines can ONLY ever boot from the defined EFI UUID which in this case both were the same, result disaster!

This may have been less of a drama if I the server machine and SSD's in my physical presence but this duty was to be performed on a dedicated server 4,000KM away with no physical access.

Intel, I guess surely have intention and expectation the requirement to deploy the OS remotely right? Well this is once scenario where we need tools to prepare media remotely for install and data transfer.

Bare Metal server ISO BTW is what i'm using.

[mdhorn]

@ketonik I don't a one-off tool is needed here.

First, I would install the clr-installer bundle on your system swupd bundle-add clr-installer. You don't need to use an installer image, you just need the clr-installer command line.

Second, use the YAML configuration file generated and saved on your active system cat /root/clr-installer.yaml as a template for your failover drive.

• Remove the bundle list from the YAML file; the installer will automatically pull in a few minimal required bundles.
• Change the name of the TargetMedia in the YAML file! This is the most important step to ensure you do not wipe out your existing installation.
  • You can use a block device alias as shown here https://github.com/clearlinux/clr-installer/blob/master/scripts/installer.yaml#L10

sed -e 's/name: sda/name: sdb/ ; /^bundles:/d' clr-installer.yaml > failover.yaml

If the drives are not exactly the same, then you can adjust the targetMedia by setting the size of the full drive and removing the size for the root partition and it will use the reset of the drive.

The just use the clr-installer command line to partition the recovery drive:
sudo clr-installer -c failover.yaml

This will partition the disk and place a very small OS image on the failover drive which you can overwrite with your rsync job.

[mdhorn]

@ketonik Does the suggestion above resolve your problem?

[ketonik]

Hi mdhorn,

It is too complicated.

This tool should achieve the task in 1 command line. macOS's 'diskutil' tool achieves this in 1 single line command.

Tool + Command + File System + Disk Name + Device
diskutil eraseDisk FILE_SYSTEM DISK_NAME DISK_IDENTIFIER

user input example for Clear Linux
diskutil eraseDisk ext4 clrbk /dev/sdb

This single line command will effectively:
Use Clear Linux's Default Partitioning Schema to:
Completely erase the existing whole disk,
all volumes on this disk will be destroyed.

Done and Dusted in 10 seconds.

Real world command example:
diskutil-list-erase-disk.png:

diskutil-list-result.png

[mdhorn]

@ketonik What tool or process would you use for other Linux distributions?

[ketonik]

@mdhorn What tool or process would you use for other Linux distributions?

I don't use any other Linux distribution.

Clear Linux only for my live server, Nginx-Mainline, PHP, MariaDB, Symfony etc.

macOS is everything else.

[mdhorn]

The ask here could be solved with writing a shell script based on the suggested solution above. I still do not see this as a part of the installer.

Give the current backlog of bugs and enhancement requests that would impact/help multiple users versus this one-off request, it will likely get no cycles from myself in the near future.

But this is an Open Source project, so we will always make time to review submitted Pull Request with changes.