# ubuntu-livecd

**Repository Path**: dev_ma/ubuntu-livecd

## Basic Information

- **Project Name**: ubuntu-livecd
- **Description**: Ubuntu livecd patch and config
- **Primary Language**: Unknown
- **License**: GPL-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 1
- **Forks**: 1
- **Created**: 2025-06-08
- **Last Updated**: 2025-07-13

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Ubuntu LiveCD



* [Virtual machine](#virtual-machine)
* [patch.livecd.sh](#patchlivecdsh)
   * [Dir entry of file grub.cfg](#dir-entry-of-file-grubcfg)
* [Persistent file](#persistent-file)
* [Configure live system](#configure-live-system)
   * [Disable auto login](#disable-auto-login)
   * [noeject](#noeject)
   * [Disable unnecessary services](#disable-unnecessary-services)
   * [Auto start GUI](#auto-start-gui)
   * [Install ibus](#install-ibus)
   * [Hardware clock local time](#hardware-clock-local-time)
   * [clean-tmp.service](#clean-tmpservice)
   * [snapd.apparmor.service](#snapdapparmorservice)
* [livecd.mounts.finalrd](#livecdmountsfinalrd)
* [finalrd](#finalrd)
   * [systemd shutdown scripts](#systemd-shutdown-scripts)
* [Build systemd](#build-systemd)
* [Livecd hibernate](#livecd-hibernate)
   * [Use swap partition](#use-swap-partition)
   * [Use swap file](#use-swap-file)
   * [zramswap.sh](#zramswapsh)
   * [Allow GUI hibernate](#allow-gui-hibernate)
* [Grub boot iso file](#grub-boot-iso-file)
   * [Grub check file](#grub-check-file)
   * [Chainload grub core](#chainload-grub-core)
   * [Grub UART terminal](#grub-uart-terminal)
   * [Install livecd grub](#install-livecd-grub)
   * [Build grub](#build-grub)
   * [Copy or move grub installed](#copy-or-move-grub-installed)
   * [Install gurb bootsector](#install-gurb-bootsector)
* [initrd](#initrd)
   * [initramfs.split.sh](#initramfssplitsh)
   * [patch.main.cpio.sh](#patchmaincpiosh)
   * [Create inird cpio file](#create-inird-cpio-file)
   * [Create initrd](#create-initrd)
   * [initrd script files and functions](#initrd-script-files-and-functions)
* [Lxd container and docker](#lxd-container-and-docker)
   * [Container working folder](#container-working-folder)
   * [Create container image and run it](#create-container-image-and-run-it)
   * [Lxd dns record](#lxd-dns-record)
   * [Mount in container](#mount-in-container)
* [apt-cacher-ng](#apt-cacher-ng)
   * [Move data folders](#move-data-folders)
   * [Offline mode](#offline-mode)
   * [Cache repository Releae and Packages](#cache-repository-releae-and-packages)
* [Windows NTP server](#windows-ntp-server)





**CAUTION**

**Please backup your data before run any test.** Test with a virtual machine, i.e. VirtualBox or qemu, make sure no data loss, then move to a real machine.



A live system with web browser and office tools only uses around 3G disk space, compared to Win10 60G. So that many virtual machines can run at same time.

Ubuntu livecd uses "casper" as initramfs-tool to boot live system.

View man page after boot up and login.

```
man casper
```


Ubuntu LiveCD iso files

```
https://releases.ubuntu.com/jammy/ubuntu-22.04.5-live-server-amd64.iso
https://cdimage.ubuntu.com/lubuntu/releases/22.04.5/release/lubuntu-22.04.5-desktop-amd64.iso
https://cdimage.ubuntu.com/lubuntu/releases/noble/release/lubuntu-24.04.2-desktop-amd64.iso
```

## Virtual machine

Audio plays sound from vbox vm when chose Intel HD Audio as audio controller.

Avoid hang in vbox headless detachable start: add "nomodeset" to kernel cmdline.

Avoid boot from CDROM, use IED slave CDROM drive instead of IDE primary CDROM drive.


After test with virtual disks and every thing works well, before move to a real machine, test with physical disk.

```
# vbox lists partitions of a raw disk
"c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands listpartitions -rawdisk \\.\PhysicalDrive0

# let vbox use raw disk
# partition 5 and 7 are linux partitions, and not used by Windows
"c:\Program Files\Oracle\VirtualBox\VBoxManage.exe" internalcommands createrawvmdk -filename sda57.raw.vmdk -rawdisk \\.\PhysicalDrive0 -partitions 5,7 -relative
```

Virtual machine can see all partitions, but can only read write assigned partitions, and cannot read write other partitions.

Add another virtual disk as EFI or grub boot partition if necessary, and then can boot with the physical partition.

Modify EFI mount in /etc/fstab, add "nofail" to mount options, to avoid boot blocked by missing EFI partition.


Win10 diskpart

```
( echo list disk && echo select disk 0 && echo list partition) | diskpart
```


Win10 host qemu use physical disk

```
d:\programs\qemu-w64-2.12\qemu-system-x86_64.exe -smp 2 -m 2G -drive file=\\.\PhysicalDrive0,if=virtio,format=raw,media=disk -boot order=a -snapshot
```

Refer to qemu doc file "qemu-doc.html" for help.
"qemu-doc.html" comes with installed "qemu-system-x86_64.exe".

Win10 host qemu "accel=tcg" requires turn on "Windows Hypervisor Platform" in "Turn Windows features on of off".

Virtualbox failed to start vm when turn on Windows Hypervisior Platform: VT-x is not available.

For x64 vm, vbox requires VT-x.

```
# turn off hypervisor, and reboot
bcdedit /set hypervisorlaunchtype off
```


## patch.livecd.sh

Run "patch.livecd.sh" to patch livecd iso file, change content of file "grub.cfg" and set file size.

```
# get help info
sh patch.livecd.sh -h

# patch iso file
sh patch.livecd.sh -v -p lubuntu22045 -s "resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=659456" lubuntu-22.04.5-desktop-amd64.iso

# restore iso file to original
sh patch.livecd.sh -r lubuntu-22.04.5-desktop-amd64.iso
```


Old head part of livecd iso file is backup to file with ext-name ".iso.old.head".

Ubuntu casper boot only allows persistent file in vfat partition. It has a bug, "/root/tmp/debconf.xxxx" folder is not deleted, if you check the folder in initramfs shell when use "break=init".

After patch, it allows writable-backing and casper-sn on fs more than vfat, patch to delete "/root/$DEBCONF_TMPDIR", allow resume from hibernate.


### Dir entry of file grub.cfg

```
# ls -U /cdrom/boot/grub/
fonts  grub.cfg  i386-pc  loopback.cfg  x86_64-efi

# stat /cdrom/boot/grub/grub.cfg /cdrom/boot/grub/loopback.cfg
  File: /cdrom/boot/grub/grub.cfg
  Size: 648             Blocks: 2          IO Block: 2048   regular file
Device: b00h/2816d      Inode: 2505        Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-08-07 15:52:06.000000000 +0000
Modify: 2023-08-07 15:52:05.000000000 +0000
Change: 2023-08-07 15:52:05.000000000 +0000
 Birth: -
  File: /cdrom/boot/grub/loopback.cfg
  Size: 386             Blocks: 1          IO Block: 2048   regular file
Device: b00h/2816d      Inode: 2512        Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-08-07 15:52:06.000000000 +0000
Modify: 2023-08-07 15:52:05.000000000 +0000
Change: 2023-08-07 15:52:05.000000000 +0000
 Birth: -

# trans to hex
# ( set -xv; printf "0x%04x\n" 648 386; printf "%02x " $((2023 -1900)) 8 7 15 52 05 )
+ printf '0x%04x\n' 648 386
0x0288
0x0182
+ printf '%02x ' 123 8 7 15 52 05
7b 08 07 0f 34 05

# grub-2.06/grub-core/fs/iso9660.c
struct grub_iso9660_dir {
  grub_uint8_t len;
  grub_uint8_t ext_sectors;
  grub_uint32_t first_sector;
  grub_uint32_t first_sector_be;
  grub_uint32_t size;
  grub_uint32_t size_be;
  struct grub_iso9660_date2 mtime;  // 7 bytes
  grub_uint8_t flags;
  grub_uint8_t unused2[6];
#define MAX_NAMELEN 255
  grub_uint8_t namelen;
} GRUB_PACKED;

# in grub_iso9660_iterate_dir():
        semicolon = grub_strrchr (ctx.filename, ';');
        if (semicolon)
          *semicolon = '\0';


# cdrkit-1.1.11/genisoimage/iso9660.h
struct iso_directory_record {
    unsigned char length        [ISODCL(1,  1)];   // 1 byte
...
    char size           [ISODCL(11, 18)];  // 8 bytes
    char date           [ISODCL(19, 25)];  // 7 bytes
...
    unsigned char name_len      [ISODCL(33, 33)];  // 1 byte
    char name           [MAX_ISONAME+1];   // 208 bytes
};
struct iso_ext_attr_record {};
```


## Persistent file

By default, for lubuntu-22.04-desktop-amd64.iso and ubuntu-22.04-live-server-amd64.persistent.iso, persistent files must be named "writable" and "home-rw" in a FAT32 partition when partition is not labeled as "capser-rw".

Set ext4 fs label to "casper-rw", then this partition can be used as persistent.

Ubuntu-22.04 LiveCD can use lvm partition as persistent. Recommend to save your data in other lvm partition, to expand or reduce partition size easily, add or remove disks dynamically.

Partition labled with "casper-rw" overrides "writable" file in folder specified by persistent-path (default to top folder in vfat partition). To use "writable" file in fodler persistent-path, change the partition lable to other string.

"home-rw" file is always searched and used when use persistent-path.


Web resource

```
https://askubuntu.com/questions/1269462/bios-uefi-template-image-for-booting-iso-files
```

Scripts handle persistent-path

```
initrd/main/scripts/casper
initrd/main/scripts/casper-helpers
find_cow_device()
```


At the first Lubuntu system boot up, login in as user "lubuntu" without password. Then create persistent cow file for "casper".

Example below uses LVM (Logical Volume Manager) as persistent writable-backing device.

```
mkdir s
sudo mount /dev/vglubuntu/lvlubuntu s
sudo mkdir s/lubuntu22045
sudo dd if=/dev/zero of=s/lubuntu22045/writable bs=1M count=124
#
( set -xv; f=s/lubuntu22045/writable; mkdir -p t && d=$(sudo losetup --find --show $f) && [ -n "$d" ] && sudo mkfs.btrfs $d && sudo mount $d t && sudo mkdir t/upper t/work && sudo btrfs property set t/upper compression lzo && btrfs property get t/upper compression && sudo umount t && sudo losetup -v -d $d && rmdir t; echo rt=$? )
```

Then reboot, login and check if "writable" file is used.

```
losetup
grep "\s/\s" /proc/mounts
```

## Configure live system

The configuration will be saved to persistent file or partition.


### Disable auto login

Add user and add to sudo group before disable auto login.

```
sudo adduser
sudo adduser user sudo
sudo adduser user lxd
```

For Lubuntu-22.04.3, disable GUI auto login:
Remove all lines of section "[Autologin]" in file "/etc/sddm.conf".

```
man sddm.conf

sudo crontab -e

sudo crontab -l
@reboot sh -xvc '( echo "[Users]"; echo "MinimumUid=1000" ) | tee /etc/sddm.conf' 2>&1 | logger -t cron.job.reboot.disable-auto-login

# check and change "MinimumUid" if it does not cover desired user ID
tail /etc/passwd

```

For ubuntu-22.04.5, disable GUI auto login:

```
sudo sed -i -e 's/^AutomaticLogin/#&/' /etc/gdm3/custom.conf
```


Prevent reset to empty passwd of user lubuntu.
Comment off (by # at line beginning) user lubuntu in file "/etc/passwd".

```
tail /etc/passwd
sudo sed -i -e 's/^lubuntu:/#&/' /etc/passwd
```

Disable tty auto login.

```
systemctl cat serial-getty@.service getty@.service

# Comment off all lines in file: /usr/lib/systemd/system/getty@.service.d/autologin.conf
```

### noeject

By default, auto eject cdrom at system reboot or poweroff, by script file "/usr/sbin/casper-stop". Patch it to enable "noeject".


```
cat casper-stop.noeject.patch | sudo patch -d /usr/sbin
```


### Disable unnecessary services

Disable unattended-upgrades.service, fwupd services and apt-daily services.
Download a new livecd instead of upgrade.

```
systemctl status fwupd*

( set -xv; for i in unattended-upgrades.service fwupd-refresh.timer fwupd-refresh.service fwupd.service fwupd-offline-update.service motd-news.service motd-news.timer apt-daily.timer apt-daily.service apt-daily-upgrade.timer apt-daily-upgrade.service; do systemctl --no-pager status $i; sudo systemctl stop $i; sudo systemctl disable $i; done )

# Disable auto upgrade
sudo sed -i -e 's/1\(\"\;\)/0\1/' /etc/apt/apt.conf.d/10periodic /etc/apt/apt.conf.d/20auto-upgrades

sudo sed -i -e 's/Prompt=lts/Prompt=never/' /etc/update-manager/release-upgrades

tail /etc/apt/apt.conf.d/10periodic /etc/apt/apt.conf.d/20auto-upgrades /etc/update-manager/release-upgrades

# disable motd
sudo sed -i 's/^ENABLED=.*/ENABLED=0/' /etc/default/motd-news
sudo chmod -x /etc/update-motd.d/*

# limit journal log disk usage
echo SystemMaxUse=6M | sudo tee -a /etc/systemd/journald.conf

# reduce syslog
grep -A5 -B5 'rotate ' /etc/logrotate.d/rsyslog
sudo sed -i -e 's/^\(\s\+\)weekly$/\1daily/' /etc/logrotate.d/rsyslog

# clean apt cache
sudo apt clean

# For Ubuntu live server, disable subiquity
sudo snap disable subiquity
sudo systemctl stop snap-subiquity-6066.mount
sudo systemctl disable snap-subiquity-6066.mount snap.subiquity.subiquity-server.service
```


Ubuntu live server UART console conflicts with Subiquity.

```
# serial-getty@ttyS0.service failed because of "Subiquity, the installer for Ubuntu Server ttyS0."
systemctl cat serial-getty@.service getty@.service
# comment off "[Service]" section in file:
/usr/lib/systemd/system/serial-getty@.service.d/subiquity-serial.conf
```


Start UART console.

```
# run getty ttyS0 without kernel parameter "console=ttyS0"
sudo systemctl start serial-getty@ttyS0.service

# Manually set terminal settings after login, change cols and rows to match your terminal.
# "vim" cannot display properly if terminal size set incorrectly. In PuTTy, stty rows shall be (rows reported by PuTTy - 1).
( set -xv; stty sane cols 135 rows 38 && env TERM=xterm-256color bash -i )
```



### Auto start GUI

```
# disable auto start GUI
sudo systemctl set-default multi-user.target

# enable auto start GUI
sudo systemctl set-default graphical.target

# check GUI status
systemctl status display-manager.service

# Manually start GUI
sudo systemctl start graphical.target

# Manually stop GUI
sudo systemctl stop sddm.service
```

### Install ibus

Use apt-cacher-ng server if you have.

```
echo 'Acquire::http::Proxy "http://192.168.58.211:3142";' | sudo tee /etc/apt/apt.conf.d/00aptproxy
```

Install input method in Lubuntu-22.04.3. Fcitx4 not work. Use ibus-pinyin.

```
sudo apt install ibus-pinyin ibus
```

### Hardware clock local time

Windows has bug to set hardware clock as UTC time for decades, it assumes not accross time zones a lot. Linux system by default set hardware clock as UTC time, and can be easily configured to use local time.
If a real machine swithes between Linux and Windows, hardware clock will mess up.

Let hardware clock use local time, same as Windows.

```
# web resource
https://askubuntu.com/questions/946516/how-to-tell-ubuntu-that-hardware-clock-is-local-time

timedatectl set-local-rtc 1 --adjust-system-clock
```

### clean-tmp.service

Another solution to delete "/tmp/debconf.xxxxx" folders in "cow/upper/tmp/": use systemd service "clean-tmp.service".

Install it and enable it.

```
sudo sh -xvc 'f=clean-tmp.service; d=/etc/systemd/system; [ -e "$f/$d" ] && exit 0; cp -a $f $d/ && f=$d/$f && chown 0:0 $f && chmod 0644 $f && systemctl enable clean-tmp.service'
```

### snapd.apparmor.service

In Ubuntu-22.04-live-server, "/var/snap/lxd/common/ns/" mount is required by lxd and is mounted by snap.lxd.daemon.service, which depends on snapd.apparmor.service and apparmor.service.
"/var/snap/lxd/common/ns/" is not mounted when apparmor.service failed.

```
sudo systemctl status apparmor.service snapd.apparmor.service
systemctl cat apparmor.service snapd.apparmor.service
#
# This line in apparmor.service prevent apparmor to start when run livecd. Comment off it to avoid this problem.
ConditionPathExists=!/rofs/etc/apparmor.d

# check lxd.daemon service in snap
snap list
snap services
```


## livecd.mounts.finalrd

By default, the mount point of loop device of "writable" file is hidden. It can be available by run file "livecd.mounts.finalrd" from initrd.

Copy file "livecd.mounts.finalrd" to your machine, and run command below to install it.

```
# get help information
sh livecd.mounts.finalrd --help

# run install
sudo sh livecd.mounts.finalrd install

# check installed file
ls -l /etc/finalrd/
tail /usr/share/debconf/confmodule
```

## finalrd

Lubuntu-22.04.3 shutdown bug, finalrd bug:
Lubuntu-22.04.3 systemd-shutdown dead loop if boot from livecd iso file in ntfs partition.

```
https://bugs.launchpad.net/snapd/+bug/1920107
https://bugs.launchpad.net/ubuntu/+source/finalrd/+bug/1905166
https://github.com/grml/grml/issues/206

https://github.com/systemd/systemd/issues/14410
https://github.com/systemd/systemd/pull/14409
```

A quick fix: replace "/lib/systemd/systemd-shutdown" with file "systemd-shutdown.sh".
Run the short script in its head part to install it.
"systemd-shutdown.sh" requires "livecd.mounts.finalrd".


A better fix: patch source code of systemd and rebuild it.
Patch file: systemd-249.11.shutdown.deadloop.patch

In file systemd-249.11/src/shutdown/umount.c:435, delete_loopback(), when last call "ioctl(fd, LOOP_GET_STATUS64, &info)" failed, it shall return 0 instead of 1 at line 442.

Further improvement, add a loopcnt in main() of file systemd-249.11/src/shutdown/shutdown.c:488. Force "changed=0" when loop too much.


### systemd shutdown scripts

```
# web resource
https://askubuntu.com/questions/236476/how-do-i-diagnose-hang-on-shutdown

# scripts
ls /usr/lib/systemd/system-shutdown/ /usr/share/finalrd/
sudo ls -a /run/initramfs/

man finalrd

systemctl cat finalrd.service
cat /usr/bin/finalrd
cat /usr/lib/finalrd/finalrd-static.conf /run/finalrd-libs.conf

man systemd-tmpfiles
```

## Build systemd

Systemd source code

```
https://github.com/systemd/systemd/
https://codeload.github.com/systemd/systemd/tar.gz/refs/tags/v249
https://github.com/systemd/systemd/blob/main/src/shutdown/shutdown.c
https://github.com/systemd/systemd/blob/main/src/shutdown/umount.c
```

Ubuntu systemd-249.11 source code

```
https://launchpadlibrarian.net/656926680/systemd_249.11-0ubuntu3.9.dsc
https://launchpadlibrarian.net/656926679/systemd_249.11-0ubuntu3.9.debian.tar.xz
https://launchpadlibrarian.net/656926678/systemd_249.11.orig.tar.gz
```


Extract source files.

```
( set -xv; mkdir t && cd t && dpkg-source -x ../download/systemd-249.11-0ubuntu3.9.dsc && mv systemd_249.11 .. && cd .. && rm -fr t; echo rt=$? )
```

Install dependencies, required packages are checked and reported when run build.

```
sudo apt install debhelper meson ...
```

Apply patch.

```
cat systemd-249.11.shutdown.deadloop.patch | patch -d systemd-249.11 -p 1
```

Run build.

```
( set -xv; history | tail -n 1; lsb_release -a; uname -a; systemd-detect-virt; cd systemd-249.11 && fakeroot -- dpkg-buildpackage --build=binary; echo rc=$? ) 2>&1 | tee systemd.build.log
```

Continue build after modify source file.

```
( set -xv; history | tail -n 1; cd systemd-249.11 && fakeroot -- make -w -f debian/rules DH_VERBOSE=1 override_dh_auto_build; echo rc=$? ) 2>&1 | tee systemd.build.log
```

Generate patch file.

```
( set -xv; history | tail -n1; tabs -4; diff -duwp systemd-249.11.old/src/shutdown/shutdown.c systemd-249.11/src/shutdown/shutdown.c; diff -duwp systemd-249.11.old/src/shutdown/umount.c systemd-249.11/src/shutdown/umount.c; diff -duwp systemd-249.11.old/meson.build systemd-249.11/meson.build ) 2>&1 | tee systemd-249.11.shutdown.deadloop.patch
```


Use static linked systemd-shutdown to avoid libsystemd-shared-249.so library dependency problem when run it as replacement of old "/lib/systemd/systemd-shutdown" in system reboot or poweroff.

Try run, shall fail for libsystemd-shared.

```
cp systemd-249.11/build-deb/systemd-shutdown .
./systemd-shudown
```

Check dependent library.

```
env LD_TRACE_LOADED_OBJECTS=1 systemd-249.11/build-deb/systemd-shutdown | grep libsystemd-shared
env LD_TRACE_LOADED_OBJECTS=1 deb.extracted/lib/systemd/systemd-shutdown
env LD_TRACE_LOADED_OBJECTS=1 systemd-249.11/build-deb/systemd-shutdown.standalone
```

Strip systemd-shutdown.standalone.

```
cp systemd-249.11/build-deb/systemd-shutdown.standalone .
strip -s systemd-shutdown.standalone
ls -l systemd-249.11/build-deb/systemd-shutdown.standalone systemd-shutdown.standalone
env LD_TRACE_LOADED_OBJECTS=1 ./systemd-shutdown.standalone

# static linked systemd-sysusers.standalone systemd-tmpfiles.standalone
grep standalone systemd-249.11/meson.build
ls systemd-249.11/build-deb/*standalone
```


Build systemd-249.11 in ubuntu-20.04.1, change debhelper-compat from 13 to 12 in file "systemd-249.11/debian/control":
Build-Depends: debhelper-compat (= 12),

```
( set -xv; f=systemd-249.11.old/debian/control; [ -e $f.old ] || cp -a $f $f.old; sed -i -e 's/\(\sdebhelper-compat\s(=\) 13),/\1 12),/' $f; diff -duwp $f.old $f )
```

Build failed at test-fs-util: systemd-249.11/src/test/test-fs-util.c:687

Solution: comment off this line.

```
( set -xv; f=systemd-249.11/src/test/test-fs-util.c; [ -e $f.old ] || cp -a $f $f.old; sed -i -e 's/^\s\+assert_se(st.st_blocks == 0)\;/\/\/&/' $f; diff -duwp $f.old $f )
```

Check meson log.

```
less systemd-249.11/build-deb/meson-logs/testlog.txt
```

Check built files.

```
grep -n " systemd-shutdown " systemd.build.log
file systemd-249.11/build-*deb/systemd-shutdown
file systemd-249.11/debian/install/*deb/lib/systemd/systemd-shutdown
#
file ./systemd-249.11/build-*deb/systemd-shutdown.standalone
env LD_TRACE_LOADED_OBJECTS=1 ./systemd-249.11/build-deb/systemd-shutdown.standalone
# try run systemd-shutdown.standalone
./systemd-249.11/build-deb/systemd-shutdown.standalone
```

Extract deb file.

```
( set -xv; mkdir -p deb.extacted && cd deb.extracted && dpkg-deb --extract ../systemd_249.11-0ubuntu3.9_amd64.deb . && file lib/systemd/systemd-shutdown && ls -l lib/systemd/systemd-shutdown; echo rt=$? )

# extract deb debug files
( set -xv; cd deb.extracted && dpkg-deb --extract ../systemd-dbgsym_249.11-0ubuntu3.9_amd64.ddeb .; echo rt=$? )
#
find usr/lib/debug/ -type f -exec sh -c 'objdump -t $1 | grep delete_loopback && echo $1' x \{\} \;
```


Binary patch systemd-shutdown is possible, the "return" entrance is "safe_close(*fd)" in assembly codes. Before jump to this, some register (r12d or rax) shall be patched. That will be another work.


## Livecd hibernate

***CAUTION***

***File system will be crashed if resume from a miss matched hibernate swap, i.e. fs is modifed since hibernate and before resume, by other OS or livecd.***

Hibernate requires swap partition or swap file, but not zram swap.

```
Web resources

https://www.kernel.org/doc/Documentation/power/swsusp.txt
https://www.kernel.org/doc/Documentation/power/basic-pm-debugging.txt
https://www.kernel.org/doc/Documentation/power/swsusp-and-swap-files.txt
https://wiki.archlinux.org/title/Power_management/Suspend_and_hibernate
https://askubuntu.com/questions/1534335/hibernate-from-command-line-works-but-not-from-menu
https://askubuntu.com/questions/1240123/how-to-enable-the-hibernate-option-in-ubuntu-20-04
```

### Use swap partition

```
sudo mkswap /dev/sdb8
sudo swapon /dev/sdb8
sudo blkid /dev/sdb8

sudo systemctl hibernate

# use swap partition
set resumeswap="resume=UUID=8b05f6cf-8c3b-4da8-b27c-619ee7761ae7"

# use swap file, UUID is the partition contains swap file
set resumeswap="resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=659456"

# append UUID of swap partition to kernel parameter:
console=tty console=ttyS0,115200 no_suspend_console break=premount $resumeswap

# grub set var
set iso_path=""
#set iso_path=/ubuntu-22.04.5-live-server-amd64.iso
echo iso_path=$iso_path
set isoscan=""
#set isoscan="iso-scan/filename=$iso_path"
echo isoscan=$isoscan
set krndbg=""
#set krndbg="console=tty1 console=ttyS0,115200 loglevel=7 ignore_loglevel keep no_console_suspend"
#set krndbg="$krndbg initcall_debug debug retain_initrd break=momuntroot,casper-bottom,bottom,init"
echo krndbg=$krndbg
set resumeswap=""
#set resumeswap="resume=UUID=cb198d63-b5ca-4d95-a274-41fe82e417a8 resume_offset=827392"
echo resumeswap=$resumeswap
set casperboot="boot=casper $isoscan persistent persistent-path=ubuntu22045server noprompt noeject username=xxxusername hostname=ubtsvr22045 cloud-init=disabled"
echo casperboot=$casperboot
# grub load kernel
linux /casper/vmlinuz nomodeset keep $krndbg $resumeswap $casperboot ...

# run resume in initrd premount break shell
sh -xv /scripts/local-premount/resume
```

"blkid -p" reports SWAPTYPE is swsuspend (not swap) when system hibernated on that swap.

Option "-p" (--probe, low-level superblocks probing, bypass cache) is necessary, otherwise resume failed.

```
# use swap partition
blkid -p -o value -s TYPE /dev/sdb8

# use swap file
blkid -p -O $((659456 * 4096)) /dev/vglubuntu/lvlubuntu
```

### Use swap file

On file systems other than Btrfs, the value of "resume_offset=" can be obtained by running filefrag -v swap_file. The desired value is in the first row of the physical_offset column.

```
# 512M swap file for 1G RAM
# 2G swap file for 16G RAM
sudo dd if=/dev/zero bs=1M count=512 of=s/swapfile.1
sudo mkswap s/swapfile.1
sudo chmod a-rw,u+rw s/swapfile.1
sudo filefrag -v s/swapfile.1
```

Cron job to use swap file.

```
sudo crontab -e

sudo crontab -l
@reboot sh -xvc 'f=/media/writable-backing/swapfile.1; [ -e "$f" ] || f=/media/isodevice/swapfile.1;  [ -s $f ] && swapon $f' 2>&1 | logger -t cron.job.reboot.swapon
```

### zramswap.sh

Hibernate while keep zram swap, may fail to resume.
Install "zramswap.sh" to avoid this problem.
Run the short script in its head part to install it.


```
head zramswap.sh

systemctl status zram-config.service
systemctl cat zram-config.service
cat /usr/bin/init-zram-swapping
cat /usr/bin/end-zram-swapping
#
man systemd-sleep
ls /etc/pm/ /usr/lib/pm-utils/sleep.d/ /lib/systemd/system-sleep/
```


For linux-5.15.0-119-generic, when "mem_limit" reached, "backing_dev" is used. Size of "backing_dev" is big enough in tests. "disksize" shall be not bigger than "mem_limit", otherwise fs of backing_dev crashes when disk is full in worst case (all data is incompressible). An echo "huge" to "writeback" tiggers flush to "backing_dev" and free memory used.


```
sudo crontab -l
# run every 15 minutes to push idle and huge pages to backing_dev
*/15 * * * * sh -xvc 'szr=/sys/block/zram0; for i in idle huge; do echo $i > $szr/writeback && sleep 1; done && echo all > $szr/idle' 2>&1 | logger -t cron.job.zramswap.writeback
```

### Allow GUI hibernate

Lubuntu-22.04.3 "Hibernate" ( mainmenu -> Leave -> Hibernate, lxqt-leave --hibernate) button not work.

Run command "lxqt-leave" in lxqt qterminal.

"lxqt-leave --hibernate" failed: systemd: "CanHibernate" = "no"


```
man polkit
man pklocalauthority
pkaction | grep -i -e upower -e hibernate
sudo grep -rni hibernat /var/lib/polkit-1 /etc/polkit-1/
sudo cat /var/lib/polkit-1/localauthority/50-local.d/disable-hibernate.pkla
sudo cat /var/lib/polkit-1/localauthority/10-vendor.d/com.ubuntu.desktop.pkla
# run in qterminal
pkcheck --action-id org.freedesktop.login1.hibernate --detail --process $$
# after change polkit config
sudo systemctl restart polkit.service
# sudo systemctl restart dbus.service

# run hibernate directly
sudo /usr/lib/systemd/systemd-sleep hibernate

# enable printk while do hibernate
cat /proc/sys/kernel/printk
grep -rn . /sys/module/printk/parameters/
echo 1 | sudo tee /sys/module/printk/parameters/ignore_loglevel
echo 0 | sudo tee /sys/module/printk/parameters/console_suspend
```

Crontab job to enable: lxqt-leave --hibernate

```
sudo crontab -e

sudo crontab -l
# enable hibernate
@reboot sh -xvc 'f=/var/lib/polkit-1/localauthority/50-local.d/disable-hibernate.pkla; [ -e $f ] && sed -i -e "s/^ResultActive=no/ResultActive=yes/" $f; f=/var/lib/polkit-1/localauthority/10-vendor.d/com.ubuntu.desktop.pkla; sed -i -e "/^\[[^]]\+\shibernate\s/,/^ResultActive=no/{s/ResultActive=no/ResultActive=yes/}" $f' 2>&1 | logger -t cron.job.reboot.enable-hibernate
```

Ubuntu-24.04 needs more to enable hibernate.

```
# show hibernate menu item
sudo sed -i -e 's/^NoDisplay=true/#&/' /usr/share/Lubuntu/applications/lxqt-hibernate.desktop

# sudo crontab -l
@reboot sh -xvc 'f=/sys/power/resume && [ "0:0" = "$(cat $f)" ] || exit 0; resume=; resume_offset=0; eval $(grep -o "resume[^ ]*=[^ ]\+" /proc/cmdline) && resume=${resume#UUID=} && [ -n "$resume" ] && resume=$(readlink -f /dev/disk/by-uuid/$resume) && resume=${resume#/dev/} && [ -n "$resume" ] && f=/sys/power/resume_offset && echo $resume_offset | sudo tee $f && cat $f && f=/sys/power/resume && cat /sys/class/block/$resume/dev | sudo tee $f && cat $f'  2>&1 | logger -t cron.job.reboot.enable-hibernate-resume
```

If "Hibernate" button is still not visable, create a shortcut of command "lxqt-leave --hibernate".


## Grub boot iso file

Grub 2.04 on uefi system (x86_64-efi) failed to boot iso file, "rmmod tpm" before run loopback to avoid this problem.

```
# web resouces
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=975835
https://askubuntu.com/questions/1520398/supergrub2-boot-iso-error-out-of-range
```

Lubuntu-18.04-desktop-amd64.iso cannot boot by grub2 (grub 2.02) loopback. "read out of range" error reported when load kernel image "/casper/vmlinuz" from loopback iso file in NTFS, and "unaligend pointer" error reported when load "/casper/initrd.lz".


Avoid the problem, run chkdsk and defrag NTFS (may try ntfsfix), defragment iso file: move the iso file to other partition, then defragment the NTFS partition and move iso file back.


### Grub check file

Check file size and checksum

```
# checksum in grub
sha256sum (hd1,2)/path/file

# compare files in grub, reports file size
cmp /grub/grub.cfg /grub/custom.cfg
```

For UEFI boot, set gfxmode and gfxpayload for fixed screen resolution.

```
# web resource
https://askubuntu.com/questions/761149/how-to-set-display-resolution-while-having-to-use-nomodeset-on-boot

set gfxmode=1024x768
set gfxpayload=keep
```



### Chainload grub core

Chainload grub core in UEFI boot (grub_platform=efi).

Chainload grub core not work in legacy BIOS boot (no-uefi, grub_platform=pc), use multiboot.

```
menuentry "Load other grub core" {
	set root=(hd0,1)
	set pc=/grub-2.12.i386-pc/grub/i386-pc/core.img
	set ec=/grub-2.12.x86_64-efi/grub/x86_64-efi/core.efi
	#
	if [ x"$grub_platform" = x"pc" ]; then
		echo multiboot $pc
		multiboot $pc
	else
		echo chainloader $ec
		chainloader $ec
	fi
}

# grub load grub.cfg
echo Delete old loop.
loopback -d loop
set iso_path=/data/lubuntu-22.04.3-desktop-amd64.iso
echo Setup loopback: $iso_path
loopback loop (hd3)/$iso_path
set root=(loop)
echo root=$root
export iso_path root
configfile /boot/grub/loopback.cfg
#
# insert debug lines in boot menu
echo grub_cpu: $grub_cpu
echo grub_platform: $grub_platform
echo prefix: $prefix
echo config_file: $config_file
echo gfxmode: $gfxmode
echo gfxpayload: $gfxpayload
echo root: $root
echo iso_path: $iso_path
# sleep --verbose --interruptible 9
if [ x"$config_file" = x"" ]; then set config_file=$prefix/grub.cfg; fi
if sleep --verbose --interruptible 5; then
    true
else
    configfile $config_file
fi
```

### Grub UART terminal

Append serial terminal to grub.

```
# web resource
https://www.gnu.org/software/grub/manual/grub/html_node/Serial-terminal.html

menuentry "Enable UART terminal" {
    serial --unit=0 --speed=115200
    serial --speed=115200
    terminal_output --append serial
    terminal_input --append serial
}
```

### Install livecd grub

Ubuntu livecd has both grub i386-pc and x86_64-efi. By default, only grub i386-pc is installed. Grub-efi can be installed from cdrom manually.

For grub i386-pc, the process is similar.

```
dpkg --no-pager -l grub*
ls /usr/lib/grub/i386-pc/
find /cdrom/ -name 'grub*.deb'
apt policy grub-efi

find /cdrom/ -name 'grub*.deb' -exec dpkg-deb --extract \{\} grub-efi \;
ls grub-efi/usr/lib/grub/
```

Install extracted x86_64-efi to ext4 partition.

```
sudo mount /dev/sdxxx d

( set -xv; history | tail -n 1; rtd=$(pwd)/d; btd=$rtd/grub-2.12.x86_64-efi; efid=$btd; d=/dev/sdxxx; itd=$(pwd)/grub-efi/usr; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=x86_64-efi  --efi-directory=$btd --directory=$itd/lib/grub/x86_64-efi --no-bootsector --removable --recheck --verbose --debug --force $d; echo rt=$? ) 2>&1 | tee grub-install.x86_64-efi.log

sudo mv grub-install.x86_64-efi.log d/grub-2.12.x86_64-efi/

ls d/grub-2.12.x86_64-efi/ d/grub-2.12.x86_64-efi/EFI/ d/grub-2.12.x86_64-efi/grub/x86_64-efi/
```


```
# Generate core.efi with different partition, hd number can be omitted.
# Grub files will be in dest dir "/grub-2.12.x86_64-efi.livecd/grub/" in dest partition.
#
( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-efi/usr; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log

( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-efi/usr; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi.livecd/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log

ls -l core.p1.efi
```

Copy installed grub files to new partition.

```
sudo mount /dev/sdxp1 b
sudo cp -a d/grub-2.12.x86_64-efi/ b/grub-2.12.x86_64-efi.livecd
sudo cp -a b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi.old
cat core.p1.efi | sudo tee b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi >/dev/null
ls -l b/grub-2.12.x86_64-efi.livecd/grub/x86_64-efi/core.efi*
cat /cdrom/boot/grub/*.cfg | sudo tee -a b/grub-2.12.x86_64-efi.livecd/grub/grub.cfg
```

### Build grub

Grub source code

```
https://git.savannah.gnu.org/cgit/grub.git/
https://ftp.gnu.org/gnu/grub/
https://git.savannah.gnu.org/git/grub.git
```

"grub-2.12.tar.gz" download from "https://ftp.gnu.org/gnu/grub/" lacks file "grub-2.12/grub-core/extra_deps.lst". It is available from git.


Run build

```
# build grub i386_pc
( set -xv; history| tail -n1; d=grub-2.12; wd=$(pwd) && { f=$d/grub-core/extra_deps.lst; [ -s $f ] || echo "depends bli part_gpt" | tee $f; } && tr=i386-pc && mkdir -p $d.$tr.build && cd $d.$tr.build && sh $wd/$d/configure && make -w V=1 && make -w V=1 install DESTDIR=$wd/$d.$tr._install; echo rt=$? ) 2>&1 | tee build.grub.i386-pc.log

# build grub x86_64-efi
( set -xv; history| tail -n1; d=grub-2.12; wd=$(pwd) && { f=$d/grub-core/extra_deps.lst; [ -s $f ] || echo "depends bli part_gpt" | tee $f; } && tr=x86_64-efi && mkdir -p $d.$tr.build && cd $d.$tr.build && sh $wd/$d/configure --target=x86_64 --with-platform=efi && make -w V=1 && make -w V=1 install DESTDIR=$wd/$d.$tr._install; echo rt=$? ) 2>&1 | tee build.grub.x86_64-efi.log
```

Build result

```
diff -rq grub-2.12.i386-pc._install grub-2.12.x86_64-efi._install

# merge installs of i386-pc and x86_64-efi
( set -xv; mkdir grub-2.12._install && mv grub-2.12.x86_64-efi._install grub-2.12._install && cp -nav grub-2.12.i386-pc._install/usr/local grub-2.12._install/usr/ && cp -a build.grub.x86_64-efi.log build.grub.i386-pc.log; echo rt=$? ) 2>&1 | tee merge.install.log
```


Install built grub

```
# install built grub x86_64-efi to EFI partition
sudo mount -t vfat /dev/sdx1 p1
( set -xv; history | tail -n 1; rtd=$(pwd)/p1; btd=$rtd/grub-2.12.x86_64-efi; efid=$btd; d=/dev/sdx1; itd=$(pwd)/grub-2.12._install/usr/local; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=x86_64-efi  --efi-directory=$btd --directory=$itd/lib/grub/x86_64-efi --no-bootsector --removable --recheck --verbose --debug $d; echo rt=$? ) 2>&1 | tee grub-install.x86_64-efi.log

# check installed files
find e/grub-2.12.x86_64-efi/ -type f -not -name '*.mod'
cat grub/x86_64-efi/load.cfg

# install built grub i386-pc
sudo mount -t vfat /dev/sdx5 p5
( set -xv; history | tail -n 1; rtd=$(pwd)/p5; btd=$rtd/grub-2.12.i386-pc; d=/dev/sdx5;itd=$(pwd)/grub-2.12._install/usr/local; export PATH=$itd/bin:$itd/sbin:$PATH; sudo grub-install --boot-directory=$btd --target=i386-pc --directory=$itd/lib/grub/i386-pc --no-bootsector --removable --recheck --verbose --debug $d; echo rt=$? ) 2>&1 | tee grub-install.i386-pc.log
# Add "--force" option if failed on some warning.
```

### Copy or move grub installed

Copy or move installed grub to other partition, need to generate core.img or core.efi with new partition number.

```
# Generate core.img and core.efi with different partition, hd number can be omitted.
# Grub files will be in dest dir "/grub-2.12.i386-pc/grub/" in dest partition.
#
( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/i386-pc --prefix "(,$p)/grub-2.12.i386-pc/grub" --output core.p$p.img --format i386-pc --compression auto ext2 fat ntfs part_gpt part_msdos biosdisk lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.i386-pc.log
#
( set -xv; history | tail -n 1; p=1; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; grub-mkimage --verbose --directory $d/lib/grub/x86_64-efi --prefix "(,$p)/grub-2.12.x86_64-efi/grub" --output core.p$p.efi --format x86_64-efi --compression auto fat ext2 ntfs part_msdos part_gpt lvm; echo rt=$? ) 2>&1 | tee grub-mkimage.x86_64-efi.log
```

### Install gurb bootsector

Before change MBR, make sure to backup it.
Use partition tool to get posible boot loader sectors after MBR, and backup them all. Here assuem they use first 2048 sectors along with main partition table from sector 0.

```
# backup bootsector MBR
sudo dd if=/dev/sdc of=hd.msdos.mbr bs=512 count=2048
sudo cp -a hd.msdos.mbr p5/boot/grub/
sudo dd if=/dev/sdc5 of=hd.msdos.p5.mbr bs=512 count=1
sudo cp -a hd.msdos.p5.mbr hd.msdos.p5.mbr.old
sudo cp -a hd.msdos.p5.mbr.old p5/boot/grub/
od -t x1a hd.msdos.p5.mbr
```

Install grub bootsector to partition 5 MBR

If file system comes immediatelly after sector 0, there is not enough sectors to embed grub "core.img".


```
# install grub bootsector to MBR
sudo mount /dev/sdx5 p5
# grub-probe --verbose --device-map=device.map --device /dev/sdc5
( set -xv; history | tail -n 1; k=sdx; p=7; d=$(pwd)/grub-2.12._install/usr/local; export PATH=$d/bin:$d/sbin:$PATH; sudo grub-mkdevicemap --verbose --no-floppy --device-map=device.map && sudo parted /dev/$k -- p unit s p && cat device.map && sudo grub-bios-setup --verbose --directory=p5/grub-2.12.i386-pc/grub/i386-pc --device-map=device.map /dev/$k$p; echo rt=$? ) 2>&1 | tee grub-bios-setup.log
# add "--force" if not-used blocks at the beginning of partition are not enough to embed core.img.
```


## initrd

Casper script file (initrd/main/scripts/casper, /usr/share/initramfs-tools/scripts/casper) in initrd has a bug, patch it to clean "/tmp/debconf.xxxx" folder.

"zcat" is missing in initrd of Ubuntu-server-22.04.5 and Lubuntu-22.04.3, so "casper-sn.cpio.gz" not work. Patch to use "gzip -dc" instead.

Apply "initramfs.casper.patch" to casper cripts in folder "/usr/share/initramfs-tools/scripts/". Then rebuild initrd by "mkinitramfs".

Or, use "unmkinitramfs" extract files from "/cdrom/casper/initrd". Apply patch and then create initrd as below.

Or, patch main.cpio in initrd.



### initramfs.split.sh

Run this script to split initrd, codes are copied from: /usr/bin/unmkinitramfs

```
sh initramfs.split.sh /cdrom/casper/initrd
ls -l early.*.cpio main.cpio*
mv main.cpio.xx main.cpio.zst
zstd -kd main.cpio.zst
```

### patch.main.cpio.sh


After patch, the initrd allows writable-backing and casper-sn on fs more than vfat, by adding kernel parameter "wbfstype=vfat,ex2,ext4" to "linux" line in file "grub.cfg".

The patch deletes "/root/$DEBCONF_TMPDIR", and allows resume from hibernate.



```
sh  patch.main.cpio.sh main.cpio
```

After patch, create initrd.

```
xz --check=crc32 --threads=0 main.cpio.patched
( set -xv; cat early.?.cpio main.cpio.patched.xz > rd && ls -l rd; echo rt=$? )
```


You can extract files from main.cpio if you want.

```
# extract main.cpio.xx
file main.cpio.xx
mv main.cpio.xx main.cpio.zst
( set -xv; mkdir -p initrd && mkdir initrd/main && cd initrd/main && zstd -dc ../../main.cpio.zst | sudo cpio -id; echo rt=$? )
```


### Create inird cpio file

Create inird cpio file from extracted folder.

```
sudo sh -xvc 'wd=$(pwd) && cd $1 && find . | LC_ALL=C sort | cpio --quiet -R 0:0 --reproducible -o -H newc > "$wd/$(echo $1 | tr "/" "-").cpio"; echo rt=$?' x  initrd/main
```

### Create initrd

Create initrd with early.cpio and main.cpio.

```
# compress main cpio and prepend early cpio
xz --check=crc32 --threads=0 initrd-main.cpio
( set -xv; cat early.?.cpio initrd-main.cpio.xz > rd )

# test it
( set -xv; history | tail -n 1; lsinitramfs -l rd && unmkinitramfs -v rd d && sudo diff -rq initrd/ d/; echo $? ) 2>&1 | tee t.log
```


### initrd script files and functions

```
/etc/casper.conf
initrd/main/conf/conf.d/default-boot-to-casper.conf
/usr/share/initramfs-tools/scripts/casper
/usr/share/initramfs-tools/scripts/casper-bottom/25adduser
/usr/lib/user-setup/user-setup-apply
/usr/share/debconf/confmodule

diff initrd/main/init /usr/share/initramfs-tools/init
less /var/log/casper.log

# casper on system shutdown
systemctl cat casper.service
#
/usr/sbin/casper-stop
/usr/sbin/casper-snapshot
/usr/share/initramfs-tools/scripts/casper-helpers
/lib/lsb/init-functions
#
root_snapshot_label  home_snapshot_label
try_snap ()
do_snap_copy ()
```

Livecd initrd boot casper call tree

```
# init process is initrd/main/init
#
# config files in folder "initrd/main/conf/conf.d" are invoked by initrd/main/init
# grep -rn  BOOT=casper initrd/main/conf/
initrd/main/conf/conf.d/default-boot-to-casper.conf:2:    export BOOT=casper
#
In file initrd/main/init:
. /scripts/${BOOT}
mountroot
read_fstab_entry
#
In file initrd/main/scripts/casper, function mountroot() calls:
find_livefs
mount_images_in_directory() -> setup_overlay();
setup_auto_log_persistence
run_scripts /scripts/casper-bottom
cp /etc/casper.conf "${rootmnt}/etc/"
#
setup_overlay() calls:
find_cow_device() - root_persistence_label();
mount .... /dev/xxx /cow
get_backing_device() -> setup_loop()
mount -t squashfs -o ro,noatime /dev/loop0 /ubuntu-server-minimal.squashfs
mount -t overlay ... /cow /root
try_snap() -> do_snap_copy() -> cp -a /mnt/tmpsnap/* /root
mount -o move /ubuntu-server-minimal.squashfs /root/rofs
#
In file initrd/main/scripts/casper-functions, function load_confmodule():
. /root/usr/share/debconf/confmodule
#
# list files invoke /usr/share/debconf/confmodule
grep -rn load_confmodule initrd/main/scripts/

# list break points, "break=xxx" of kernel parameter at boot
grep -rn maybe_break initrd/main/scripts/ initrd/main/init
```


## Lxd container and docker

Ubuntu live server is lxd ready. But docker needs to be installed manaully.
Recommend to use another lvm as container storage.


### Container working folder

Install systemd mount files and enable them.
"bind" mount can also be used for lxd working folder. For "overlay" mount, "upper" and "work" folder shall be created manually before the first use.

Stop all lxd related services and sockets before change mount.

Copy systemd mount files to folder "/etc/systemd/system/" and enable them.

```
mnt-data.mount
var-snap-lxd-common-lxd.mount
```


Change docker container working folder.

```
sudo systemctl stop docker.service docker.socket
sudo mkdir -p /mnt/data/var_lib_docker
printf "%s\n" "{" '"data-root": "/mnt/data/var_lib_docker",' '"storage-driver": "overlay2"' "}" | sudo tee -a /etc/docker/daemon.json
sudo mv /var/lib/docker/ /var/lib/docker.old
```


### Create container image and run it


Install binfmt support for various CPU arch.

```
sudo apt-get install qemu-user-static binfmt-support
```

There are many online images ready to run.
Here create image from scratch.

Change "--arch=armhf" as desired, i.e. "--arch=riscv64".
Delete "http://10.100.93.1:3142/" if not use apt-cacher server.

```
# if you are in a regular lxc container, it shall run with mknod allowed:
lxc config set ubtsvr security.syscalls.intercept.mknod=true

# prepare rootfs
( set -xv;  history | tail -n 1; date -Is; systemd-detect-virt; uname -a; lsb_release -a; sudo debootstrap --verbose --keyring=/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg --foreign --arch=armhf --include=autoconf,automake,libtool,bash-completion,bc,bsdmainutils,btrfs-progs,build-essential,ca-certificates,cloud-guest-utils,command-not-found,curl,device-tree-compiler,dosfstools,git,gnupg,initramfs-tools,less,libinline-files-perl,libnss-mdns,libnss-systemd,libpam-systemd,linux-base,locales,man-db,net-tools,openssh-server,overlayroot,patch,rsync,ssl-cert,sudo,systemd-timesyncd,tree,u-boot-tools,unzip,vim,wget,zstd --components=main,universe,multiverse jammy rootfs http://10.100.93.1:3142/ports.ubuntu.com/ && sudo chroot rootfs debootstrap/debootstrap --verbose --second-stage && sudo chroot rootfs sh -xvc 'adduser user; adduser user sudo; apt clean; rm -fr /var/cache/* /var/lib/apt/lists/*'; echo rt=$? ) 2>&1 | tee debootstrap.log

# create metadata.yaml for lxc
( set -xv; ( echo architecture: "amd64"; echo creation_date: $(date +%s) ) | tee metadata.yaml && tar -cJf metadata.tar.xz metadata.yaml && rm metadata.yaml; echo rt=$? )
```

For docker image, rootfs files must in folder ".".
For lxc image, rootfs files must in folder "rootfs" if tar contains metadata.yaml for "lxc image import metafata.and.rootfs.tar", while rootfs files must in folder "." for "lxc image import metadata.tar rtfs.tar".


```
# import docker image
docker image import rtfs.tar.xz armhf

# run docker container
docker run -it --rm --name armv7l --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw armhf /sbin/init
docker run -it --rm --name riscv64 --tmpfs /tmp --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw riscv64 /sbin/init

# run privileged container
docker run -it --rm --name armv7l --privileged -v /mnt/data/work:/mnt/work armhf /sbin/init

# import lxc image
lxc image import metadata.tar.xz rootfs.tar.xz --alias armv71

# run lxc container
lxc launch armv7l armhf --console

# check files in container
lxc file edit armhf/etc/hosts
lxc file pull armhf/bin/bash /tmp/ -v

# change init process from "/sib/init" to "/bin/bash"
lxc config set systemd raw.lxc 'lxc.init.cmd = /bin/bash'
```

Systemd-networkd failed in container.

```
systemd-networkd[226]: Could not create manager: Protocol not supported
"systemd-networkd.socket" starts OK but failed when "systemd-networkd.service" failed.
```

Workaround for this problem: use static ip

```
# sudo crontab -l
@reboot /bin/sh -xvc 'sleep 6 && ip addr add dev eth0 10.100.93.198/24 && ip route add default via 10.100.93.1; f=/etc/resolv.conf; [ -s $f ] || { systemctl stop systemd-resolved.service; systemctl disable systemd-resolved.service; rm -f $f; ( echo nameserver 10.100.93.1; echo search lxd ) > $f; }' 2>&1 | logger -t cron.job.reboot.1
```

### Lxd dns record

Host can add host-record for containers dns query.

"host-record" is not supported by old version of dnsmasq, followed by "local" to prevent forward query to upstream server. Client DNS query slow down while dnsmasq try to reach upstream server configured in "/etc/resolv.conf" when no internet available.

```
( set -xv; s="$(printf "%s\n" "host-record=my.fileserver.com,192.168.58.211" "local=/fileserver.com/")"; lxc network set lxdbr0 raw.dnsmasq="$s"; echo rt=$? )
```


### Mount in container

By default, privileged (unconfined) container allow mount and umount.
If configured properly, regular (no-privilegded) container can mount and umount.
Umount returns quickly but "/proc/mounts" updated slowly in container. Wait a second after umount before check if umounted.

Device mapper does not work properly in privileged container. Use "losetup --partscan".

"losetup --partscan" in privileged container works and "lsblk" can see partitions, but dev node needs manually create.

```
sudo sh -xvc 'f=$1; p=$(losetup --find); t=${p#/dev/}p1; { [ -b /dev/$t ] && [ -e /dev/mapper/${t} ]; } || ( kpartx -d $p; losetup -d $p; sleep 1; [ -e $p ] || { mknod $p b 7 ${p#*loop} || exit $?; }; losetup --partscan $p "$f" && sleep 1 && m=$(lsblk -r $p | grep $t | cut -d " " -f 2) && n=${m#*:} && m=${m%:*} && { [ -z "$m" ] && m=253 || true; } && { [ -z "$n" ] && n=0 || true; } && cd /dev/ && rm -f $t && mknod $t b $m $n && cd mapper && rm -f $t && ln -s ../$t . )' x image
```


## apt-cacher-ng

Use apt-cacher-ng as apt proxy server to speed up apt work.


```
# install apt-cacher-ng
sudo apt-get install apt-cacher-ng

# check files installed
dpkg -L apt-cacher-ng

# html page for web browser
vi /usr/lib/apt-cacher-ng/userinfo.html

# check config
grep -rni 'CacheDir\|acng-doc' /etc/apt-cacher-ng/
grep -rni remap /etc/apt-cacher-ng/
grep -rni 'Port\|Offlinemode' /etc/apt-cacher-ng/
sudo cat /etc/apt-cacher-ng/security.conf
ls -l /var/log/apt-cacher-ng/
tail -f /var/log/apt-cacher-ng/apt-cacher.log /var/log/apt-cacher-ng/apt-cacher.err
#
ls -l /var/lib/apt-cacher-ng /var/cache/apt-cacher-ng
ls -l /usr/share/doc/apt-cacher-ng/
```

### Move data folders

```
pgrep -a apt
sudo systemctl stop apt-cacher-ng
#
sudo sh -xvc 'rd=/mnt/data; for d in /var/lib/apt-cacher-ng /var/cache/apt-cacher-ng; do [ -e $d.old ] || { i=$(echo $d | tr -s "/" "_"); cp -a $d $rd/$i && mv $d $d.old && ln -s $rd/$i $d; }; done'
```

### Offline mode

When apt-cacher-ng cannot reach outside website, client failed to run "apt update". To solve the problem, set "Offlinemode: 1" and restart apt-cacher-ng.

When "Offlinemode: 1", cached "Release" file is not updated when client run "apt update". When "Offlinemode: 0" (default), cached "Release" file is always updated when client run "apt update".


Toggle "Offlinemode" on network interface up and down.

Use file "99_enp0s8.sh" to auto toggle "Offlinemode", run the short sript at its head part to install it.


### Cache repository Releae and Packages

```
# cache repository Releae InRelease and Packages.xz
# http://192.168.58.101:3142/acng-doc/html/config-serv.html#repmap
# https://hub.docker.com/r/gijoe88/docker-apt-cacher-ng
#
# check if Release file and Packages.xz are cached:
http://192.168.58.101:3142/acng-report.html
Click "Start Scan and/or Expiration" in "Scan and semi-automatic expiration".
#
grep -rn uburep /etc/apt-cacher-ng/
ls -l /etc/apt-cacher-ng/backends_ubuntu* /usr/lib/apt-cacher-ng/
less /usr/lib/apt-cacher-ng/ubuntu_mirrors
find /var/cache/apt-cacher-ng/ -iname 'packages*' -o -iname 'release*' -o -iname 'inrelease*'  -exec ls -l \{\} \; | grep /uburep/
find /var/cache/apt-cacher-ng/ -iname 'packages*' -o -iname 'release*' -o -iname 'inrelease*'  -exec ls -l \{\} \; | grep -v /uburep/

# add lines in file "/etc/apt-cacher-ng/acng.conf" to cache Release file for these repositories
grep 'Remap-secdeb' /etc/apt-cacher-ng/acng.conf
Remap-ubuportsrep: ports.ubuntu.com /ubuntu-ports ; ports.ubuntu.com/ubuntu-ports
Remap-secubu: security.ubuntu.com /ubuntu ; security.ubuntu.com/ubuntu
Remap-uburcneerep: repos.rcn-ee.com/ubuntu ; repos.rcn-ee.com/ubuntu
Remap-debrcenrep: repos.rcn-ee.com/debian ; repos.rcn-ee.com/debian

# restart apt-chacher-ng and monitor log
tail -f /var/log/apt-cacher-ng/apt-cacher.log
```


## Windows NTP server

Except Linux NTP server, Win7 and Win10 have NTP server, it can be used to sync time.


```
# web resources
https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tools-and-settings
https://learn.microsoft.com/en-us/troubleshoot/windows-server/active-directory/configure-authoritative-time-server
https://support.hanwhavision.com/hc/en-us/articles/26570683589529-How-to-Setup-an-NTP-Server-on-Windows-10
https://support.nortekgroup.com/hc/en-us/articles/9657669911452-Use-Windows-10-as-NTP-Server

# cmd.exe commands
w32tm /?
w32tm /query /configuration /verbose
w32tm /query /status /verbose
w32tm /query /peers /verbose
w32tm /dumpreg /subkey:TimeProviders\NtpServer

# regedit
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer
# change: Enabled=0
Enabled=1
#
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config
# change: AnnounceFlags=0x0a
AnnounceFlags=0x05

# check Windows Time service status
sc query | find /i "time"
sc query W32Time

# apply cinfig changes
w32tm /config /update
sc stop W32Time
sc start W32Time

# allow firewall income
netsh advfirewall firewall add rule name="NTP Server" dir=in protocol=udp localport=123 profile=any enable=yes action=allow

# test
w32tm /stripchart /computer:192.168.58.1 /samples:4 /dataonly
```

Client side

```
# sntp report: "not in sync, skipping this server"
# Set Windows 10 NTP server "AnnounceFlags=0x05" to avoid this problem.
# test ntp server
sntp -d -d -d 192.168.58.1

# systemd timesyncd
systemctl cat systemd-timesyncd.service
man timesyncd.conf
man systemd.time
less /etc/systemd/timesyncd.conf
timedatectl status
timedatectl show-timesync
timedatectl timesync-status
#
# configure systemd-timesyncd to user Windows 10 NTP server
# Change lines in file /etc/systemd/timesyncd.conf
# Then restart systemd-timesyncd.service
# sudo systemctl restart systemd-timesyncd.service
#
( set -xv; ( echo "FallbackNTP=192.168.58.1"; echo "RootDistanceMaxSec=5356800" ) | sudo tee -a /etc/systemd/timesyncd.conf )
```