[Discuss] --sandbox switch for Ubuntu's do-release-upgrade/update-manager

[Tom Metro]

I was looking to do a "dry run" test with do-release-upgrade to see if a
system could successfully be upgraded from 12.04 to 14.04 after some
changes had been made. (An earlier attempted failed to map packages from
the old release to the new release.)

do-release-upgrade doesn't have a dry run option, but does have a
--sandbox option, which is tersely documented[1] as:

"Test upgrade with a sandbox aufs overlay"

OK...but that leaves a lot of unanswered questions.

Anyone used this feature?

I searched but couldn't find any more in-depth documentation, or a wiki
page. All I could turn up were oblique references to --sandbox, such as
bug reports of it not working[2][3].

I did find a tangential comment[4] in a stack exchange Q&A asking
whether it was better to use 'apt-get dist-upgrade' or
do-release-upgrade, which said, "do-release-upgrade has some other cool
features, like --sandbox, which remounts your filesystems with aufs so
that the upgrade can be rolled back!"

OK, so how do you roll it back?

Another blog posting[5] says, "Simply just run 'do-release-upgrade -s'
to sandbox the upgrade. From that point any changes made to your system
will actually be written to a ramdisk, and reset upon reboot via the
magic os Aufs. This lets you test your machine thoroughly in a fully
functioning state, without risking breaking it."

So how do you "test" an OS upgrade if you can't reboot to the new OS
version? (Use of RAM disk also seems unlikely.)

Eventually I ran across a 2009 mailing list posting describing the
option when it was first introduced:

https://lists.ubuntu.com/archives/ubuntu-devel/2009-March/027747.html

"The idea...was to create a writable overlay into /tmp on top the
systemdirs in "/" and then run the release upgrade. This way we can
test easily if the system would upgrade cleanly (if no dpkg
errors/maintainer script failures happen). All writes go into /tmp so
after the upgrade and on the next reboot the system is back to its
pre-upgraded state again (modulo /home, that is not overlayed)."

OK, more details. Now we know the files are stored in /tmp. But I'm
still not following why you wouldn't want to preserve this across a
reboot, and have an explicit command to roll back the upgrade.

It seems like they only care about testing the package conflict
resolution and the package installation scripts. (Half of that you can
test without even writing to the file system.) What I care about is how
badly will my alternate desktop manager be broken after an upgrade.

Digging into the Python code that implements do-release-upgrade
uncovered the 'mount' command line they use to create the AuFS[6] file
system (a derivative of UnionFS), that there are options specified via
config file to specify the location of the overlay file system (so you
could put it outside of /tmp; it defaults to /tmp/upgrade-rw), and code
to rsync the overlay filesystem back to the real file system (but
unclear what calls that code).

Using this feature with update-manager potentially addresses an idea I
brought up here previously: an apt-get that can revert an upgrade. (If
only it was a console program instead of a GUI. And it would still need
to be tweaked to persist across reboot.)

I'm wondering why this isn't better documented and promoted. I'm left
with the impression that the feature may be half-baked. Particularly due
to things like the bug report[2] suggesting that any upgrade that causes
grub to update the bootloader will fail, as the overlay FS gets in the
way. Then again if it all evaporates when you reboot, you've already
learned what you're going to learn by the time the grub update stage of
the upgrade runs.

-Tom

1.
http://manpages.ubuntu.com/manpages/precise/man8/do-release-upgrade.8.html
2. http://osdir.com/ml/ubuntu-bugs/2014-04/msg30955.html
3. http://ubuntuforums.org/showthread.php?t=2113373
4.
http://serverfault.com/questions/322810/whats-the-difference-between-apt-get-dist-upgrade-and-do-release-upgrade
5.
http://blog.coopersphotos.co.uk/technology/two-tech-tips-sandboxing-ubuntu-ssh-configs-on-the-fly
6. http://www.thegeekstuff.com/2013/05/linux-aufs/

[Mike Small]

Post by Tom Metro
I was looking to do a "dry run" test with do-release-upgrade to see if a
system could successfully be upgraded from 12.04 to 14.04 after some
changes had been made. (An earlier attempted failed to map packages from
the old release to the new release.)

...

Post by Tom Metro
test without even writing to the file system.) What I care about is how
badly will my alternate desktop manager be broken after an upgrade.

This is what Nix (and Guix) is supposed to provide, but designed in from
the start.

http://nixos.org/nix/
https://www.gnu.org/software/guix/manual/html_node/Introduction.html#Introduction

[Matthew Gillen]

Post by Tom Metro
I was looking to do a "dry run" test with do-release-upgrade to see if a
system could successfully be upgraded from 12.04 to 14.04 after some
changes had been made. (An earlier attempted failed to map packages from
the old release to the new release.)

Sorry, no help w.r.t. to question you asked, but it reminded me of a
related issue. I remember wanting to do something similar, but thinking
that LVM snapshots were the way to go. The issue for non-virtualized
servers for me was typically a problem because of my
small-scale/hobbyist scenario: you need a working system to roll back
the snapshots, but rolling back the root fs was especially tricky. What
this meant was using rescue disks, and the rescue disks had to have the
right version of LVM to allow you to roll back to the older snapshot.
And I had long since gotten out of the habit of burning media for every
OS upgrade... And then you had the whole boot-loader issue, which could
be managed (/boot isn't so big that you couldn't just stuff it somewhere
for manual reconstitution, but again you need to boot into a rescue
environment for that).

I think it makes more sense if you've got a VM. I think (it was a while
ago, forgive my foggy memory) I successfully used qcow2's snapshotting
for this sort of roll-back scenario.

With btrfs the situation might be better, since you can more easily
define the mount points for directly interacting with snapshots. For
instance, you could have a grub config that booted the old kernel using
the snapshot from the old fs.

The trick is still that all upgrades try to "fix" your boot loader
config by removing the old stuff. If you're careful and have a big
/boot, you could hide everything needed for the old system, and
reconstitute it (and manually edit grub.conf) after the upgrade is
complete but before it reboots into the new system. There may be other
little gotchas like needing to reformat your swap, but that's easy
enough if you can get the old system back online.

But then some new release will come out with grub 3 and you'll be in a
new kind of pain, since there is still only one /boot, and you can only
have one boot loader.

I think I'm convincing myself that if you want to roll back, use VMs.
/boot is too complicated to deal with.

Matt

[Richard Pieri]

Post by Matthew Gillen
I think I'm convincing myself that if you want to roll back, use VMs.
/boot is too complicated to deal with.

No, you have the right idea. It's possible to roll back with LVM but the
problem with this is that you /must/ eventually roll back or re-create
the file systems else you'll eventually run out of space in the volume
group. This is the same problem that AUFS and other union (overlay) file
systems have. They're not file systems as we typically think of them.
They're logs of transactions relative to a master file system.

Ubuntu has a long history of not being upgradable in place. Ubuntu uses
Debian's package management system but it doesn't use Debian's stability
mandate. That's Debian's definition of "stability" which more or less
means "updates and upgrades never break a working system."[1] That
Ubuntu's Johnny-come-lately upgrade tool needs a sandbox mode is
demonstrable proof of how poor it is in its current state. Personally,
on the rare occasions where I have to do a major upgrade on Ubuntu I
make sure that my configuration documents are in order, I do a clean
install, and I recreate the custom configuration based on the documentation.

[1] I've only once encountered a situation where Debian was required to
break that mandate and that was due to a security change in Dovecot up
stream that broke everybody.

[John Hall]

Tom I would not trust the sandbox feature for reasons you outlined.

I have managed multiple versions of multiple distros for a dev workstation
using lvm. it is by no means painless. I did learn the grub2 command line.
It is not a painless plan!

*A much better plan is to make a copy onto new hardware or into a virtual
machine and test it there*, but if you go forward... does 12.04 use grub1
and 14.04 use grub2? (I think so!)
You might upgrade that first in 12.04 if it's still using grub1 to make the
process smoother!

If you have LVM2 and plenty of free space, make a new LV, create your file
system then create a file level copy of the old system.(do not use dd**),
then upgrade. When the time comes, just lvremove the one you don't want. This
method works better if you have data compartmentalized nicely in
separate volumes.
and don't already have snapshots.

Any method you use to test the system, you are going to have to do some
config wrangling to go back to the old system, especially if it's temporary
like fiddling with fstab, and using UUID's there.

The other advantage of a two system volume path is that if you decide you
want to clean install you can mount the old system for reference.



Where a snapshot will succeeded where a full copy won't is if you have alot
of data in the same volume as the system and don't have room for another
copy of it. I think that could save some copy time as well as be more
transparent. I believe snapshots are copy on write so this could result in
weird performance during testing.
If you forget about the snapshot it will consume some space. You have to
make up your mind at some point and commit it back to the original or drop
it.

**Do not use dd and leave the lv online, only use dd in the case of a
separate vg you can deactivate. and in the case that you have a
self-contained bootalbe lv setup with grub etc! ( this is far from
anything standard especially for Ubuntu!

Post by Tom Metro
I was looking to do a "dry run" test with do-release-upgrade to see if a
system could successfully be upgraded from 12.04 to 14.04 after some
changes had been made. (An earlier attempted failed to map packages from
the old release to the new release.)
do-release-upgrade doesn't have a dry run option, but does have a
"Test upgrade with a sandbox aufs overlay"
OK...but that leaves a lot of unanswered questions.
Anyone used this feature?
I searched but couldn't find any more in-depth documentation, or a wiki
page. All I could turn up were oblique references to --sandbox, such as
bug reports of it not working[2][3].
I did find a tangential comment[4] in a stack exchange Q&A asking
whether it was better to use 'apt-get dist-upgrade' or
do-release-upgrade, which said, "do-release-upgrade has some other cool
features, like --sandbox, which remounts your filesystems with aufs so
that the upgrade can be rolled back!"
OK, so how do you roll it back?
Another blog posting[5] says, "Simply just run 'do-release-upgrade -s'
to sandbox the upgrade. From that point any changes made to your system
will actually be written to a ramdisk, and reset upon reboot via the
magic os Aufs. This lets you test your machine thoroughly in a fully
functioning state, without risking breaking it."
So how do you "test" an OS upgrade if you can't reboot to the new OS
version? (Use of RAM disk also seems unlikely.)
Eventually I ran across a 2009 mailing list posting describing the
https://lists.ubuntu.com/archives/ubuntu-devel/2009-March/027747.html
"The idea...was to create a writable overlay into /tmp on top the
systemdirs in "/" and then run the release upgrade. This way we can
test easily if the system would upgrade cleanly (if no dpkg
errors/maintainer script failures happen). All writes go into /tmp so
after the upgrade and on the next reboot the system is back to its
pre-upgraded state again (modulo /home, that is not overlayed)."
OK, more details. Now we know the files are stored in /tmp. But I'm
still not following why you wouldn't want to preserve this across a
reboot, and have an explicit command to roll back the upgrade.
It seems like they only care about testing the package conflict
resolution and the package installation scripts. (Half of that you can
test without even writing to the file system.) What I care about is how
badly will my alternate desktop manager be broken after an upgrade.
Digging into the Python code that implements do-release-upgrade
uncovered the 'mount' command line they use to create the AuFS[6] file
system (a derivative of UnionFS), that there are options specified via
config file to specify the location of the overlay file system (so you
could put it outside of /tmp; it defaults to /tmp/upgrade-rw), and code
to rsync the overlay filesystem back to the real file system (but
unclear what calls that code).
Using this feature with update-manager potentially addresses an idea I
brought up here previously: an apt-get that can revert an upgrade. (If
only it was a console program instead of a GUI. And it would still need
to be tweaked to persist across reboot.)
I'm wondering why this isn't better documented and promoted. I'm left
with the impression that the feature may be half-baked. Particularly due
to things like the bug report[2] suggesting that any upgrade that causes
grub to update the bootloader will fail, as the overlay FS gets in the
way. Then again if it all evaporates when you reboot, you've already
learned what you're going to learn by the time the grub update stage of
the upgrade runs.
-Tom
1.
http://manpages.ubuntu.com/manpages/precise/man8/do-release-upgrade.8.html
2. http://osdir.com/ml/ubuntu-bugs/2014-04/msg30955.html
3. http://ubuntuforums.org/showthread.php?t=2113373
4.
http://serverfault.com/questions/322810/whats-the-difference-between-apt-get-dist-upgrade-and-do-release-upgrade
5.
http://blog.coopersphotos.co.uk/technology/two-tech-tips-sandboxing-ubuntu-ssh-configs-on-the-fly
6. http://www.thegeekstuff.com/2013/05/linux-aufs/
--
Tom Metro
The Perl Shop, Newton, MA, USA
"Predictable On-demand Perl Consulting."
http://www.theperlshop.com/
_______________________________________________
Discuss mailing list
http://lists.blu.org/mailman/listinfo/discuss

[Greg Rundlett (freephile)]

Post by Tom Metro
I was looking to do a "dry run" test with do-release-upgrade to see if a
system could successfully be upgraded from 12.04 to 14.04 after some
changes had been made. (An earlier attempted failed to map packages from
the old release to the new release.)
do-release-upgrade doesn't have a dry run option, but does have a
"Test upgrade with a sandbox aufs overlay"

This doesn't answer your specific question, as I don't have experience
using that option.

However, the March 2015 Linux Magazine has an article [1] about
"Systemback" [2] which is designed to make it easy to create restore points
on Debian / Ubuntu / Linux Mint as well as test upgrades etc.

From the website:

Systemback makes it easy to create backups of system and users
configuration files. In case of problems you can easily restore the
previous state of the system. There are extra features like system copying,
system installation and Live system creation.


[1]
http://www.linux-magazine.com/Issues/2015/172/Systemback/(language)/eng-US
[2] http://sourceforge.net/projects/systemback/ and
https://launchpad.net/systemback


Greg Rundlett
http://eQuality-Tech.com <http://equality-tech.com/>
http://freephile.org