Subject: Re: GCD005: Regular and efficient releases

From: Efraim Flashner <[email-masked]>
Date: Mon, 12 May 2025 15:38:20 +0300

On Mon, May 12, 2025 at 11:09:21AM +0100, Steve George wrote:
> Hi Ian,
> 
> On 11 May, Ian Eure wrote:
> > Hi Steve,
> > 
> > Steve George <[email-masked]> writes:
> > 
> > > 3. Rolling updates aren't suitable for all users.
> > 
> > If a Guix release is a point in time snapshot of a rolling release, and the
> > first `guix pull' after
> > installation puts you back into the rolling release model, I don’t think
> > more frequent releases address this need.
> (...)
> 
> Yes, so this GCD doesn't resolve any issues with the rolling release not being suitable for all users (problem 3). An earlier version did, but following input from the sponsors I cut the GCD down to focus it and make it easeir to implement - one change at a time! That's why it refers to the idea that in future we could implement slower moving branches, or something else - something along those lines would resolve that issue - but it's out of scope for this GCD.
> 
> This GCD focuses on problem (1) and (2) only.
> 
> Rereading it this morning, it's not clear, so I will alter the paragraph to make it clear. Thanks!
> 
> 
> > > Adding a slower-moving branch akin to Nix's stable could be an eventual
> > > goal as
> > > it would increase Guix's suitability for some users and use-cases [^2].
> > > However,
> > > this GCD only sets out to implement regular releases which is a
> > > substantial
> > > change that would be a big improvement for our users.
> > 
> > 
> > > A specific issue caused by irregular releases is that new users/installs
> > > face a
> > > significant first "guix pull". This provides a poor initial user
> > > experience,
> > > and in some cases may even deter users [^4]. Additionally, it requires
> > > the
> > > project to keep old substitutes on our servers.
> > 
> > It also causes user confusion becasue the assumption is that the release is
> > what they should be using.  We get a good number of folks in #guix reading
> > the 1.4.0 manual, which doesn’t accurately reflect the current state of
> > things.
> > 
> 
> Agreed.

I think we can change the links on the website so that the manual points
to the latest manual and then one with /$release-version/ in the URL
would be the one associated with that release specifically.  Over the
past 2 years that it's been brought up no combination of people have
been brave enough to create a patch for the maintenance repo and to
apply the patch.

> 
> > > This GCD proposes an annual release cycle, with releases **in May**.
> > > 
> > > To move onto this cycle the first release would be a little later:
> > > aiming for
> > > the **November of 2025**, with a short cycle to release in May 2026.
> > 
> > I think it’d be better to align the initial release with the schedule we
> > want to keep, so if we plan for a November release, plan to release every
> > November.
> > 
> 
> An annual spring release would be better due to alignment in the ecosystem around stabilisation: it's beneficial to get the additional stabilisiation that the commercial distro's do and RHEL/LTS's always do a spring release [0]. I outlined this a bit more elsewhere in the thread, but can go into more detail.
> 
> I personally would like us to do a release before $SPRING 2026. Doing releases more often, should help us get better at them, automate them (see comments from Ekaitz/Reza), so hopefully doing a November and then a $SPRING would be achievable.
> 
> I would give shifting to a $SPRING time-frame, priority over doing the November one if it was an absolute choice, as I think that time of year is optimal. 

We debated quickly what month might work well and we ended up any
release is better than no release, so we figured we'd leave it.
Before/after February runs into FOSDEM and we don't want to race FOSDEM
for a release or be in freeze while people have Big Ideas™.  Likewise
August is difficult for many Europeans when they have off from work, so
we didn't want to overlap with that month either.

> 
> > > ## Release artifacts
> > > 
> > > Using the primary architecture tier and the package sets would involve
> > > creating
> > > the following release artifacts:
> > > 
> > > - GNU Guix System ISO image
> > > - GNU Guix System QCOW2 image
> > > - GNU Guix installer
> > 
> > I’m not sure what the difference is between the installer and system ISO
> > image is, could you elaborate please?

The ISO and the QCOW2 image are almost identical, but the ISO is more of
a live image for actual hardware and the QCOW2 image is for VMs.  The
installer is the installer.

Combining a couple of different ideas, I think we should add aarch64
images that uses grub-efi and starts with an offset of 16MB (as
suggested by vagrant), and hopefully that'll work for at least some
machines.  We can always tweak it later.

> 
> AIUI we have two install paths (product?), one is installing 'Guix System' as a Linux distribution, the other is installing 'Guix' (package manager) on a foreign distribution. For a release we update both. So 'GNU Guix installer' was refering to the Guix as a package manager, and the other two were 'Guix System'.
> > > Again in an effort to reduce developer toil, additional release
> > > artifacts could
> > > be created but would not be part of the formal release testing and
> > > errors would
> > > not block a release.
> > 
> > The 1.4.0 release artifacts are[1]:
> > 
> > - Installer image for i686 and x86_64.
> > - QCow image for x86_64.
> > - Binary for i686, x86_64, armhf, aarch64, and powerpc64e.
> > - Source tarball.
> > 
> > Are the binaries and source tarballs "additional release artifacts?"
> > 
> 
> Yes, good catch. I'll add 'GNU Guix Source tarball', I believe that the existing 'GNU GUIX installer' covers the binary point.
> 
> This section is covering the minimum created artifacts, which would be the focus of developer work and testing. Nothing stops additional artifacts being created - for example the latest downloads page (https://guix.gnu.org/en/download/latest/) has a Pinebook Pro image.

I'd also like to point out that I think at this point only Chris Baines
has direct access to all the machines we support, so IMO it goes without
saying that the release team would likely need to delegate creation of
the release artifacts for some architectures to someone with the
machine.  Of course if we have 2 someones then we can compare the built
artifacts.

> (...)
> > > ### 4. Toolchain and transition freeze
> > > No major changes to toolchains (e.g. gcc-toolchain, rust-1.xx) or
> > > runtimes
> > > (e.g. java).  There should be no changes that will cause major
> > > transitions.
> > > 
> > > Debian defines a transition as one where a change in one package causes
> > > changes
> > > in another, the most common being a library.  This isn't suitable for
> > > Guix since
> > > any change in an input causes a change in another package. Nonetheless,
> > > any
> > > change that alters a significant number of packages should be carefully
> > > considered and updates that cause other packages to break should be
> > > rejected.
> > > 
> > > No alterations to the Guix daemon or modules are accepted after this
> > > point.
> > > Packages and services in the 'minimal' package set should not be
> > > altered.
> > 
> > I think it’s worth considering doing this the other way around: instead of
> > freezing master, cut a release branch and cherry-pick fixes into it as
> > needed.  I don’t expect that development on non-release features will stop
> > during the freeze, which means we’ll have a large backlog of work to merge
> > once the freeze ends; this is a thing Guix has historically not been good at
> > working through in a timely manner.
> > 
> > A release branch would also support longer-term stable releases, if we
> > wanted to do that.
> > 
> > The downside is that it’s more work to cherry-pick fixes between branches,
> > and there’s the potential for merge conflicts.
> (...)
> 
> In the context of doing a release I think we _could_ cut the release branch earlier in the project. Rutherther also brought this up. It's definitely a trade-off. I selected to stay on the master branch for as long as possible because (a) the group has had difficulty with long-running branches, (b) the social signal of doing a release, for a few weeks we all focus on it. I agree we may block up with items to go to master, though given our current velocity a 5 week freeze doesn't seem that big to me.
> 
> I think that Efraim has an idea that would also try to reduce the freeze time - which I think is along the lines of what you're outlining Ian - Efraim do you want to chime in here?
> 
> I agree a release branch would also support some form of longer-term stable branch. I don't think doing it that way is a necessary precursor for this GCD, but I do take the point! For now I would keep the idea of maintaining two branches outside the scope of this GCD.

We historically haven't done well with freezes (see for example the
v0.9.0 tag in the git repo), and Free Software projects often loose a
lot of steam if a freeze is too long.  My idea was to freeze master for
about 2 weeks where everyone really just fixes packages that don't build
or are otherwise broken.  Then we'd branch off the release branch and
focus on release related work there and sync back regularly with master
as needed/possible.  The release would be cut from the release branch,
master would be merged into the release branch, and then the release
branch would be pushed to the master branch.  Then the two branches
converge and any new commits are downstream from either the regularly
rolling process or the release commit.

> 
> > > ### 7. Updates freeze
> > > Major package updates are frozen on 'master' as the focus is on fixing
> > > any
> > > blocking packages.  Security updates still go to 'master'.
> > 
> > This could prove troublesome, as the current Guix approach is to update
> > packages to the version containing the fix.  Are you thinking that we’d
> > maintain that, or adopt a Debian style of backporting fixes where possible?
> (...)
> 
> We'd maintain our current approach as we are ultimately a rolling release.
> 
> My point here was to hold us for a couple of weeks only pushing 'security' issues to the branch, so that the whole of the team can focus on testing the release and fixing release-critical bugs on the 'release' branch. For devs that don't want to focus on the release they can continue to push to their team-branches and/or the "staging" (z572 thinks this sould be called something else).

The aim is to to say "this package is at x.y. We can fix something by
going to x.y+1 or we can go all the way to x.y+25". Normally we'd go for
the big jump and fix things that break, here we'd aim for as small as
possible to try to keep everything working and not need to fix (as many)
broken packages.

> > > ### 8. Breaking changes to staging
> > > To avoid a period of time where teams can't commit breaking changes,
> > > these are
> > > sent to a new 'staging' branch, rather than directly to master. The
> > > master
> > > branch slows down from this week.
> > 
> > This is going to be difficult to manage, especially for contributions from
> > those new to Guix development.
> 
> We already have this problem in the sense that new contributors send contributions against master, and a lot of work is actually against team-branches. For example, people send package updates for rust/python and we already have them in rust-team/python-team branches. I agree it's difficult to manage as it will be a "new thing".
> 
> > Thank you for starting the discussion!
> (...)
> 
> It sounds like you're broadly in favour, but the main improvement would be flipping the branches around.
> 
> Thanks for the input!
> 
> Steve / Futurile
> 
> 
> [0] Ubuntu/Canonical LTS's are always the April release. RedHat is a bit less clear to me, but RHEL 9 was in May, and they've announced that RHEL 10 is mid-2025 so I _think_ they are also on a spring release cycle 
>  - https://www.redhat.com/en/blog/red-hat-enterprise-linux-10-beta-now-available - mentions "to release a new major edition every three years .... RHEL 10, due to reach general availability in mid-2025"
>  - https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux - RH9 May 2022, RH8 May 2019
> 
> 

-- 
Efraim Flashner   <[email-masked]>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted