Subject: Re: GCD005: Regular and efficient releases

From: Steve George <[email-masked]>
Date: Fri, 9 May 2025 22:24:28 +0100

On 9 May, Rutherther wrote:
> 
> Hi Steve,
> 
> thanks for this. I have a few notes left in the text, but generally I
> like this.
(...)

Thanks for the supportive comment, appreciate it and the detailed thoughts below.

> I think the proposal is missing the version numbering that will be used,
> like, let's say the releases are x.y.z, is y increased? Or should the
> numbering going to be switched to something like yyyy.mm?
(...)

AIUI the project did 1.2.0->1.3.0->1.4.0. There isn't a proposal to change the numbering of the releases. I don't think it's _required_ to change it to implement this GDC.

Equally I'm very comfortable with yyyy.mm, perhaps something to discuss within the group?

> Also, should the GCD document what happens if something goes horribly
> wrong after the release is made, ie. a new minor release created to
> mitigate the issue?
(...)

Always a risk! To keep the scope of this proposal realistic it only impacts the release artifacts so the installers. After an initial 'guix pull' the user will be back the rolling release of master. Consequently, for anything that goes "horribly wrong" the question is whether it's within the long-term "release artifacts", if it is then I agree the Release Team could re-roll the artefacts and label as a new minor release. I would leave it to the Release Team's discretion.

> Steve George <[email-masked]> writes:
(...)
> > The benefits will be:
> >
> > 1. New installations of Guix will be better because installation media and
> >    manuals will be up to date, and the initial 'guix pull' will be faster.
> 
> The initial pull won't be faster, the pull is so slow because there is
> no checkout and no commits checked for auth, not because the iso is so
> old. No matter the age of iso, the full clone has to be made and authenticated.
> If there was a checkout in the cache, it would be faster, the more up to
> date it is, the faster the pull is. Additionally what would greatly
> improve the experience is if this checkout has been copied to the
> installed system.
> But that's not related to the releases themselves, but to how the iso is made.

I agree the initial pull is a significant problem. Nicolas Graves told me at Guix Days 2025 of Andrew Tropin talking about not using Guix in a lab set-up because the initial pull was so significant that the lab would have finished before it had completed!

Some solutions were discussed at Guix Days 2025 which (e.g. shallow copies, using a checkout cache, finding different ways to chain the commit auth) I think we should prioritise. IF that happened, THEN we'd be in the situation where releasing new artefacts improves the user experience both at initial install and initial usage.

(...)
> > As a rolling release Guix immediately provides the latest improvements to
> > users.  Consequently, it could be argued that releases are unnecessary.
> > However, they provide a focal point for the project to undertake additional
> > testing and stabilisation across the repository.  They also ensure we update
> > installation media, documentation, themes and web site.
> 
> I think releases are also needed for other distro package managers to
> update. While users are encouraged to use the guix-install.sh script,
> some will end up installing with their system's package manager. Maybe
> it would be good to mention that?
(...)

Agreed, I can add that. 

Perhaps Vagrant can comment on the postive (or negative) aspects of having a regular release from Guix when packaging Guix for a distribution (Debian). 

> > Many desktop distributions release every six months to align with the major
> > desktop environments (GNOME/KDE) who release two or three times a year. This is
> > why Nix (May/November) and Ubuntu (April/October) align their releases.
> >
> > Since Guix is used [extensively as a desktop] (https://guix.gnu.org/en/blog/2025/guix-user-and-contributor-survey-2024-the-results-part-1/)
> > it would make sense to align with these upstream releases.  However, given that
> > Guix is a volunteer project that doesn't have the practise of releasing it's
> > unrealistic to move to two releases a year. Something along these lines could
> > be a future goal [^5].
> >
> > This GCD proposes an annual release cycle, with releases **in May**.
> 
> Is May smart if GNOME is to be targeted? I think NixOS really struggles
> to make the GNOME updates in time. And there is less people working on
> it on Guix as far as I know. Gnome is like two releases back? If we
> wanted to include GNOME, wouldn't June make more sense to give an extra month?
> 
> Or for a better approach: is there a specific list of packages that
> would be targeted so it can be used for the decision?
(...)

The number of people we have working on Guix, and the bandwidth they have "as volunteers" is the constraint we're working with.  In a context where our archive is very large compared to those that many core teams work on (see extended notes on other distributions I put togeher). The balance I've tried to strike is how to make a concrete step forward in predictability/stability without having unrealistic expectations on what we can achieve. I hope that's clear in the GCD!

Given those constraints a practical step forward it to introduce `package sets` which limit the amount of work around a release. This doesn't directly address whether a particular desktop is up to date, but it does mean we don't spend time fixing parts of the package archive that don't directly lead to an installed system (the example I use is games).

The Release Team would have to work in conjunction with the Teams on the package sets, there's initial work here:

  https://git.savannah.gnu.org/cgit/guix.git/tree/etc/manifests/release.scm

There's a possible secondary "focus" benefit where we'd all know that we're targeting $SPRING for a release, so the whole group would (I hope) pay attention on the key packages in the package sets.

I have no strong feel on when in the $SPRING it is, so if the group wants June then great.

(...)
> > # Appendix 1: Release Project Time-line
> >
> > To illustrate the major steps of a release project this is a rough time-line.
> > The aim is for a 12 week active release project, with the first one using 16
> > weeks in total to give teams time to prepare.
> >
> > The current outline builds from our current [release document](https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org)
> >
> > | Week of Project | Event |
> > | --- | --- |
> > | -5 | Nominate a release team |
> > | -4 | Notify teams of upcoming release |
> > | 01 | Release project start |
> > | 04 | Toolchain and transition freeze |
> > | 05 | Package set finalisation |
> > | 06 | Initial testing |
> > | 08 | Updates freeze |
> > | 08 | Breaking changes to staging |
> > | 08 | Ungraft master branch |
> > | 09 | Bugs and documentation focus |
> > | 09 | Branch and tag release branch |
> > | 09 | Testing and hard freeze |
> > | 10 | Release candidate |
> > | 12 | Final release |
> > | 13 | Staging merged to master |
> > | 14 | Release retrospective |
> > | 15+| Relax - it's done! |
(...)
> > ### 4. Toolchain and transition freeze
> > No major changes to toolchains (e.g. gcc-toolchain, rust-1.xx) or runtimes
> > (e.g. java).  There should be no changes that will cause major transitions.
> 
> So total of 9 week freeze? Isn't that too long for users to not receive
> those updates if they were ready?
> 
> I think that should be at least mentioned in the drawbacks, even if most
> people won't consider it a major drawback, it sounds like a drawback to me.
> 
> But I suppose it will still be possible to commit those changes, you
> just won't be able to switch the default toolchain. Though Rust, Python
> etc. currently don't really support more than one version for building
> other packages, at least not well.
(...)

This sets out a specific freeze period where we wouldn't make those changes once a year, so 9 weeks in a year for major transitions.

It's somewhat arguable about how quickly we're making transition changes at the moment. Look at the available versions of toolchains from Perl, Python and these are taking 3-4 months on team branches. 

If we were in a position where we regularly had transitions ready to go ("if they were ready") then we could look at ways to shorten. 

To avoid developers feeling blocked up toolchain/transition changes can go to a <staging or some-name> branch.

(...)
> > Debian defines a transition as one where a change in one package causes changes
> > in another, the most common being a library.  This isn't suitable for Guix since
> > any change in an input causes a change in another package.  Nonetheless, any
> > change that alters a significant number of packages should be carefully
> > considered and updates that cause other packages to break should be rejected.
> >
> > No alterations to the Guix daemon or modules are accepted after this point.
> 
> What modules exactly, all? And what kind of alterations? I suppose some
> changes will still be possible, like adding a new package or service,
> no?
> I think it doesn't make sense to completely block alterations to the
> daemon, ie. if there was a bug or security issue.

I'll amend that to say 'Guix daemon' as this is still the same concept about breaking changes.

In _all cases_ security updates will still be allowed as we don't want to negatively impact users.

New updates and packages keep flowing through master until 'Updates freeze' (week 8).

>
> > Packages and services in the 'minimal' package set should not be altered.
> >
> > ### 5. Package set finalisation
> > Specify the package sets for this release.  Identify all packages and their
> > inputs so that a full manifest for the release is created.
> >
> > ### 6. Initial testing
> > An initial testing sprint to look at packages, services, install media and
> > upgrade testing. This should identify:
> >
> > * packages or services that may need to be removed because they fail on a
> >   primary architecture
> 
> So hypothetically, if a package/service worked on non-primary
> architecture or one primary architecture, but not the other, it will be
> removed, completely? That doesn't sound right. Am I misunderstanding
> here? Where is the package removed from, the guix repository or just
> from the package set that should be passing for the release?
> Either way, like this it conflicts with the earlier 'packages and
> services in the minimal package set should not be altered', so I think
> it should be mentioned it doesn't apply to those packages.
> 

In terms of a package that works on one primary architecture but not on another the Release Team would have some options on how to mitigate:

1. Fix it to work on both architectures
2. Promote an alternative for that architecture
3. Document as a limitation, with a workaround in the release notes
4. Remove it from the archive

In part the discussion should be about whether it's a regression or a case where package 'foo' has never worked on that architecture. I would leave it to the Release Team to make a case-by-case decision.

If the decision was to remove the package note that it would go through the current Deprecation Policy which notifies and gives developers a two months to pick up the package.

The section earlier about the Platforms and Architecture tiers should lay this out clearly - anything I can improve there to make it clearer? 

> > * packages and services in the package sets install and can be used
> > * installation artifacts can be created and used
> > * example system definitions can be used
> > * system upgrades
> >
> > A build failure of a package or service will result in it being identified for
> > removal.  A build failure of a package or service that's in a package set will
> > be marked as a blocker for the release.
> >
> > ### 7. Updates freeze
> > Major package updates are frozen on 'master' as the focus is on fixing any
> > blocking packages.  Security updates still go to 'master'.
> 
> So 5 weeeks. (I am counting the retrospective week btw)
> Maybe the text should mention how many weeks the freeze is for so it's
> easier to orient? It seems important to me.

From 'Updates Freeze' (week 8) to 'Staging merged to master' (week 13) so 5 weeks in this plan. For those 5 weeks updates would go to the team branches or to 'staging'. After staging is merged (week 13) then team branches can be merged into master and we proceed as normal.

The main thing is how short a time period can we do this in, since we'd only have from week 8->10 to ungraft the master branch, do the release branch, test the release and fix any RC bugs.

The balance is a window that lets us focus on the release without divergence that would destabilise, versus a long closed window which is bad for our rolling release. Given the volunteer nature of the team, 5 weeks seemed about right but we can iterate this as we learn.

In my mind this project plan is an initial one, and then as we move forward from release to release each Release Team can iterate and improve.

> >
> > ### 8. Breaking changes to staging
> > To avoid a period of time where teams can't commit breaking changes, these are
> > sent to a new 'staging' branch, rather than directly to master.  The master
> > branch slows down from this week.
> >
> > This concept comes from the Nix project where they flow big changes into a
> > staging branch while they do release stabilisation to prevent big flows of
> > breaking changes into master which broke one of their releases [^7].
> 
> Nix doesn't use staging because of that, they use it always, mostly
> because the CI cannot build the whole package set. They don't use
> grafts, so there are world rebuilds like two times a month, when the
> staging merges to master, and they do that only if staging has already
> been built so that people get substitutes.
> 
> I think this is a misunderstanding, the linked text states that breaking
> changes shouldn't be merged to staging branch, not to master branch. In
> turn, it also isn't merged to master branch, because staging is merged
> there, quite regularly. This can be most easily seen from the table they
> have with affected branches mentioned.
> 
> So while I agree with the general motion that changes should be merged
> somewhere else, it's not right to state Nix does this.

Overloaded use of "staging" - I should have used a different branch name.

> By the way, since you linked it already, citing their drawbacks
> 
> Citing from [^7]:
> > Breaking changes to Release Critical Packages will have to wait a
> > maximum of 4 weeks to be merged into staging. Other breaking changes
> > targeting staging will have to wait a maximum of 2 weeks to be merged.
> > Staging-next iterations will follow a 1 week development cycle during
> > the release timeline. Breaking changes to Release Critical Packages
> > will not appear in master for around 7 weeks (4 weeks being disallowed
> > to be merged to staging, 1 week for last staging-next during ZHF, ~2
> > weeks average in staging-next). Pull requests which are able to target
> > master will not be interrupted d

Their RFC is worth reading, I found it really interesting. It's definitely a balancing act between wanting to do stabilisation and meanwhile teams want to move forward. If too many changes flow in then the whole release is destabilised, equally how long a closed window makes sense!

> So they do have 7 week period to not merge anything critical to master,
> that's less than what is proposed here. I think it should be kept to as
> low number as reasonably possible. That probably have to be analyzed,
> and I think the longer time from bigger projects, like GNOME, the more
> time to prepare to it even before the release schedule kicks in.
(...)

For everyone else who isn't as aware (as you are) of Nix's details:

1. I encourage you to read their release wiki which covers many of the aspects they are trying to balance: https://nixos.github.io/release-wiki/Release-Editors.html

2. I worked up a summary of Nix's release strategy (as I understood it): https://codeberg.org/futurile/guix-org/src/branch/master/release-mgmt/nix-release-mgmt.md

I would also love PR's on the release management notes to fix things etc! I covered Nix, Arch and Ubuntu - I also have some raw notes on Fedora if anyone's interested.

I agree with as "low as possible", I've been cautious because it's a small volunteer team - I'm open to what the group thinks.

> Back to Steve's message:
> >
> > Team branches can still be folded into the release branch as long as changes are
> > minor package upgrades.
> >
> > ### 9. Ungraft master branch
> > Guix master is ungrafted to minimise the difference with users of the release
> > initial 'guix pull' experience.
> 
> Could you clarify what this means?

Someone more experienced that me should address this aspect in all it's gory details!

> > ### 10. Bugs and documentation focus
> > The master branch should be quiet at this point as everyone should focus on
> > testing and resolving any bugs.  New documentation can also be done.
> >
> > ### 11. Branch and tag release branch
> > The master branch is tagged and a new release branch is created.
> >
> > * master branch: security only.
> > * release branch: security updates as normal. Only RC blocking bugs.
> >
> > Only security updates go to the master branch from week 9->13.  All other
> > changes stay in a team branch or go to the `staging` branch. The focus on the
> > release branch is to stabilise so only RC bugs should be pushed.
> 
> Am I reading correctly, no updates whatsoever for 4/5 weeks!! :O
> Shouldn't this at least only affect the packages in the package sets?
> Even so, feels like too long.
(...)

Correct, only security to master for 5 weeks. 

Since we have to degraft the whole of the tree, cut a release, test that it installs as a Linux distro, and that it installs as a package manager on the popular Linux distros - this time will fly!

I actually don't have a fixed view on the window, so open to what the group thinks is achievable! 

Steve / Futurile