Subject: Re: GCD005: Regular and efficient releases

From: reza <[email-masked]>
Date: Sat, 10 May 2025 10:56:03 +0000

Hi Steve

Thanks for the effort you put into this, I just wanted to add my
thoughts when reading it...

> Hi all,
>
> It's been ~2.5 years since the last Guix release, which led me to think we should do another one! Initially, I was just going to see if anyone wanted to create a release project. But, before I knew it I was writing a GCD! ...
>
> Below you'll find a proposal for moving to a regular release cycle.
>
> Thanks to Andreas Enge, Ludovic Courtès and Efraim Flashner for their initial comments and insights. They've agreed to sponsor it - which means they agree with the general direction (but not necessarily with all aspects), and will help to 'garden' the discussion to move us towards a consensus on whether it's a good proposal.
>
> This is a **draft** submission so I look forward to your consideration of it, thoughts and comments.
>
> Thanks,
>
> Steve / Futurile
> title: Regular and efficient releases
> id: 005
> status: draft
> discussion: https://issues.guix.gnu.org/78332
> authors: Steve George
> sponsors: Andreas Enge, Ludovic Courtès, Efraim Flashner
> date: <date when the discussion period starts>
> SPDX-License-Identifier: CC-BY-SA-4.0 OR GFDL-1.3-no-invariants-or-later
> ---
>
> # Summary
>
> Guix doesn't have regular release cycle which has led to infrequent new
> releases. Sporadic releases are detrimental for our users, contributors and
> the project.  This GCD proposes we implement an annual release cadence and
> simplify the release process to make releases easier.
>
> The project currently releases new versions of Guix on an ad hoc frequency.
> The 1.4.0 release happened in December 2022 [^1], which is almost 2.5 years
> ago, at the time of writing.
>
> The weaknesses in this release strategy are:
>
> 1. No clarity on when the next Guix release is.
> 2. Releases are complex and toil for developers.
> 3. Rolling updates aren't suitable for all users.
>
> This GCD proposes the following combined solution:
>
> 1. Regular releases: switch to a time-based release of Guix every year.
> 2. Efficient releases: use *package sets* and *supported architectures* to
> reduce the scope of work required to create a Guix release.
>
> The benefits will be:
>
> 1. New installations of Guix will be better because installation media and
>    manuals will be up to date, and the initial 'guix pull' will be faster.
> 2. Package installation will improve for all users.  Packages will be ungrafted
>    during each release cycle.
> 3. Package quality will improve for all users, because regular releases will
>    provide a cadence for focusing on our quality.
> 4. A regular cadence for promoting the project to potential users.  Helping us
>    to inform more people about the benefits of using GNU Guix!
> 5. A regular release cycle is a rallying point for our contributors giving them a
>    consistent calendar of when to focus on releases versus other hacking.
>
> Adding a slower-moving branch akin to Nix's stable could be an eventual goal as
> it would increase Guix's suitability for some users and use-cases [^2].  However,
> this GCD only sets out to implement regular releases which is a substantial
> change that would be a big improvement for our users.
>
>
> # Motivation
>
> Releases are important for any Free Software project because they update the
> user experience and are a focal point for excitement [^3].  Regular releases
> help us to improve the quality of our software by bringing focus, and
> exercising regular usage scenarios (e.g. testing the installer).
>
> The majority of distributions follow time-based releases, six months is a
> common cycle time.  For further comparison see the research on the
> [release strategies of distributions] (https://codeberg.org/futurile/guix-org/src/branch/master/release-mgmt)
> . A summary is:
>
> - NixOS: releases every six months (May/November), both rolling release and
> slower stable branch.
> - OpenSUSE: rolling release, slow-roll release and fixed releases.
> - Ubuntu: every six months, with 9 months maintenance. LTS releases every
> 2 years.
> - Debian: Every 2 years (not time fixed), with about six months of package
> updates freeze before the release while bugs are ironed out.
>
> As a rolling release Guix immediately provides the latest improvements to
> users.  Consequently, it could be argued that releases are unnecessary.
> However, they provide a focal point for the project to undertake additional
> testing and stabilisation across the repository.  They also ensure we update
> installation media, documentation, themes and web site.
>
> A specific issue caused by irregular releases is that new users/installs face a
> significant first "guix pull". This provides a poor initial user experience,
> and in some cases may even deter users [^4]. Additionally, it requires the
> project to keep old substitutes on our servers.
>
> Regular releases are also good for casual users because they provide an
> opportunity for us to promote new features and improvements.  For prospective
> users promotional activity about the release means they are more likely to hear
> about capabilities that will attract them to experiment with Guix.
>
> Many desktop distributions release every six months to align with the major
> desktop environments (GNOME/KDE) who release two or three times a year. This is
> why Nix (May/November) and Ubuntu (April/October) align their releases.
>
> Since Guix is used [extensively as a desktop] (https://guix.gnu.org/en/blog/2025/guix-user-and-contributor-survey-2024-the-results-part-1/)
> it would make sense to align with these upstream releases.  However, given that
> Guix is a volunteer project that doesn't have the practise of releasing it's
> unrealistic to move to two releases a year. Something along these lines could
> be a future goal [^5].
>
> This GCD proposes an annual release cycle, with releases **in May**.
>
> To move onto this cycle the first release would be a little later: aiming for
> the **November of 2025**, with a short cycle to release in May 2026.
>
>
> ## Package Sets
>
> There are currently over 30,000 packages in the archive, it's unrealistic for
> all packages to receive the same amount of QA effort for a release.
>
> Many other distributions focus attention on the critical parts of their
> repository by identifying those packages that are required for a particular
> user-case.  For example, Arch Linux limits their efforts to a specific
> repository (called "main").  Ubuntu identifies various seeds for specific
> use-cases which determines their maintained packages; other packages outside
> these sets do not receive security updates.
>
> Guix is both a package manager that can be hosted on other Linux distributions,
> and a Linux distribution.  Limiting the range of packages that receive attention
> is consequently more complicated. Guix already has manifests to track which
> packages are used by [Guix System's installer](https://git.savannah.gnu.org/cgit/guix.git/tree/etc/manifests/release.scm)
> , so this proposal extends that concept.
>
> For this proposal Guix would identify key package sets which would receive the
> most attention for QA'ing a release.
>
> The package sets would be:
>
> * minimal: packages required to boot and install a minimal Guix or Guix System
> install.  This would include the guix daemon, kernels, boot loaders,
> file-systems and minimal utilities.
> * standard: packages that create a core installation for all other uses.
> * desktop: provides the packages necessary for a desktop.  This would include
> things like X11/Wayland, desktop environments, key applications and themes.
> * server: provides packages and services for a server.  This would include
> standard server packages and services.
> * hosted: provides packages and services which are necessary in hosted usage.
>
> Guix would still make all packages and services part of a release (the entire
> archive), but only those in the `package sets` could block a release due to a
> significant bug.  The goal would be for this to be as small a set of packages
> as reasonably possible.  It would mean that developers could focus on the
> critical packages and services during a release.  As an example, this would
> mean that a major issue in the Linux kernel could block a release, but not an
> issue in a game.
>
>
> ## Platforms and Architecture tiers
>
> Guix is built and maintained on multiple different architectures, and two
> kernels (Linux, GNU Hurd).  As the goal of the project is to maximise user
> freedom this variety is significant and is a key motivator for developers.
>
> However, with limited resources (developer and CI) we want to make it as
> efficient as possible to create a release.  The more toil involved in a release
> the less likely developers are to work on it.
>
> The [2025 Guix User Survey](https://guix.gnu.org/en/blog/2025/guix-user-and-contributor-survey-2024-the-results-part-2/)
> showed that 98% of users were on x86_64 and 19% on AArch64.  Consequently, the
> proposal is the following tiers:
>
> - Primary architectures:
>   - Architectures: x86_64, AArch64
>   - Kernel: Linux
>   - Coverage: all packages and services that are not explicitly platform
>     specific must work to be included in the archive.
>   - Package status: package updates should build for this architecture.  If a
>     package update is broken it must not be pushed to users (e.g. master).
>   - Security: all packages that are maintained upstream receive updates
>
> - Alternative architectures
>   - Architectures: all others
>   - Kernel: Linux, Hurd
>   - Coverage: all packages and services that are not explicitly platform
>     specific may build
>   - Package status: package updates should work for this architecture.
>     Updates that do not build for this architecure, but do build for a primary
>     architecture may be pushed to users.
>   - Security: no commitment to providing security updates for this architecture.
>
> Packages or services that do not build for the Primary architectures as part of
> a release would be removed from the archive using Guix's deprecation policy.
>
>
> ## Release artifacts
>
> Using the primary architecture tier and the package sets would involve creating
> the following release artifacts:
>
> - GNU Guix System ISO image
> - GNU Guix System QCOW2 image
> - GNU Guix installer
>
> Again in an effort to reduce developer toil, additional release artifacts could
> be created but would not be part of the formal release testing and errors would
> not block a release.
>
>
> ## Release team and project
>
> A regular release cycle should galvanise all Guix developers to work on our
> releases.  But, to ensure there are sufficient people involved a call will be
> put out to create a specific release team for each release project.  We would
> expect the final release team to be around four-six members. The release team
> will work together to fix issues and test the various release artifacts. The
> expectation is that the release projects will be as short as possible, around
> a 12 week commitment with each team member having a few hours a week to take
> part.
>
> To manage the release it's proposed that each release will have a Release Manager
> role. The role of the Release Manager is to: 
>
> - co-ordinate the release project
> - communicate with the release team and wider developers status and blockers
> - arbitrate changes to release blocking bugs, package sets and release
>   artifacts
> - influence and assist teams to resolve problems
> - define the release artifacts and create them
> - encourage and excite **everyone to create and test the release**
>
> The Release Management role is likely to require the most effort, so it will
> be rotated and consist of two people from the release team.  For each release
> there would be a primary person and a secondary person in the role.  The
> primary person is new to the role.  The secondary person has previously done it
> and is mentoring the new person.  The impact of this is that each new release
> manager is agreeing to take responsibility during two release cycles.
> This system is modelled on the [Nix release management](https://codeberg.org/futurile/guix-org/src/branch/master/release-mgmt/nix-release-mgmt.md)
> approach.
>
> One of the release team will take on the role of Release Advocate [^6].  They
> will take responsibility for preparing the release announcement and 
> coordinating the creation of content to promote the new release (e.g. web site)
> This role can be done by any member of the Guix community who has sufficient
> interest.
>
> The final release team is:
>
> - a new Release Manager
> - a returning Release Manager
> - up to 4 other members, one of whom acts as the Release Advocate
>
> The Release Managers of each release will create and communicate a release
> project plan setting out the stages and dates for each stage.  To try and
> galvanise the Guix development team to focus on the release it's envisioned
> that a release project will be about 12 weeks. See Appendix 1: Release Project
> Time-line for an example.
>
> In order to improve knowledge transfer and reduce the toil of doing releases
> the Release Managers for a release will document the release process.  There is
> inspiration for this in [NixOS's release wiki](https://nixos.github.io/release-wiki/Home.html)
> and we already have detailed [release documentation](https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org)
> .
>
>
> # Cost of Reverting
>
> If regular releases were not successful then the project would switch back to
> irregular releases.  There would be no impact for exiting users as they will
> be tracking the rolling release's master branch.
>
> If the project is able to successfully undertake regular releases then over
> time it may be possible to undertake full releases every six months or some
> other release cadence.
>
>
> # Drawbacks and open issues
>
> There's no particular drawback to attempting regular release.  It should be
> noted that the project is entirely dependent on volunteers so it may be that
> contributors don't have enough time available to achieve regular releases.  If
> that's the case we would revert back to irregular releases.
>
> There are various improvements that could be made to this process over time,
> but no known issues.
>
>
> # Appendix 1: Release Project Time-line
>
> To illustrate the major steps of a release project this is a rough time-line.
> The aim is for a 12 week active release project, with the first one using 16
> weeks in total to give teams time to prepare.
>
> The current outline builds from our current [release document](https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org)
>
> | Week of Project | Event |
> | --- | --- |
> | -5 | Nominate a release team |
> | -4 | Notify teams of upcoming release |
> | 01 | Release project start |
> | 04 | Toolchain and transition freeze |
> | 05 | Package set finalisation |
> | 06 | Initial testing |
> | 08 | Updates freeze |
> | 08 | Breaking changes to staging |
> | 08 | Ungraft master branch |
> | 09 | Bugs and documentation focus |
> | 09 | Branch and tag release branch |
> | 09 | Testing and hard freeze |
> | 10 | Release candidate |
> | 12 | Final release |
> | 13 | Staging merged to master |
> | 14 | Release retrospective |
> | 15+| Relax - it's done! |
>
> ### 1. Nominate a release team
> Nominate a release team with two Release Managers (1 is the previous RM), and
> up to 4 other people who will work on the release. Put out a call for a Release
> Advocate who can be anyone in the Guix community who's willing.
>
> ### 2. Notify teams of upcoming release
> Make sure all teams are aware of the upcoming release.  This gives them 4 weeks
> to undertake any large transitions or major changes.
>
> ### 3. Release project start
> Start the project with weekly updates to guix-dev and regular meetings of the
> release team.  Encourage participation in testing and identifying bugs from
> the community.
>
> ### 4. Toolchain and transition freeze
> No major changes to toolchains (e.g. gcc-toolchain, rust-1.xx) or runtimes
> (e.g. java).  There should be no changes that will cause major transitions.
>
> Debian defines a transition as one where a change in one package causes changes
> in another, the most common being a library.  This isn't suitable for Guix since
> any change in an input causes a change in another package.  Nonetheless, any
> change that alters a significant number of packages should be carefully
> considered and updates that cause other packages to break should be rejected.
>
> No alterations to the Guix daemon or modules are accepted after this point.
> Packages and services in the 'minimal' package set should not be
> altered.

Should this not be done on a separate release branch as soon as we start
with the release? There seems to be no need to freeze master as soon as
we branch off a release branch. Or did I understand something wrong?
>
> ### 5. Package set finalisation
> Specify the package sets for this release.  Identify all packages and their
> inputs so that a full manifest for the release is created.
>
> ### 6. Initial testing
> An initial testing sprint to look at packages, services, install media and
> upgrade testing. This should identify:
>
> * packages or services that may need to be removed because they fail on a
>   primary architecture
> * packages and services in the package sets install and can be used
> * installation artifacts can be created and used
> * example system definitions can be used
> * system upgrades
>
> A build failure of a package or service will result in it being identified for
> removal.  A build failure of a package or service that's in a package set will
> be marked as a blocker for the release.
>
> ### 7. Updates freeze
> Major package updates are frozen on 'master' as the focus is on fixing any
> blocking packages.  Security updates still go to 'master'.
>
> ### 8. Breaking changes to staging
> To avoid a period of time where teams can't commit breaking changes, these are
> sent to a new 'staging' branch, rather than directly to master.  The master
> branch slows down from this week.
>
> This concept comes from the Nix project where they flow big changes into a
> staging branch while they do release stabilisation to prevent big flows of
> breaking changes into master which broke one of their releases [^7].
>
> Team branches can still be folded into the release branch as long as changes are
> minor package upgrades.
>
> ### 9. Ungraft master branch
> Guix master is ungrafted to minimise the difference with users of the release
> initial 'guix pull' experience.
>
> ### 10. Bugs and documentation focus
> The master branch should be quiet at this point as everyone should focus on
> testing and resolving any bugs.  New documentation can also be done.
>
> ### 11. Branch and tag release branch
> The master branch is tagged and a new release branch is created.
>
> * master branch: security only.
> * release branch: security updates as normal. Only RC blocking bugs.
>
> Only security updates go to the master branch from week 9->13.  All other
> changes stay in a team branch or go to the `staging` branch. The focus on the
> release branch is to stabilise so only RC bugs should be pushed.
>
> ### 12. Testing and Hard Freeze
> RC bugs and issues should be solved for the release branch.
>
> Only changes that will fix a non-building package, or a bug in a package are
> allowed.  Ideally avoid new upstream versions, but it's acceptable to use a new
> minor upstream version to solve a bug.
>
> Any non-building packages are removed.
>
> ### 13. Release candidate
> Release artifacts are created for the release candidate.  There's a final 2
> weeks of testing with these artifacts.  If there are no release blocking bugs
> discovered then the releas uses these artifacts.  If bugs are found/fixed then
> release artifacts are regenerated as needed.
>
> ### 14. Final release
> Final release is announced and new release artifacts are published.
>
> ### 15. Staging merged to master
> If there were any breaking changes placed onto the `staging` branch then these
> can be merged into the `master` branch at this point.  The master branch then
> continues as normal.
>
> ### 16. Release retrospective
> A retrospective is undertaken by the release team to understand how the release
> process can be improved to make it more reliable for users and easier/efficient
> for developers.
>
> ### 17. Relax!
> The release has been cut, everyone is now excited, and hopefully all is well.
> Take some time off from release work!  There's some time built-in here to
> relax and get back to other hacking before it's time to start again with the
> next release.
>
> ---
>
> [^1]: https://guix.gnu.org/en/blog/2022/gnu-guix-1.4.0-released/
>
> [^2]: Examples of distributions that have cadences for different users and screnarios
>       are Nix's stable branch, OpenSUSE's SlowRoll branch and Ubuntu's LTS
>       (Long Term Support) releases.
>
> [^3]: the aspect of creating news and excitement for casual users is well-known
>       in the FOSS community. An example from 
>       [GNOME's earlier days](https://mail.gnome.org/archives/gnome-hackers/2002-June/msg00041.html).
>
> [^4]: GuixDays 2025 release discussion brought up examples of Guix not being
>       used to teach users because the initial pull was so slow that the
>       teaching session would have completed before 'guix pull' would finish.
>       We know guix pull being slow was identified by users as a challenge.
>
> [^5]: OpenSuse has a [SlowRoll branch](https://en.opensuse.org/Portal:Slowroll)
>       where they release a smaller set of package updates on a monthly basis.
>       This is an interesting innovation as it allows users to still benefit from
>       a rolling release but at a slower rate of change (fewer regressions).
>       They are also not dropping too far behind the rolling release, so there's
>       not as much maintenance for OpenSUSE developers dealing with an out of
>       date release branch and having to backport software.
>
> [^6]: Nix has the concept of a [Release Editor](https://nixos.github.io/release-wiki/Release-Editors.html)
>       who is responsible for improving the legibility of the release notes.  Our
>       version extends the idea to make sure other artifacts and activities that
>       promote the release happen.
>
> [^7]: https://github.com/NixOS/rfcs/blob/master/rfcs/0085-nixos-release-stablization.md

I am missing the automation aspect a little in this GCD. As I understand
the release take so long because there is a lot of manual effort
involved in doing one. I think it is worthwhile to also work towards
more automation in preparing a release and testing it. I have by no
means a lot of insight into doing a release but my impression is that
automation could drastically improve the time to relase.

Thanks again!

Best,
Reza