Subject: Re: GCD005: Regular and efficient releases

From: Steve George <[email-masked]>
Date: Sat, 10 May 2025 10:42:43 +0100

Hi Vagrant,

On  9 May, Vagrant Cascadian wrote:
> On 2025-05-09, Steve George wrote:
> > It's been ~2.5 years since the last Guix release, which led me to
> > think we should do another one! Initially, I was just going to see if
> > anyone wanted to create a release project. But, before I knew it I was
> > writing a GCD! ...
> 
> Thanks for nudging this forward! :)
(...) 

Thanks for your active interest and commenting!

(...)
> > - Debian: Every 2 years (not time fixed), with about six months of package
> > updates freeze before the release while bugs are ironed out.
> 
> Maybe more like 4 or 5 months, but who's counting? :)
> 
> Notably, Debian has time based freezes, and releases "quando paratus
> est" ... e.g. when it is ready (e.g. no major blockers to release).
> 
> I think this is a more realistic and honest model for a volunteer
> project than trying to make a time-based "release", as you can predict
> or decree when you are going to start the process, but you cannot
> reliably predict when you will finish.
(...)

I researched various other distributions release strategies to make sure my recollections were up to date (linked to my codeberg account). Echoing your comment here I noticed that Fedora has come up with an interesting element - they are still time-based but they have "buffer weeks with early and later release targets" [0]

They run a 10 week release project (they have full-time devs), and then have four possible release weeks, so it gives them quite a nice landing zone. The still have the benefits of a consistent release cadence, but have some of the time to do "no major blockers to release". It's easy to see in their current step-by-step plan:

https://fedorapeople.org/groups/schedule/f-43/f-43-key-tasks.html

Conceptually, do you think adding something that provides 3-4 landing zone weeks would be useful? 

> > Many desktop distributions release every six months to align with the major
> > desktop environments (GNOME/KDE) who release two or three times a year. This is
> > why Nix (May/November) and Ubuntu (April/October) align their releases.
> ...
> > This GCD proposes an annual release cycle, with releases **in May**.
> >
> > To move onto this cycle the first release would be a little later: aiming for
> > the **November of 2025**, with a short cycle to release in May 2026.
> 
> For what it is worth, May would be awful timing for getting guix into
> Debian stable releases, which tend to approach a hard freeze around that
> time of year, so Debian stable will always end up effectively (at least)
> one release behind... but you cannot please everybody. :/
> 
> A yearly release cycle would at least allow Debian to provide backports
> of guix more reliably, at least...
> 
> Of course, we are about to see the second Debian release since the
> release of guix 1.4.0, so this proposal would still be a huge
> improvement!
(...)

I think here you're +1 on an annual release as it would allow for backports.

In terms of the Debian freeze is moving to June for Guix any better/worse, (Rutherther suggested it), or it's just anything in $SPRING?

I guess there will always be a bit of a log-jam if the whole of the community is somewhat aligning. In this case Debian is both a distribution (like Guix) aligning on a $SPRING release, and a downstream that Guix is trying to get into.

  {upstream projects} -> {guix/fedora/ubuntu/debian} -> {downstream into Debian}

> > ## Package Sets
> ...
> > Guix is both a package manager that can be hosted on other Linux distributions,
> > and a Linux distribution.  Limiting the range of packages that receive attention
> > is consequently more complicated. Guix already has manifests to track which
> > packages are used by [Guix System's installer](https://git.savannah.gnu.org/cgit/guix.git/tree/etc/manifests/release.scm)
> > , so this proposal extends that concept.
> >
> > For this proposal Guix would identify key package sets which would receive the
> > most attention for QA'ing a release.
> >
> > The package sets would be:
> >
> > * minimal: packages required to boot and install a minimal Guix or Guix System
> > install.  This would include the guix daemon, kernels, boot loaders,
> > file-systems and minimal utilities.
> 
> This seems a reasonable broad-strokes starting point, nice!
> 
> 
> > * standard: packages that create a core installation for all other
> > uses.
> 
> I fail to see a distinction between minimal and standard, perhaps due to
> "core installation", so this perhaps could use a bit more elaboration.
> 

The package sets are a starting point, we don't have them, so the specific detail could change as this is worked through.

This set-up is a pared-back version of what we worked with in Ubuntu [1]. AFAIK this concept comes from Debian (seeds).

We had a minimal 'package set' that was used for all variants (e.g. used in the desktop and server install). Then `standard` was the next block-up with the minimum core utils for a non-X install.

Maybe having both is over-complicated for a first pass.

> > * desktop: provides the packages necessary for a desktop.  This would include
> > things like X11/Wayland, desktop environments, key applications and themes.
> 
> It seems like all desktop environments is awfully broad, as there are
> some unusual or niche desktop environments in Guix, although Guix users
> maybe also tend to fill certain niches more than the general
> population. Still, I suspect this should maybe be narrowed down to a
> more specific set...

Agreed.

My perspective on 'package sets' is that they should be kept as small as reasonably possible, because once something is in then it's difficult to remove it since users 'workflow [2] will come to rely on it. Of course, what is "reasonable" is up for debate within the group, and it's part of each Release Teams project to consider it.

We currently have kind of defacto selection of desktops in:

https://git.savannah.gnu.org/cgit/guix.git/tree/etc/manifests/release.scm

To avoid derailing the conversation around the GCD I would defer to the Release Project to discuss the precise definitions of the `package sets`.

> 
> > * server: provides packages and services for a server.  This would include
> > standard server packages and services.
> 
> What are "standard server packages and services" ?
> 

From the Guix User Survey we know that roughly a third of Guix System users use it as a server [3]. But, our teams are actually set-up by file-kinda-build-system-type. So there's a dissonance between users usage-scenarios and our internal organisation. Package sets might allow us to align these two since we can have teams that are sponsoring (using Greg's terminology) different packages in the package set.

As we don't have popcon I don't know what's popular with users, but I was thinking of things like Web server, Mail, etc

>
> > * hosted: provides packages and services which are necessary in hosted usage.
> 
> What are these?
(...)

This is the weakest one as I have no clue. I was thinking that we need to reflect a unique part of Guix which is that we are not just a GNU/Linux distribution but also a hosted (foreign) package manager. If there are packages that are required/popular when using in a hosted set-up, then they would be in this package set.

You've caught me trying to slide it past, because I couldn't actually think of any at the time heh :-)

(...) 
> > ## Release artifacts
> >
> > Using the primary architecture tier and the package sets would involve creating
> > the following release artifacts:
> >
> > - GNU Guix System ISO image
> > - GNU Guix System QCOW2 image
> > - GNU Guix installer
> >
> > Again in an effort to reduce developer toil, additional release artifacts could
> > be created but would not be part of the formal release testing and errors would
> > not block a release.
> 
> Do we currently have all these for x86_64-linux and aarch64-linux, the
> two proposed primary architectures?
> 

According to https://guix.gnu.org/en/download/ we have:

- GNU Guix System ISO image -> x86_64 (no AArch64)
- GNU Guix System QCOW2 image -> x86_64 (no AArch64)
- GNU Guix 1.4.0 installer (called binary on that page) ->  x86_64 & aarch64

> > # Drawbacks and open issues
> >
> > There's no particular drawback to attempting regular release.  It should be
> > noted that the project is entirely dependent on volunteers so it may be that
> > contributors don't have enough time available to achieve regular releases.  If
> > that's the case we would revert back to irregular releases.
> >
> > There are various improvements that could be made to this process over time,
> > but no known issues.
> 
> I think someone already mentioned freezing master should be noted as a
> drawback. I would, in a weird way, second that, despite thinking it is
> actually a feature. :)
>

Yes, I will add some text hear about the freeze window. In my mind it's a necessary element if we want releases, there's no free lunch.

> > # Appendix 1: Release Project Time-line
> >
> > To illustrate the major steps of a release project this is a rough time-line.
> > The aim is for a 12 week active release project, with the first one using 16
> > weeks in total to give teams time to prepare.
> 
> Ambitious, but may as well aim high!
> 
(...)

Yes, and to be explicit there is a trade-off in what we can expect in _quality_ vs developer morale.

On the one side, asking a volunteer project to work on a release for a long time is likely to lead to a loss of motivation. I'm sure we can all think of release projects that have been "toil" and bad for morale. So, keeping it to a reasonably short, focused and high energy project seems the best way to have a positive result.

On the other side quality takes time. We're trying to release a Linux distribution that works on (a) a range of hardware (b) other distributions as a hosted package manager) (c) for different use-cases such as desktop and server, (c) provides delcarative services for configuring everything - this is a really large amount of work. 

So bearing that all in mind, having a reasonably tight project and trying to minimise the testing scenarios/areas of concern seems the right trade-off.

> > The current outline builds from our current [release document](https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/release.org)
> >
> > | Week of Project | Event |
> > | --- | --- |
> > | -5 | Nominate a release team |
> > | -4 | Notify teams of upcoming release |
> > | 01 | Release project start |
> > | 04 | Toolchain and transition freeze |
> > | 05 | Package set finalisation |
> > | 06 | Initial testing |
> > | 08 | Updates freeze |
> > | 08 | Breaking changes to staging |
> > | 08 | Ungraft master branch |
> > | 09 | Bugs and documentation focus |
> > | 09 | Branch and tag release branch |
> > | 09 | Testing and hard freeze |
> > | 10 | Release candidate |
> > | 12 | Final release |
> > | 13 | Staging merged to master |
> > | 14 | Release retrospective |
> > | 15+| Relax - it's done! |
> 
> The weeks listed here are -5 indexed, but are described as a 1 indexed
> bulleted list, it would be nice to keep them using the same indexing!

Ah, so it was actually a time-table showing the weeks of the project plan.

And the second one (below) is a numbered list of the steps. I will change it to remove the numbers, and I'll put the weeks in brackets.

>
> > ### 1. Nominate a release team
> (a.k.a. ### -5.)
> 
> > ### 4. Toolchain and transition freeze
> > No major changes to toolchains (e.g. gcc-toolchain, rust-1.xx) or runtimes
> > (e.g. java).  There should be no changes that will cause major transitions.
> ...
> > No alterations to the Guix daemon or modules are accepted after this point.
> > Packages and services in the 'minimal' package set should not be altered.
> >
> > ### 5. Package set finalisation
> > Specify the package sets for this release.  Identify all packages and their
> > inputs so that a full manifest for the release is created.
> 
> It seems like you would need to clarify the toolchain package set the
> week prior.
>

Yes, agreed it's wrong. I'll adjust it to a bit earlier. We have 5 weeks before the release starts to bike-shed the package sets.

> > ### 7. Updates freeze
> > Major package updates are frozen on 'master' as the focus is on fixing any
> > blocking packages.  Security updates still go to 'master'.
> 
> Yay! This should also give the build farms a chance to "catch up" with
> builds.
> 

Also AIUI we have world-rebuild with the ungraft. This is not my area of competence so I'm hoping Andreas, Ludo, Efraim and others will comment about this area if the discussion develops ... 

>
> > ### 8. Breaking changes to staging
> > To avoid a period of time where teams can't commit breaking changes, these are
> > sent to a new 'staging' branch, rather than directly to master.  The master
> > branch slows down from this week.
> ...
> > Team branches can still be folded into the release branch as long as changes are
> > minor package upgrades.
> 
> If they are minor package upgrades, do you even need a team branch?
> 

No, and generally the length of the freeze and where things go is something we need to debate. I want to avoid a big discussion about our team branches, but I think we all know there are concerns. 

> > ### 11. Branch and tag release branch
> > The master branch is tagged and a new release branch is created.
> >
> > * master branch: security only.
> > * release branch: security updates as normal. Only RC blocking bugs.
> >
> > Only security updates go to the master branch from week 9->13.  All other
> > changes stay in a team branch or go to the `staging` branch. The focus on the
> > release branch is to stabilise so only RC bugs should be pushed.
> 
> Spell out "RC" at least once in the document, presumably "Release
> Critical"?

Yes, Release Critical.

I had this in the Release team responsibilities: "communicate with the release team and wider developers status and blockers"

I'll finesse the wording.

> 
> > ### 13. Release candidate
> > Release artifacts are created for the release candidate.  There's a final 2
> > weeks of testing with these artifacts.
> 
> Not sure how you cram two weeks into week 13! (or... is that week 10?)
> :)
>

Heh! Using my terrible weeks it would be week 10 "release candidate", and then "final release" is week 12.

> > If there are no release blocking bugs discovered then the releas uses
>                                                          ^^ release ^^
> > these artifacts.  If bugs are found/fixed then release artifacts are
> > regenerated as needed.

Thanks!

>
> As a downstream packager of Guix for Debian, I would appreciate
> (pre)release candidates be included earlier in the process somehow.
> 
> Maybe even in sync with a weekly cadence in line with this release
> process?
> 
> I usually do find bugs in the process of packaging Guix for Debian, and
> it would be nice to more easily catch those earlier.
(...)

I don't know how complex it would be to do that, but it makes sense. In week 6 we do "initial testing" which means we have to generate the release artifacts, so assuming it's not too complicated we could then do a weekly cadence from there.

> > ### 14. Final release
> > Final release is announced and new release artifacts are published.
> 
> Yay!
> 
> 
> Thanks again, really glad to see some good thoughts and energy go into
> this!

Your welcome, look forward to follow-up comments on the topics above =-)

Steve / Futurile

[0] https://docs.fedoraproject.org/en-US/releases/
[1] https://wiki.ubuntu.com/SeedManagement
[2] https://xkcd.com/1172/
[3] https://guix.gnu.org/en/blog/2025/guix-user-and-contributor-survey-2024-the-results-part-2/