Subject: Re: GCD005: Regular and efficient releases

From: Steve George <[email-masked]>
Date: Sun, 22 Jun 2025 17:13:35 +0100

On Thu, Jun 12, 2025 at 09:08:54PM +0200, Simon Tournier wrote:
> Hi Steve,
> 
> Thanks for this proposal!
> 
> Well, I’ve not commented yet because I’m very doubtful that releasing
> more often can be solved with a GCD.  I mean, yeah we all would like
> more releases, and then?  :-)  Look, on February, we were all in the
> same room and we all agreed that we had to work on releasing soon and
> we cooked a plan to make it happen.  But the release has not come yet…

Understood.

You're not the only person in the group that thinks a GCD won't solve this.

> Do not take me wrong.  I do not want to be the pessimistic one cutting
> the positive circle trying to move forward.  As I wrote many times over
> the years, yeah for sure, releasing more often would be more than great!
> 
> Somehow, I think that “regular and efficient release“ is a good target
> but considering the current situation, I’m doubtful we will be able to
> go in only one go – just by deciding from the top and writing down a
> planning.
> 
> Instead, I think we need two intermediary steps:
> 
>  (1) Clarify what means being a member of a team.

I'd be happy to collaborate on something in this area. I genuinely don't really know what the responsibilities and accountabilities can be in this sort of set-up.

>  (2) Clarify the branching model; especially how to deal with grafts.
> 

I don't really understand this area enough to comment.

> To me, they are requirements before being able to have regular releases.
> 
> Therefore, I read this GCD as setting the goal in order to address the
> well known blockers as (1+2) and, only after, we will be able to have
> regular releases without burning out.
> 
> Moreover, I think it’s difficult to find the right balance between
> volunteering work and commitments.  Today, make a release is a piece of
> work.  For sure, it’s possible to smooth the things and automatize with
> various helpers etc. but if it’s not done yet, maybe it’s because it’s
> not trivial or maybe even more work than the ‘piece of work’ itself. :-)
(...)

Agreed, the difficulty of being a 'volunteer' project means we can't guarantee anything.

I hope this GCD at least (a) shows our goal/intent (b) provides some concrete ways to make 'doing the work' easier.

> Yes, I’m convinced that committing to have “regular releases” will
> improve all the releasing process over the next releases.  For sure!
> And it’s probably by making such commitment that the releasing process
> will be improved.  But we are not there yet, IMHO, so it’s still a piece
> of work and it’ll still remain one for some release cycles.
> 
> Therefore, the core of this proposal appears to me what is the bar?
> 
> Until now, the bar was high.  Maybe, one way to release more often is to
> decrease the bar so it’s easier to pass it and then we can rise it up,
> by incremental improvements.  This is the section about Package Sets,
> Tiers and Release Artifacts.
(...)

Agreed. Hopefully, adding clarity makes it easier for someone to join a Release Team and work on the release. 

> To me, that’s the key.
> 
> And concretely, who is ready to commit for 12 weeks (maybe 2 times) as
> Release Manager?
(...)

Efraim has agreed to be the first Release Manager.

> All that said, some comments hoping it might improve the bootstrap of
> “regular releases”. :-)
> 
> 
> > ## Package Sets
> 
> [...]
> 
> > The package sets would be:
> >
> > * minimal: packages required to boot and install a minimal Guix or Guix System
> > install.  This would include the guix daemon, kernels, boot loaders,
> > file-systems and minimal utilities.
> > * standard: packages that create a core installation for all other uses.
> > * desktop: provides the packages necessary for a desktop.  This would include
> > things like X11/Wayland, desktop environments, key applications and themes.
> > * server: provides packages and services for a server.  This would include
> > standard server packages and services.
> > * hosted: provides packages and services which are necessary in hosted usage.
> >
> > Guix would still make all packages and services part of a release (the entire
> > archive), but only those in the `package sets` could block a release due to a
> > significant bug.  The goal would be for this to be as small a set of packages
> > as reasonably possible.  It would mean that developers could focus on the
> > critical packages and services during a release.  As an example, this would
> > mean that a major issue in the Linux kernel could block a release, but not an
> > issue in a game.
> 
> I understand the meaning however it’s unclear to me.  Instead, I propose
> that the Release Team defines when they start their cycle which packages
> the release will support.
> 
> Somehow, each package set should be defined by a manifest file.  Here
> you picked 5 use-cases, so the Release Team lists the packages in the
> manifests minimal.scm, standard.scm, desktop.scm, server.scm and
> hosted.scm.  These manifests are considered as blockers.
> 
> If one package from one manifest fails and it’s hard to fix, then it can
> be excluded; i.e., moved from Tier to another Tier.
> 
> It also would help in monitoring the CI outside the release period.
(...)

I think they're a good idea, but I've managed to confuse the issue due to my own assumptions coming from my previous experiences. I've reworked the package sets to reflect what the current state of play is (Ludo and Efraim input on what is the current state)

I've also made the bar lower for testing by a Release Team, as there was a lot of feedback about it being too much.

>
> > ## Platforms and Architecture tiers
> 
> [...]
> 
> > - Primary architectures:
> >   - Architectures: x86_64, AArch64
> >   - Kernel: Linux
> >   - Coverage: all packages and services that are not explicitly platform
> >     specific must work to be included in the archive.
> >   - Package status: package updates should build for this architecture.  If a
> >     package update is broken it must not be pushed to users (e.g. master).
> >   - Security: all packages that are maintained upstream receive updates
> 
> This does not read as the rolling release, does it? :-)
> 
> I propose:
> 
>         - Tier 1:
>           - Architectures: x86_64, AArch64
>           - Kernel: Linux
>           - Coverage: defined by package sets.
> 
>         - Tier 2:
>           - Architectures: all others
>           - Kernel: Linux
>           - Coverage: defined by package sets.
> 
>         - Tier 3:
>           - Architectures: all
>           - Kernel: Linux, Hurd
>           - Coverage: all packages
> 
> For a release, Tier 1 must be all green; Tier 2 should be green; Tier 3
> as good as possible.
> 
> Obviously, we can add more Tier than only 3.  The main idea here is to
> define what is Tier (= list of architectures, kernel, package coverage)
> and what expects when this Tier in a release.
> 
> Well, maybe we could add a “service coverage”.
(...)

Yeah interesting, so my definition is specifying the circumstances under which something can be considerd a "Primary Architecture". It's basically trying to define how you would decide if something should be promoted. I think the reason I have it from this perspective is because if something is promoted, then it can prevent a release if any of the criteria fail. For example, if a package in a package set didn't build for a primary architecture, this would be an RC bug.

I've removed the 'Security' line from my definition, as I don't think it's useful being there given what's practically happening.

See if you think I've landed up where you are, but through slightly different words. I added a section to tie everything togehter:

-- >8 -- >8 -- 

1. Packages and services within the `package sets` must build on the `primary
   architectures`. Other packages are optional for a release.
2. Release artifacts are only made from packages within the `package sets` and
   on the `primary architectures`. Other architectures or release artifacts are
   optional for a release.
3. Testing and QA is on `package sets` on the `primary architectures` only.  The
   Release Team has flexibility to choose their response to bugs and issues.

-- >8 -- >8 -- >8

> > ## Release team and project
> 
> [...]
> 
> > The final release team is:
> >
> > - a new Release Manager
> > - a returning Release Manager
> > - up to 4 other members, one of whom acts as the Release Advocate
> 
> Hum, 2 times 12 weeks of commitment?  That’s something! :-)
> 
> For sure, back up the Release Manager with 2 people is a very good idea;
> especially about knowledge transfer.  Cool!
> 
> If the primary Release Manager is not able to keep the commitment (busy
> by life, away from keyboard, unexpected event, etc.), then the secondary
> Release Manager takes the floor.  Right?  But then, this “absent”
> primary Release Manager, does they become the next secondary Release
> Manager?
(...)

I guess in most circumstances people will say that they can't continue the commitment, rather than simply disappear. At that point the whole Release Team can discuss and hopefully someone can volunteer to step up, and the secondary one can take over. 

> Moreover, what does it happen if the both Release Manager are not
> responding?  Do we delay the time to find two new Release Managers?  Or
> is it two of the other members?
(...)

That's very much going to depend on the circumstance. At the moment my perception is that there's only a couple of people with the requisite knowledge who can drive a release. If everyone disappears then yeah we're in trouble!

Hopefully, over time - with more release experience and documentation - it would be easier for the group to recover and pick up the slack.

> The other question: what does it happen if the Release Team is not able
> to fix a blocker?  Do we delay until the fix?  Do we drop the blocker?
> Who decides?  What is the process for making such decision?
(...)

This one falls into the dificulty of a Release Team not being able to rely on individual experts in Teams (because they are volunteers). So, if the Release Team isn't an expert in something but it's on the critical path for a release (e.g. the installer doesn't work), then they have to either figure out a workaround or delay the release.

There's not the concept of a "landing zone" in the release plan template - it says if they can't do a release after three attempts then they should probably stop, but that it is ultimately up to them.

> > # Cost of Reverting
> >
> > If regular releases were not successful then the project would switch back to
> > irregular releases.  There would be no impact for exiting users as they will
> > be tracking the rolling release's master branch.
> >
> > If the project is able to successfully undertake regular releases then over
> > time it may be possible to undertake full releases every six months or some
> > other release cadence.
> 
> Well, I do not have the answer and my honest question is: Is it more
> detrimental to have irregular^W no release since years! than to
> communicate on regular releases and fail?
(...)

We are all volunteers doing something because it's interesting and fun. We should encourage everyone to trying new things, who knows - maybe we will achieve something!

If some of those things don't happen, then we just brush it off and try something else. The alternative is to be stuck and there's no possibility of change or learning at that point.

I don't really see how anyone can criticise us for trying things.

> > # Appendix 1: Release Project Time-line
> 
> [...]
> 
> > ### 1. Nominate a release team
> > Nominate a release team with two Release Managers (1 is the previous RM), and
> > up to 4 other people who will work on the release. Put out a call for a Release
> > Advocate who can be anyone in the Guix community who's willing.
> 
> Who nominates?  How?
> 
> Who decides if 7 people volunteer to be the next release team?  Hum, it
> will not happen, but still. :-)
> 
> The converse, what does it happen if no one volunteer?  IIUC, the
> primary Release Manager becomes the next secondary Release Manager, so
> what does it happen if there is no one who volunteers to be the next
> primary volunteer?
> 
> What does it happen if after the release, the primary Release Manager
> steps down and do not become the next secondary Release Manager?
(...)

In all these circumstances either someone steps up and give it a go, or they don't and we fall back to irregular releases. I don't think there's any way for me to give you the certainty you want here. 

(...)
> > ### 9. Ungraft master branch
> > Guix master is ungrafted to minimise the difference with users of the release
> > initial 'guix pull' experience.
> 
> This will take more than one week, I guess. :-)  For sure, more than few
> hours in the week.
> 
> I think this needs to be done by each team all over the changes
> independently of the release process.  As said above, I think we should
> clarify what a team does.  For instance, if a team introduces a graft,
> then this very same team must open a branch that ungrafts and then merge
> this branch as soon as possible; independently of the release.
> 
> Somehow, we removed core-updates but we have never defined a clear
> strategy for ungrafting.  Well, that’s outside the scope of this GCD
> but, IMHO, the success of this GCD depends on how to deal with grafts.
(...)

OK, have to admit a massive shortcoming for me is I have not taken part in any ungrafts so I don't really understand the overall difficulty since I haven't done it!

Thanks for all the details and time to comment!

Steve / Futurile