Forgejo monthly update - September 2023

The lifecycle of Forgejo v1.20 is coming to an end and v1.21 is entering the release candidate phase this week. In addition to work on the codebase a lot also happened on the website, the documentation and the Forgejo runner.

Judging from the activity of Forgejo contributors, there is every reason to believe this new release will go smoothly. No-one is overworked or stressed, dependencies are up to date, features are added, the technical debt is kept in check and it looks like it could go on forever. But that should not hide the fact that there are many areas where progress could happen if only there were more contributors.

The past month was again dominated by the aftermath of the storage settings regressions but this unfortunate episode is, at last, concluded. It was a lesson for everyone involved on how to manage bugs that require action from the Forgejo admins because they cannot be fixed with a new release and an unattended upgrade. The recipe is simple enough but also quite time consuming: understand the problem, write tests to verify the conclusions, clearly explain what happened and provide detailed recovery recommendations.

Development

Fixing S3 configuration bugs and regressions

The storage configuration regressions fixed with the Forgejo v1.20.3-0 release were verified with newly introduced upgrade tests. They were focused on local storage and assumed S3 configuration was not subject to the same issues.

This assumption was not verified with any test and turned out to be wrong. A Forgejo instance setup to use garage instead of MinIO faced two simultaneous issues blocking the upgrade: the storage unexpectedly went from being in the filesystem to S3, and the S3 backend failed to initialize. The failure to initialize was a rather simple error in the settings, hidden behind a non human readable error message. With a configuration change, the Forgejo instance was upgraded successfully.

To help other Forgejo admins running into these bugs the storage documentation was updated with examples and references. The recommendations in the Forgejo v1.20.3-0 blog post were also extensively updated to address both S3 and local storage. They were verified with more upgrade tests that do the following for a variety of storage configurations:

  • start an S3 server
  • start a Forgejo instance at a given version
  • upload objects into all subsystems (avatars, packages, attachments, etc.)
  • verify they are found where they are supposed to be

The upgrade tests are run before each pull request is merged. They can also be extended to identify breaking changes between major Forgejo versions and verify the recommended actions to deal with them are accurate.

Changing the update time of issues via the API

The pull request to allow for setting the update times of issues and comments via the API was merged. It is one of the most fragile commits in Forgejo and was made significantly more robust with an extensive set of tests.

When upgrading Forgejo dependencies, Forgejo can be impacted in two ways:

  • the API or the codebase changed and Forgejo won’t compile
  • there is no conflict but the behavior changed in an incompatible way, and Forgejo tests will fail

The Forgejo codebase is organized in a set of about 100 commits (as of today) and heavily relies on tests during upgrades. It allows maintainers to focus on meaningful problems instead of spending their valuable time manually verifying, over and over, the same features keep working.

User blocking

When a user tries to transfer a repository to a user or organization that has blocked them, that transfer is denied. Pre-existing transfer requests are also denied when the user is blocked. Read more in the moderation section of the documentation.

See all moderation pull requests.

Publishing development versions

A few months ago it was proposed to publish Forgejo development versions on a weekly basis. This has happened in the past month and is what https://next.forgejo.org is running. The version number is something like vX.Y.Z-test, to clearly state it is not to be used for real.

Federation

The F3 Forgejo driver entered a new development phase. It goes like this:

  • forgejo-cli f3 mirror is run for upload or download on an existing repository
  • a bug shows and is fixed either in:
  • repeat

It is still experimental.

See all F3 pull requests.

CURL based Forgejo client

forgejo-curl.sh is a new thin curl wrapper that helps with Forgejo authentication. Beyond that it does not provide anything. It is low maintenance because it only relies on the authentication logic and does not need updating when the REST API (or the web UI endpoints) change.

Website and documentation

Astro was upgraded to version 3, and further improvements were made to the file layout to further separate content from code.

A number of small improvements were made, including a switch to system fonts to improve performance and fix a layout issue which sometimes caused scroll anchors to misbehave.

The documentation content was moved to its own dedicated repository to ease contribution and separate the content from the website code.

Detailed instructions are now provided for working locally on the documentation. Tooling is available to preview the results before sending a PR, and to fix linting errors. A Git hook helps to ensure badly-formatted content is not committed, and the Forgejo Actions CI helps to apply checks to the content before PRs are merged, as well as helping to backport changes to older versions of the docs where necessary.

Creating previews for documentation PRs without exposing secrets is not a trivial problem. Some CI have a setting to take the risk. But Forgejo Actions does not work that way and that requires a different strategy, similar to what is used when publishing Forgejo releases in order to protect the release signing key. It depends on a feature that will only be available in v1.21 and it needs more manual work in the meantime.

Forgejo Actions

The actions supporting the Forgejo runner release process are grouped into a new repository, forgejo-build-publish. The build phase and the publishing phase. They are not new actions, they were copy/pasted from the Forgejo main repository, generalized to also be usable for the runner and verified with integration tests.

The new version of the Forgejo runner that came out of this new release process has binaries named differently than before and unified with the Forgejo binary naming scheme.

The container image already existed but was not supported or thought through. It is now tested and documented with a standalone docker-compose example that is verified to work.

The container image only contains the runner binary and does not run as root. The docker-compose example shows there is no need for anything else, even when using docker:dind.

Infrastructure

The infrastructure as well as the runner backends rely on LXC system containers and use lxc-helpers.sh to implement patterns common to Forgejo. Among other things, it sets the permissions of the container to run docker, nested LXC or libvirt but lacked flexibility to:

  • Add more permissions to run a kubernetes cluster
  • Restrict permissions for better isolation

The new --config option provides a range of pre-defined permissions to remedy that problem. It includes k8s which is tested to work when installing k3s.

Governance and communication

Contributors team

The Forgejo “contributors” team was created informally and liberally to grant permissions to label issue, manage CIs and pull requests etc. To make it official it was formally proposed in the governance repository to be decided according to the decision making process.

Debconf23

There were discussions about Forgejo at Debconf23 and a contributor to FreedomBox was interested in packaging Forgejo for Debian so that it can be distributed with FreedomBox. They could join the forgejo-deb which already provides functional Debian GNU/Linux packages.

Sustainability

A new discussion started on how to absorb the workload from the Forgejo issue tracker. There is no conclusion or action planned and the problem unfortunately remains. However, Codeberg independently sent a call for help to get help with handling their scaling issues and it will hopefully attract more contributors to Forgejo.

The second payment of the NLnet grant was received. In total 40% of the workplan approved for the grant in June 2023 was implemented.

Moderation

Earlier this year ad-hominem attacks were published in the Forgejo spaces. This goes against the Forgejo Code of Conduct and some of these messages were redacted. The author repeatedly refused to acknowledge this was not an acceptable behavior in Forgejo spaces and recently sent threats to publish more ad-hominem attacks. Read more in the moderation report.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.

A minority of Forgejo contributors earn a living by implementing the roadmap co-created by the Forgejo community, see the sustainability repository for the details.