Forgejo monthly update - January 2023

Forgejo 1.18.0-1, the first release, was published 29 December 2022, followed by patch releases which included security fixes. There were no reports of failed installations or upgrades. The 15 December 2022 announcement made a promise and a lot of work went into fulfilling it. A release is more than uploading a file and hoping for the best, users deserve reliable distribution channels and it requires dedicated people and proper tooling. Forgejo proved to be a product that users can rely on for their day to day operations, from large instances such as Codeberg to personal ones running on low-end hardware.

What goes into each Forgejo release is a snapshot taken from an constantly evolving patchwork of Free Software components carefully assembled together. It is based on Gitea, which is itself built on one hundred packages, and when digging deeper it amounts to almost a thousand. But wait, there’s more: container images include git, ssh and rely on Docker. The CI runs on Woodpecker with LXC and Debian GNU/Linux. This is why adding a feature or fixing a bug in Forgejo often means contributing to one of its many dependencies. This is the healthy way to grow the global Free Software ecosystem. For instance, an LXC backend was contributed to Woodpecker to improve the Forgejo release process. If a proprietary CI was used, such a contribution would not be possible and progress would be blocked. Another example is the accessibility improvements that were contributed to Gitea.

The Forgejo community is made of people who are citizens of the larger Free Software ecosystem. They may identify themselves as members of the Forgejo community because they like that it furthers the interest of the general public. Or because they enjoy participating in a democratically led community. Or when they write a piece of code somewhere in the thousands of Forgejo dependencies. It is not an elite community, it is not an affinity group; it is an inclusive and fluid group of people who share values and visions. It is engaged in work to define its governance, make sensible decisions and preserve a safe space in accordance to its Code of Conduct.

Forgejo 1.18

From a technical point of view, Forgejo is a drop-in replacement for Gitea: it can be used without any modification by simply replacing the Gitea binary (or container image). On 29 December 2022 Forgejo v1.18.0-0 was released and had just a little glitch: the version number started with a w instead of a v. The same day Forgejo v1.18.0-1 was published to fix that mistake.

This typo was the tip of the iceberg: the release process had to be redesigned. Back then, it took more than one hour for the Forgejo release process to complete. Furthermore it could be interrupted when a release was partially published, throwing the rest away. Unfortunately, that happened while releasing v1.18.0-0 and a clever workaround to resume with a version starting with a w created an artifact. That could have been problematic, or at the very least raise an eyebrow from Forgejo admins.

In the following weeks, a two-phase process was created wherein the releases are first built in the Forgejo integration organization and published there as drafts, to be thrown away if anything goes wrong. This is the most time-consuming step and also the one that is most likely to fail because something was overlooked by the Forgejo developers or because of an unexpected environmental failure. The second phase is copying the release to its final destination; although it is not atomic, the window of opportunity for a half published release is significantly smaller.

Security releases

The security team was quickly put to the test and had to prepare two Forgejo releases (Forgejo v1.18.1-0, 18 January and Forgejo v1.18.2-1, 22 January). The problems were explained in plain English for Forgejo admins to quickly understand their impact and assess the urgency of an upgrade. Handling security issues responsibly is a heavy burden and while the process in place proved to work effectively on those two occasions, it will require a continuous effort for as long as Forgejo exists.

Although it did not put Forgejo users at risk, there were problems during the making of these security releases, as can be expected for any newly formed team. The expectations were not clearly laid out, the security@forgejo.org mail was a redirection that revealed the recipients under some circumstances, unencrypted messages were exchanged, security issues were not always reported responsibly and embargo requests were overlooked. Although it worked out fine in the end and all these issues were properly addressed, it speaks to the challenges of bootstrapping a security team.

Accessibility

Accessibility in Forgejo needs a lot of work and it was difficult to figure out where to start. The accessibility team works in multiple issue trackers (meta, forgejo, user research, etc). The discussions started in a dedicated Matrix room, which turned out to not be very accessible, so a Fediverse hashtag #ForgejoAccessibility was created as an alternative to experiment with.

Despite the shortcomings of the current Forgejo UI framework, concrete changes were made to the codebase to get things moving. The challenge is to guard them against regressions because there are no automated tests. Since the community impacted is small, the chance that accessibility improvements are broken is high.

Federation

Federation development has been progressing slowly. Most of the time was spent on refactoring the current code, working on the ForgeFed vocabulary and the F3 schemas.

Forgejo CI

Since it started out on Woodpecker instances provided by individuals due to performance issues with the database integration tests on Codeberg-CI, the work on centralizing / unifying the CI has been ongoing. Codeberg solved part of the problem by dedicating hardware resources to Forgejo: using a RAM disk to workaround the slow HDD performances, it runs fast without thrashing short-lived SSDs, reducing ecologically problematic ewaste. The Forgejo devOps team weren’t used to working with LXC containers (which is what Codeberg is built upon), so there was (and still is) a learning curve to work on.

To allow for better safeguarding of the CI against malicious PRs and to better protect the secrets (signing keys, upload credentials) used to actually release Forgejo, forgejo-ci.codeberg.org has no secrets at all. This was also necessary due to some hardware limitations: there is not enough disk space to build a Forgejo release. Another Woodpecker CI instance is used instead and its access is restricted to better protect the release signing keys.

End-to-end testing

The key to a robust release process is the ability to run tests verifying that it works as expected. Not just unit tests that are routinely run by the Forgejo CI or integration tests included in the codebase. End-to-end tests deploy the release itself and are required to run scenarios that resemble what users would do. The alternative is to ask human beings to run these tests manually, but it is error prone and gets boring very quickly.

The Forgejo release is published in the Forgejo experimental organization and then used to run tests that do something along these lines:

  • Deploy Woodpecker
  • Deploy Forgejo
  • Create a user and repository in Forgejo
  • Submit a job to Woodpecker by pushing to the repository
  • Verify the outcome of the job is as expected

Because Forgejo is small, it takes around two minutes for this test to complete. It is now run against every release, and while verification by human beings is still needed, they can focus on what matters instead of trivial mistakes.

Contributions to Woodpecker

Running end-to-end tests is technically challenging, there’s a reason it is still not common practice. Although it would be possible to use Docker, it’s a stretch (see this example). The ideal situation is when the test can begin on a freshly provisioned operating system that will be disposed of when the test completes, cleaning up all the left overs even in the event of a catastrophic failure.

An LXC backend was developed for Woodpecker to provide such an environment. Woodpecker container images are published for internal use by Forgejo, but are not supported.

It is common practice to use a patched version of some software while waiting for pull requests to be accepted. The risk is to forget to do the work to get the changes accepted upstream, accumulating the associated technical debt. The patched version of Woodpecker is published in a dedicated repository as a reminder and each commit is labelled with the corresponding upstream pull request.

Forgejo contrib

Over the past few months, people who rely on Forgejo came to expect a certain quality of work as well as long term maintenance. When someone has a great idea and lots of energy, they may not be willing to match those expectations right away. Sharing a clever hack, bootstrapping documentation, creating a list of resources related to Forgejo and temporarily distributing Woodpecker container images are but a few examples.

The Forgejo contrib organization was created as a space where projects related to Forgejo can exist without being subject to high expectations. It is fine to create a repository that will be abandoned a few weeks later. It is fine to engage in a project that is not agreed upon by other Forgejo community members. The only requirement (and it’s a serious one) is to behave in a way that does not go against the Code of Conduct.

A curated list of delightful Forgejo-related projects and resources was created there and quickly became a useful reference. If you’re working on a Forgejo-related project, feel free to submit it for inclusion on the list.

Well-Being & Code of Conduct

Forgejo created a Well-Being team and adopted a Code of Conduct from day one because tensions are bound to happen in such an inclusive environment. And with over three hundred people in the chat room, it happened a few times in the past weeks. When someone uses coarse language and engages in a heated discussion, it has a chilling effect and silently drives people away. To help prevent that, the Well-Being team does its best to intervene in a gentle way before it escalates to a point where the Code of Conduct would have to be enforced. There were only two instances where a reminder of the Code of Conduct had to be mentioned: there is zero tolerance for calling another community member names or making derogatory comments towards an ethnic community.

Obviously it would be better if no such comments had been made at all, but it is hoped that the actions taken will alleviate any fears that any intolerance would be permitted. Hopefully any affected individuals will be encouraged to continue to participate in discussions as endeavors are made to ensure this is a safe space for all community members.

Governance

Decision process

As expected, the process to define the governance is taking its time. The need for a decision-making progress emerged during the last two meetings on 23 December and 27 January (part 1 and part 2) and was partially formalized. The general idea is that a decision leads to an agreement that is documented so that Forgejo community members can conveniently refer to it when needed. The list of agreements has just one item at the moment related to branding (name, logo, etc).

The first decisions

Meanwhile, this ongoing governance work inspired community members to move forward, and a few discussions that may lead to decisions are ongoing. Topics include very concrete aspects of development such as the criteria in order to define how a PR should be approved which was followed by a pull request to document the agreement. Additionally, there are very high level and much longer discussions such as licensing Forgejo under copyleft. It’s still blurry but something is emerging. It boils down to:

  1. Discussing
  2. Proposing an agreement
  3. Documenting the agreement

Agreements for team members

As time passes, the need to replace the teams that have been in place since Forgejo started becomes more pressing. For instance, given how critical their work is, members of the security team must be agreed upon by the community. It was suggested during the last governance meeting that the procedure goes as follows:

  • Discussion: people willing to participate in the security team send a formal and public application (see this application for the release team for an example)
  • Proposing an agreement: if the discussion about the applicant seems to meet the approval of the Forgejo community, a pull request to modify the security team is proposed
  • Documenting the agreement: if the pull request is approved, it is merged and the agreement is published in the meta repository.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.

A minority of Forgejo contributors earn a living by implementing the roadmap co-created by the Forgejo community, see the sustainability repository for the details.