Forgejo monthly update - December 2023

  • Codeberg migrated to Forgejo v1.21 and suffered a downtime related to performance issues. Although it was quickly recovered, it shows work is still needed to improve scaling and stability.
  • A new forge comparison page is available. Gitea turned Open Core this month and it articulates why Forgejo is a safe heaven for admins who want to escape this trap.
  • The Forgejo Helm Chart reached general availability with version 1.0.0.
  • Forgejo end-to-end testing can now be triggered from Forgejo pull requests by setting a label. For instance, they were used to verify a bug fix in how scheduled actions work by automatically running an actual Forgejo instance and a runner.
  • Two new machines were added to the Forgejo infrastructure to service https://code.forgejo.org, an instance dedicated to Forgejo development.

As Forgejo grows, the format of this monthly report changed to distribute the workload among all Forgejo contributors. The summary is replaced with a bullet list of the highlights and the sections are written by the Forgejo contributors who have been active on a given subject. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.

Forgejo

https://codeberg.org/forgejo/forgejo

Notable improvements or bug fixes:

  • Allow changing the email address before activation: In case someone accidentally registered with the wrong email address (made a typo, for example), with this PR, they will be able to change the email address, and request a new activation mail to be sent. This requires logging in, which is possible, even while the account isn’t activated yet. Previously, this required help from an instance administrator, now it can be self-serviced.
  • Allow viewing the latest Action on the web: a tiny little convenience route that allows linking to the latest action of a repository. Useful for READMEs and CI badges.

In flight pull requests

Most pull requests are opened and closed within a week. But some of them take a longer time, either because they are more complex or because they are taken care of by volunteers who can only occasionally work on them in their free time. This is a list of those that were updated since the last monthly report. If they are of interest to you, reviewing the changes or providing solutions would be appreciated.

Federation

The pull request to implement federated stars can now be tested manually and an activity summary was published.

The F3 refactor is making daily progress.

The federation implementation task list was updated.

Releases

Forgejo maintains two stable releases at any given time:

  • Stable (latest major version): receives full support, bugfixes and security fixes.
  • Old Stable (previous major version): receives only critical security support.

There has been one release a week in the past month, an unusually high frequency explained by a series of security vulnerabilities. They are labeled with a different color Stable and Oldstable to make it easier for the Forgejo admin to figure out which one applies to their instance.

The CVE-2023-49946 was created a week after the publication of the Forgejo v1.20.5-1 release that fixes this critical vulnerability. As a reminder the Forgejo v1.21 stable release already includes the associated security fixes and was never vulnerable.

Forgejo admins are encouraged to subscribe to security announcement so they can better plan their upgrades.

References

End-to-end testing

Forgejo end-to-end tests require running an actual instance and were moved to a dedicated repository which requires a significant number of manual steps to run them on a given Forgejo pull request. It was made simpler by triggering them with the run-end-to-end-tests label. For instance, in this Forgejo pull request setting the label triggered this workflow in the end-to-end repository, using a binary created with the pull request.

References:

Forgejo helm

Forgejo Helm Chart reached GA version 1.0.0 and is basically HA ready thanks to upstream Gitea Chart.

The only remaining issue is that all cron jobs are run on all instances because there’s no leader elections yet.

References:

Codeberg

Codeberg migrated to Forgejo v1.21 and despite preliminary testing and a curated list of potential regressions, a performance issue prevented it from running. It was quickly diagnosed and fixed but the root problem remains: there is a lack of performance tests in Forgejo. That could be mitigated if there were other Forgejo (or Gitea) instances running at the same scale (~100,000 users and projects, publicly available) as they would also run into the same problems. But in the past years the vast majority of performance issues were discovered in the context of Codeberg, which strongly suggests it is the largest instance in existence.

A priority list has been established so Forgejo contributors can quickly figure out which problems deserve their attention.

References:

Forgejo runner

Forgejo runner 3.3.0 was published and is now IPv6 capable. Unfortunately there is a bug in Debian LXC support that limits the scope of tests it can support.

References:

Governance

Mergers team

The Contributors team has no permissions on “Code” which prevents members from merging pull requests although they have write permissions on “PullRequests” for the Forgejo repository. There is not much scrutiny to enter the contributors team because it bears almost no risk to the integrity of the Forgejo project. Being given permission to write the repository is a different matter. The new “Mergers” team is created and requires a formal application process. However light, it will make a difference that matters in terms of trust and commitment.

Moderation

Two moderation actions were carried out: one regarding an ad-hominem attack and another regarding the publication of private information.

References:

Professional services

Forgejo, just like any other Free Software can be the base of commercial activity from service providers independent from the project itself. Forgejo exists under the umbrella of the Codeberg e.V. non profit which does not provide professional services so there is no competition and will never be.

Even if to address a handful of requests per year, it is convenient to have a place where people in need of professional services can meet service providers with the will and the skills to meet their needs. Here are some similar places in other Free Software projects:

It was decided to create a kind of job board which was bootstrapped as a dedicated issue tracker.

References:

Hardware infrastructure

Two new machines were added to the Forgejo infrastructure and are now hosting code.forgejo.org that was previously in an OpenStack virtual machine. It is faster (10 times more bandwidth) and also is IPv6 capable. The LXC containers are setup and maintained using lxc-helpers.

A LXC container with extended capabilities required to run a nested k8s cluster was setup on an dedicated hardware for better isolation. Forgejo runners are installed for both lxc-helpers and forgejo-helm so they can run workflows that depend on k8s. For instance, it helped detect regressions with the Forgejo Helm chart prior to version 1.0.0.

The next.forgejo.org instance that is running the development branch of Forgejo (updated weekly) is now also hosted on this hardware. It was reset entirely on that occasion and displays a prominent reminder that there is no guarantee that any data will persist. It is only for experimental purposes.

References:

OCI mirrors

Container images hosted at https://hub.docker.com/ are subject to rate limiting which can be disrupting, for instance when the CI is used intensively by multiple contributors.

Some of the most commonly used container images used by Forgejo were manually mirrored in a dedicated organization to partially resolve that problem. Updating these images should be done automatically but there still is no script to do so.

References:

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.

A minority of Forgejo contributors earn a living by implementing the roadmap co-created by the Forgejo community, see the sustainability repository for the details.