Forgejo monthly update - September 2024

The monthly report is meant to provide a good overview of what has changed in Forgejo in the past month. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.

Forgejo v9.0 release candidates

The first release candidates for Forgejo v9.0 was published 25 September and code.forgejo.org upgraded. A call for participation was published, asking for help with translations and testing.

A regression was discovered in the v9.0 release candidate that incorrectly deleted some of the images mirrored from the Docker hub. It disrupted the CI intermittently during 48h before a fix was published and deployed. The root cause was a bug in the cron task that cleanup dangling container images.

Forgejo Design process

Efforts on long-term improvements to Forgejo have been kept low in the past month, however there was a noteworthy exchange on moderation features in Forgejo. Initially, an idea was investigated to track reported content in a specific issue tracker, but the idea was discarded some time later due to the volume of spam issues on Codeberg, which probably requires a more efficient UI. The moderation features will not make it into Forgejo v9, but they will remain in the focus of design work.

Contributor documentation and test suite

By the end of August, a discussion emerged to improve Forgejo’s testing infrastructure, making it more friendly to new developers.

Since the last monthly update, multiple improvements have been made:

  • The contributing resources in the documentation have been cleaned and updated, with the goal of encouraging more diverse contributions, sending a warm welcome to new contributors and clarifying the motivation and instructions for writing tests. If you did not yet contribute to Forgejo, now is a good time to get started and provide us with feedback.
  • The in-repo hints for testing have been deduplicated and updated to make getting started easier. And numerous smaller improvements have been made to the end-to-end test suite that uses real browsers to ensure actions in the Forgejo UI work as expected. They added and improved examples.

Using the improved test infrastructure, the frequency of new browser tests has increased a lot compared to recent months.

Helm chart

A new major version, 9.0.0 was published. It sets proper namespaces and allows override.

The Forgejo helm chart had many minor and patch updates, in both v7 and v8. Helm chart v7.1.2 and v8.2.3 are the latest.

Localization

A significant effort was made to backport translations to the LTS version (Forgejo v7.0). There was hope for new languages added in this version to reach better completion in its primary lifetime, but it is taking longer than expected. This and other considerations led to a proposal to do less backporting to old stable to prevent breakage and save time.

A very large change was ported from Gitea and is good for internationalization. But it was not easy to also preserve the existing strings and it could have broken a few non-English strings in Forgejo v9. Some translators caught the problem and it was luckily fixed in time.

The overall translation activity was about twice lower than last month, which was very active.

Forgejo Actions

The security audit bootstrapped last month has its own repository to track the work done as transparently as possible. A suitable pentester was found and the scope of the audit was determined during a call. A rough plan was drafted and approved by NLnet who is funding the audit. The work should begin in November 2024.

Optimizing CI pipelines

In order to ensure a high software quality, Forgejo (like most larger project) runs CI/CD pipelines that perform a series of automated checks on the source code, ensuring that contribution meet certain quality standards.

Running these pipelines consumes significant amount of energy and adds to the climate footprint of free/libre software development.

In a quest to make Forgejo more frugal in the use of computing in the development lifecycle, optimizations to the CI/CD pipelines have been considered and a part of the work was merged.

The optimizations include caching the playwright environment in a test image and improvements to the caching of Go dependencies that improves on the caching available from the setup-go action that spent 10x2 minutes per job creating compressed archives. The action is also available to other projects and a dedicated contribution to the Forgejo Actions ecosystem.

Infrastructure

On 9 September code.forgejo.org was down during 10 hours. It was overwhelmed by excessive crawling and the response time was so slow that it kept accumulating a backlog and answering every request with a timeout. On top of that it happened late at night and although it was trivially fixed by restarting Forgejo, it only happened the next morning.

This was the first significant downtime and impacted a number of Forgejo instances that are using Forgejo Actions hosted on code.forgejo.org. A number of measures were taken to prevent that from happening:

  • Rate-limiting is imposed on the most aggressive crawlers.
  • Exclusion rules were defined and added to robots.txt.
  • Members of the devops team are notified on their mobile when monitoring detects a problem.
  • An ad-hoc script was written to detect excessive timeouts during extended periods of time and automatically restart Forgejo if needed.

The script is a hack that must not stay. It proved useful a couple of times while working on strategies to reduce crawling to manageable levels. It still represents over 50% of the incoming requests but they do not impact the instance performances.

The long term solution, as code.forgejo.org audience grows, is to improve its availability. The test instances at v*.next.forgejo.org are already using Forgejo helm, each in a dedicated k3s cluster. A long lived k8s cluster is being deployed to use the same Forgejo helm so code.forgejo.org can be migrated there. The goal is for built-in health monitoring to automatically react to an unhealthy Forgejo instance and restart it using idiomatic k8s methods instead of an ad-hoc script.

There is no urgency for the k8s cluster to replace the LXC based infrastructure. But it will take some time to improve code.forgejo.org availability in this way and the works started right away so that it has a chance to be ready before the next incident happens.

Sustainability

A procedure for receiving payment from Codeberg on Forgejo work was documented and discussed. The details of the funds received and spent in 2024 were updated.

The progress of the ongoing grant was updated. It was extended until the end of 2024 and got an informal agreement to increase the funding by 10K€. The legal status of the donations in Europe was documented with an example based on a Freelance established in Portugal, with links to do the same for other European countries.

The progress of the federation grant was also updated and a request for payment drafted for 2,500€. The grant will expire 1 December 2024 and the unspent funds will be returned to NLnet where they can be used by other projects.

Questions were received from NLnet on the latest grant application and an answer sent which led to followup questions. Because of the required delay in answering those questions, the grant application was moved by a few months, to the next call.

The sustainability team elected its first member. They helped with following up with the current grants and document the progress made so far.

The relevance of creating a non profit exclusively dedicated to managing the Forgejo funds and governed by the Forgejo decision making process was discussed. A balance should be found between the burden of managing a new organization and the benefit of being more flexible than Codeberg. The current situation is problematic as a significant amount of the funds obtained in the past two years (in excess of 60,000€) will have to be returned when the grants expire by the end of 2024.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.

A minority of Forgejo contributors earn a living by implementing the roadmap co-created by the Forgejo community, see the sustainability repository for the details.