Forgejo monthly report - July 2025

The monthly report is meant to provide a good overview of what has changed in Forgejo in the past month. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.

Releases

Forgejo v12

Forgejo v12.0.0 was published on July 17 as scheduled. The release also marks the end of support for the LTS release Forgejo v7. We recommend that everyone upgrades to at least the new LTS (Forgejo v11).

A regression involving Minio was identified and resolved less than 24 hours before the release. Another regression was found and fixed after the release. Workarounds were added to the release notes, and it was decided to not publish an immediate patch release.

A week later, Forgejo v12.0.1 was published, which reverts a breaking change that should have been prepared more carefully. An effort has begun that will hopefully allow us to reintroduce it.

Helm chart

A bug fix release was made for v7, which corresponds to Forgejo v7, which is now end of life (EOL). Similarly, a bug fix release was made for v12, which corresponds to Forgejo v11.

v13 and a patch version were released with the corresponding Forgejo v12 versions. v13 represents a significant change because Redis was replaced with Valkey. Furthermore, the PostgreSQL HA subchart underwent a significant update and requires manual migration if used.

Additionally, there is a plan to migrate away from Bitnami for the included PostgreSQL setup because the images are no longer public.

Forgejo runner

Forgejo runner v6.4.0 was published, but unfortunately, it contained a serious regression. After it was fixed, Forgejo runner v7.0.0 was released, and the previous version was no longer recommended.

A few weeks later, the longest-standing security bug of the Runner was fixed (namely, redacting multiline secrets), which was released in Forgejo runner v8.0.0. This version also validates actions and workflows against a schema. Although it is a significant breaking change, it is not optional because:

  • A workflow or action with random content may lead to unexpected behavior that is difficult to understand and debug.
  • When something works by chance but is neither documented nor supported, users will depend on it, and it will have to be supported or migrated in a backward-compatible way. This can be very difficult once the Runner is stable.
  • Errors that are only detected at runtime due to incorrect behavior can be difficult to relate to the root cause, especially when the error is syntactic. Syntactic errors displayed in the Forgejo UI or in the output of a failed job are much easier to identify.

However, some validations were too strict in v8.0.0, so they were relaxed in v8.0.1.

Tooling

The release process is not fully automated yet, but the release manager intends to automate it eventually. The checklist of manual tasks has been clarified. With a major release every three months, ambiguities have time to surface.

The release notes assistant, which collects and compiles release notes, was the main pain point for this release because compiling the notes took about two hours. Created a year ago, the assistant focused on correctness rather than efficiency, and it became problematic, holding up the release for hours for a process that should take only a few minutes. The process was speed up by 20 times and now completes in under two minutes. See also the rationale for not using the most commonly known alternative.

The toolchain used to build Forgejo and Forgejo runner relies on the same action, which builds the binary in a container for each supported architecture. This ensures that the same binary is used both standalone and in the container. It is specialized for Forgejo and unlikely to be used for any other purpose. However, it relies on the forgejo-release action, which has received a few contributions over the past year. Apparently, it is used by other projects, although it was first designed to be specific to Forgejo. It gained support for efficiently using the release-notes-assistant. Forgejo Runner v8.0.1 started using it, meaning changes from pull requests won’t need to be managed manually. Only breaking changes and how to deal with them will need to be explained.

Discussions

AI usage in the context of Forgejo

At the end of June, a discussion began in the Forgejo space about how to handle AI contributions. There had already been informal discussions on the topic beforehand.

The discussion focuses on how to handle AI contributions and attribute the use of AI. Several proposals have emerged from the discussion and are currently being voted on. The plan is to officially adopt the proposals afterwards.

Once a decision has been made, it will be announced in either a subsequent monthly report or a separate blog article.

Attribution of contributors in the docs

At the beginning of the month, a discussion began about whether and how authors of the documentation should be credited. The idea is to motivate contributors and show appreciation for their work. The discussion is still in its early stages but has been dormant for the past few weeks.

Communication in the Forgejo space

A conversation about Matrix began in the UI development space, particularly due to the introduction of premium accounts. A discussion was started to document future conversations. This discussion covers what future communication in the Forgejo space should look like.

Other FLOSS communities have experienced similar issues with Matrix and its evolution. Currently, there is a proposal to send an open letter to Matrix encouraging them to return to their core values.

Forgejo runner

Security audit

In August 2024, a security audit was planned for the Forgejo runner. The goal of the audit was to advance the Runner from the Alpha stage of development to the Beta stage and beyond.

Forgejo requested the audit from Radically Open Security through the NLnet grant. ROS helped draft a plan for the audit, which was accepted in September. Actual testing began in November.

Throughout the testing process, we received updates on any findings so that we could develop fixes immediately. These updates resulted in a series of security-focused changes to the Forgejo runner.

The following issues were fixed:

  • CLN-007: Unrestricted container options (found by ROS).
    It was possible to perform privilege escalation by specifying certain container options. This was resolved by filtering the container options.
  • CLN-002: Runner containers had access to the Docker socket (found by ROS).
    The Docker socket was mounted into job containers by default. This allowed workflows to perform privilege escalation. This default has been changed.
  • Cache access was not authenticated (programmerjake).
    Previously, access to the caches was not authenticated. This meant that workflows for one repository could receive caches from other repositories by guessing the correct cache key. This issue has been resolved by requiring authentication to access the caches and by adding a proxy to transparently perform this authentication.
  • Multiline secrets were not correctly redacted.
    Secrets with multiple lines are now redacted properly when printed to the logs.

The complete details about the findings and non-findings of the audit can be found in the code audit report.

As stated above, the last fix was included in v8.0.0, therefore it is highly recommended that you upgrade your runners to v8.0.0 or newer.

No more GITHUB in variable names

Starting with version 7.0.0, the runner allows you to create new workflows without using GITHUB in the names of contexts or variables. This change is mostly cosmetic, which is why it took years to implement. Previous names that include GitHub are preserved for backward compatibility and to facilitate reusing actions originally developed exclusively for GitHub. Shortly after this update was deployed to the Forgejo infrastructure, a series of pull requests were merged to take advantage of the more aesthetically pleasing names.

Reorganization of bugs and features

All issues and pull requests for ACT and the Runner were triaged and sorted. Bug reports are now grouped in the Forgejo runner issue tracker, and feature requests have been moved to a dedicated space. A new workflow was also discussed and implemented on that occasion. In short, it strikes a balance between the expectations of someone filing a new feature request and the need for firsthand testimonies to guide implementation.

To avoid reinventing the wheel, an exhaustive inventory of commits in the Nektos ACT and act_runner repositories was taken and compared to the Forgejo runner implementation. This process covered the last year, during which there was little activity, so it did not take long, and the discovered commits will save valuable development time.

User research

Efforts were made in design and user research (1 and 2) to determine how Forgejo Actions CI results should be handled. This will help determine the next steps now that Forgejo v12.0 has the building blocks to send email notifications when a workflow fails.

Federation

For a high-level overview, check out the federation roadmap.

This month, blockers related to test timeouts (1, 2) that affected development were removed. This now enables further progress on federated user activities.

Sustainability

This month marks the end of the 2022-12-01 grant from NLnet.

The funding enabled some changes to be made to the Forgejo codebase. This was done in exchange for financial compensation.

The completed tasks were:

  • 2023:
    • C2: Improvements to the Forgejo Runner and automated setup of Forgejo instances using it.
    • A1: Tools to produce the binary Forgejo distribution.
    • A2: Tools to produce OCI container Forgejo images.
  • 2024:
    • B1: Trigger the release pipeline on a Forgejo instance with a tag pushed by the release manager.
    • C1: Improvements to the LXC helper, which provides low-level system containers for use in CI.
    • C3: Add support for an LXC backend to ACT.
    • B3: Improvements to the webhook implementation and support for https://builds.sr.ht/.
    • D3: Improved accessibility when JavaScript is disabled or unavailable for other reasons.
    • D4: Fixed accessibility issues identified by an audit.

Funding was extended, enabling the completion of additional tasks in 2025:

  • F: Moderation - Support for reporting users to the instance admin.
  • E3: Improved documentation for Forgejo Actions, including splitting it into several easier-to-read pages.
  • E2: Improved navigation in the documentation (version selector, sidebar).

Further information can be found in the corresponding documentation.

We would like to thank NLnet for supporting these tasks.

Infrastructure

Disaster recover exercise

At the end of 2024, the Forgejo infrastructure migrated to a K8s cluster that is self-hosted on three bare metal machines running only free software in Germany and Finland. The cluster is designed to recover from a data center going down in either country without data loss. This was tested when the cluster was first set up. However, it is beneficial to practice the disaster recovery scenario periodically, which occurred on July 24, 2025.

The detailed task list was published for the record. In short, everything went as smoothly as could be expected, except for one detail: NFS-backed Forgejo instances can be extremely slow when going back and forth between Germany and Finland (a Git clone can go from 15 seconds to 5 minutes).

Network outage

Coincidentally, on July 21, the day the disaster recovery exercise was announced, a network outage affected the Forgejo Action runners servicing Codeberg between midnight and 4 AM UTC. While the machines hosted at Hetzner in Finland could no longer reach Codeberg, those hosted in Germany using the same exit point (core5.fra.hetzner.com) experienced no issues. We submitted a support request to Hetzner less than one hour after the outage began, but it did not clarify the problem, which apparently fixed itself. Alternatively, the issue may have been resolved without any information being provided.

With that exception, the Forgejo infrastructure required little to no attention from the DevOps team. This was a welcome change compared to the work required to mitigate and understand the excessive crawling of the previous months.

Hardware costs

The cost of the Forgejo hardware infrastructure was broken down, and the rationale behind it was explained. It has been informally paid for by individuals so far, and a more sustainable model was discussed. It was proposed that Codeberg cover the costs, and the necessary administrative steps were taken. The following months will reveal whether this transition was successful.

Other

Matrix security disclosure

On 16 July 2025, Matrix announced in a predisclosure that a security vulnerability had been found in the current room versions and that it had been fixed. According to Matrix, this was only possible by introducing a new room version.

For us, this means that all rooms on Matrix must be updated. It also means that we have to migrate the Forgejo chat room again. In this case, there will be no manual migration and the update will be performed as direct update of the room version. This means that all clients that support room updates will automatically join the new room.

Nevertheless, there will be a message in the room shortly before the update so that everyone whose clients do not support updates is informed. The distributed rollout of the security fix is planned for 11 August 2025. The plan is to update the Forgejo rooms shortly after the release.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is intended to reflect this diversity and to acknowledge all the contributions made over the past month. If you are missing, please ask for an update.