Forgejo monthly update - February 2025

The monthly report is meant to provide a good overview of what has changed in Forgejo in the past month. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.

Releases

Forgejo Security Releases v10.0.1 and v7.0.13 were published 8 February 2025 mainly for fixing permissions enforcement of Forgejo Actions and projects; few other bug fixes were also included.

The Forgejo Helm chart had 4 bugfix releases, 2 for each of both v11 and v7.

FOSDEM

As briefly mentioned in the last monthly update, some of the Forgejo developers have been to FOSDEM and met at the Codeberg+Forgejo stand. They met a lot of happy users, discussed ideas and code architecture and finally managed to meet each other in person.

There have been meetings with other projects that are considering to move to Forgejo or became curious about the project. There has been more in-depths meeting with the people at Fedora and talks about future collaboration. Two Forgejo developers have been invited to dinner by Fedora, thank you for that!

You can find a discussion with more impressions from FOSDEM.

Moving to Forgejo

During FOSDEM, we have learned about many projects that are considering or planning a migration to Forgejo, but are blocked due to various aspects. These range from missing features to the need of help setting up or planning the migration. To help keep in touch with interested projects, coordinating migration assistance and tracking necessary additions to Forgejo, a dedicated repository for migrations to Forgejo has been set up.

If you have a success story to share about your usage of Forgejo, or if you have projects that you’d like to see on Forgejo but that are not yet decided or blocked for some reason, feel free to submit to the tracker and let us know.

Forgejo for Enterprise

There is high interest in using a simple and lightweight Git forge for commercial projects. During FOSDEM, we have been surprised to learn about quite some big companies (“off the recording”) using Forgejo. However, many potential users are hesitating due to the perceived difficulty in finding commercial support for the software.

Forgejo’s ecosystem has several advantages to enterprise customers. Contrary to competing software such as the fully proprietary GitHub, the open core GitLab and Gitea, there is no vendor lock in for Forgejo since is not under the control of a single organization. There is an inventory of professional service providers that slowly evolves to an ecosystem where Forgejo users can freely choose from depending on their needs and satisfaction.

However, these advantages are not yet clearly used to attract customers, and open questions about the influence of business interest have to be discussed. A discussion issue about Forgejo for enterprise has been created for this purpose, tracking the status quo and potential improvements.

If you are using Forgejo for commercial purpose, it might help us a lot if you (ask if you can) publicly write about your case as a success story. Let us know in the issue.

DDoS on code.forgejo.org

On 9 February, code.forgejo.org got hit by a DDoS. To this day, it is still unclear if it was malicious or an excessive number of requests from a crawler.

When code.forgejo.org timedout for the first time on that day, it was thought to be excessive crawling to be blocked, like a number of others in the past months. And indeed, such a crawler was quickly discovered by looking at the logs. It was not obeying robots.txt and did not identify itself with a proper User-Agent header. A small range of IP addresses was added to the existing block list.

When code.forgejo.org timedout again that same day, it was suspected that some IPs of the range were missed. However, after analyzing the logs, it appeared that the large number of requests did not originate from a small number of IP addresses: they were coming from thousands of them.

The k8s cluster hosting code.forgejo.org was setup at the end of 2024 and lacked good visibility on the incoming requests, which turned out to be a problem in this situation. While trying to figure this out, the DDoS slowed down code.forgejo.org so much that it stopped responding. The monitoring in place worked and rang the alarm of the devops team. However it recovered a few minutes later, as if the author of the DDoS was satisfied the service went down and left the scene.

After two days of general slowness and occasional disruptions enough logs were collected and analyzed to conclude that:

  • The DDoS happened in waves, a few hours apart.
  • A wave had from 10,000 to 50,000 unique IPs, each sending an average of two requests.
  • Between 200,000 and 300,000 unique IPs sent requests on a given day.

That was enough to try a simple mitigation:

  • The IP ranges to which each IP belongs were collected (e.g. 1.7.64.1).
  • The top 500 IP ranges that contributed the most number of IPs to the DDoS were blocked.

This successfully stopped over 90% of the DDoS and code.forgejo.org recovered. However, blocking entire IP ranges, sometime millions of IPs at a time, was bound to also impact legitimate users. It was discovered on that occasion that the Golang proxies (keeping a copy of Go packages) are using a few different IP ranges and send requests from hundreds of different IPs. They were unblocked, as well as an individual contributor also blocked by accident.

A week later, on 18 February, the DDoS stopped as suddenly as it started. The number of unique IPs hitting code.forgejo.org went from over 200,000 daily to 15,000 overnight. That concluded a DDoS which used around one million unique IPs to hit code.forgejo.org.

DDoS mitigation

The mitigation of the above mentioned DDoS was hand made. It was surprising to discover that Traefik, the reverse proxy used by the k8s cluster hosting code.forgejo.org, does not provide a method to compute and block the top most used IP ranges. It is a radical but effective measure, a practical way to keep the service afloat until the DDoS goes away.

It took a few days to figure out and implement this mitigation, using a number of shell snippets and learning how IP ranges are distributed over the net. In the spirit of saving time to deal with a similar DDoS, an implementation of this method was described to be part of Forgejo. But it does not belong in Forgejo: it should be implemented in a reverse proxy that sits in front and the idea was abandoned.

A blocker to such an implementation was however resolved: figuring out the list of IP ranges currently in use worldwide. In theory it could be extracted from the same database that is used by whois. But some of the Regional Internet Registries (RIRs) publish their database under restrictive usage terms. Instead, the Routing Information Base (RIB) used by BGP is analyzed weekly and around 500,000 IP ranges are compiled and published.

The next time a DDoS hits code.forgejo.org, the logs will be ready for forensic analysis. The latest database of IP ranges will be used for blocking some of them, keeping the DDoS contained until it goes away.

Federation

A second Federation Dev-Gathering took place on 21 February during which the attendees discussed about the current state and possible ways to involve more contributors in order to speed up the development. Also, an old PR for fixing federated stars in case of servers using allow lists for email domain was finally completed and merged while further work was done for unstar action (a first PR was merged and another one is in progress).

Localization

A new component forgejo-next is now in use. Unlike the main component, this one uses JSON for storing strings. It is easier to parse and free of many quirks the INI format has. Currently it only has a few strings, but it suggested that new strings are now added to it. We’ll also be moving old strings to it, to eventually get rid of the INI translations completely.

With JSON format Forgejo now also supports language-specific plurals, making translator experience more pleasant and UI more cohesive for languages where there’s different number of plurals than two. This improvement will be available in Forgejo v11.

New translations have been started: Hebrew and Romanian.

Sustainability

NLnet, a dutch foundation devoted to the support of free/libre software development, agreed to extend a previous grant for Forgejo by a few months and repurpose the leftover budget. An amendment workplan was agreed on and several contributors started working on it.

The new tasks focus on improving the Forgejo documentation as well as bringing limited moderation capabilities (the option to report content to an instance’s administrators). The architecture and database structure has been agreed on. The next tasks include an admin UI. If you are moderating communities via other software or otherwise have valuable insights that could help us design the UX workflows for admins, feel free to let us know in the corresponding design issue.

We Forge

Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is meant to reflect this diversity and acknowledge all contributions since the last monthly report was published. If you are missing, please ask for an update.