Forgejo monthly update - April 2025
The monthly report is meant to provide a good overview of what has changed in Forgejo in the past month. If you would like to help, please get in touch in the chatroom or participate in the ongoing discussions.
Releases
Forgejo v11.0
It has been a few weeks since Forgejo v11.0 was released and Codeberg upgraded a little later. Preliminary testing by developers who upgraded their own instances before the release date revealed a few regressions that were fixed quickly, resulting in a on-time release on 16 April 2025. Some users encountered minor issues unrelated to the release, and the release notes have been updated to document the fixes (e.g. a case of an internal Alpine container change that is not backward compatible).
Forgejo Actions related work led to an undocumented default that broke workflows relying on it. This may have been fixed for backward compatibility, but Forgejo Actions is still in an experimental stage, so supporting undocumented behaviors wouldn’t be sustainable. Nevertheless, every effort is being made to ensure that all changes to Forgejo Action are backward compatible and the end-to-end testing is constantly being improved to increase coverage.
There is additional pressure on this release because it is the second LTS release. Along with upgrades from Forgejo v10.0, there will be upgrades from v7.0, which may cause their own issues. So far, all reported upgrades have been successful, except one. An instance was migrated to v7.0 and from PostgreSQL to SQLite. Migrating from one database to another is not advertised as supported, but it is a common misconception that an SQL dump and the associated table schema can be used with any SQL database. In this case, the Forgejo instance worked, but due to the unexpected format of the SQL table schema, migrations gradually corrupted its definition. It was possible to restore and upgrade the instance with some manual work, but this is not something you want to try yourself.
Security releases v11.0.1 and v7.0.15
There will be a security update for the two current LTS versions, v11 and v7, on 2 May 2025. We strongly recommend that all Forgejo installations are upgraded to the latest version as soon as possible.
Helm chart v12
After the release of Forgejo v11.0, the helm chart was updated to version 12 a day later to ensure compatibility with the latest major Forgejo release. In addition, the release is no longer published to https://code.forgejo.org/forgejo-contrib/forgejo and several dependencies have been updated. In addition, there have been minor releases adding support for Anubis to make it easier to use Anubis. Work has also started on replacing Redis with Valkey.
Call for help
Accessibility and usability
Contributions to Forgejo are growing steadily, but the human effort to maintain it doesn’t seem to be growing proportionally. Achieving high quality in Forgejo is a goal, but this also requires reviewing pull requests for accessibility and regressions in usability.
Currently, the accessibility and user research teams are understaffed. So if you have some spare time and are interested in these topics, we would appreciate your help to make Forgejo even better and more inclusive. Please have a look at the call for help if you are interested.
Federation
For a high-level overview, check out the federation roadmap.
Earlier this month a PR about enabling HTTP signatures on all ActivityPub endpoints was merged. HTTP signatures in the context of ActivityPub are often required to interact with a server for authentication and moderation purposes. This merged PR provides an important foundation for further federation work.
In addition, work resumed on the pull request from last year that implements following user activities. This will provide support for following a forgejo user from an ActivityPub server and receiving their activity in the feed.
Sustainability
A new grant application has been submitted for the April 2025 NLnet funding round. Several contributors have applied for a total of €50,000 to complete various tasks. We intend to work on improved moderation tooling, better Git LFS support, making our themes more accessible, and overhauling the contributor workflow. We expect to hear a decision on this application in a few months.
Funded work on the moderation tooling has continued, and we hope to complete all remaining tasks from previous grants soon.
User interface
New improvements to the user interface were merged, making it more modern, consistent and accessible. Some highlights include a reworked switch between comment editor modes and a change to the pagination buttons to ensure that screen readers announce them correctly.
Infrastructure
SSH connection failures
Not long after https://code.forgejo.org/ was upgraded to run the Forgejo v11.0 release candidate, intermittent SSH connection failures were noticed.
It took a few days to realize that the problem was not client-side, but server-side: trying to git fetch a repository every 30 seconds would fail at least once an hour for a few minutes.
This was frequent enough to start looking for the root cause.
A number of theories were considered, ranging from a v11.0 regression to a client-side bug mistaken for server-side behavior. Finally, it was suggested that the cause might be OpenSSH’s built-in mechanism to terminate a connection early if it comes from an IP with a history of abuse, such as an excessive number of failed login attempts.
SSH connections to https://code.forgejo.org/ are reverse proxied, and all connections to the OpenSSH server originate from the same internal (IPv4 or IPv6) IP. It follows that if a bot tries to brute force the SSH port, it will count against these internal IPs, and if the server decides that’s too much, it will effectively block everyone for a few minutes by blocking a single IP. The problem is not new, and such intermittent outages have been occurring since https://code.forgejo.org/ migrated to the k8s cluster. It was only recently that they became more frequent and people started to notice.
The chosen solution was to disable OpenSSH’s built-in rate limiting feature altogether (PerSourcePenalties no) and rely instead on the reverse proxy that sits in front of OpenSSH.
This is not without its challenges, but it is where such a feature belongs instead of being implemented in every service (OpenSSH, Forgejo, etc.) in a different and sometimes unexpected way.
Crawlers hitting Forgejo instances
The discussion about the global trend of excessive crawling hitting Forgejo instances continued as https://code.forgejo.org/ continues to be hit by waves of up to 600,000 unique IPs per day. Thanks to improvements to the k8s infrastructure since the February event, it held up and there was no downtime.
Analysis of this month’s event (and various experiments) over ten days revealed the following
- This is a case of excessive crawling rather than a DDoS, as it continued at the same pace without impacting service availability.
- Most of the millions of unique IPs involved are from residential proxy providers.
- This is not cheap: 10TB of data was sent over 10 days, exceeding all advertised proxy provider plans. This is a competitive market where users, voluntarily or through malicious applications, contribute their private IPs and bandwidth in exchange for an estimated $0.1 per GB, which is then sold to the customer for profit.
These findings do not provide clarity on why such excessive crawling occurs, how it is funded, or by whom. However, they are useful in eliminating some speculation. For example, the theory that an individual or organization maliciously tries to increase the load of a Forgejo instance out of anger. This would be plausible if it cost a hundred USD or less. It is very unlikely if it costs at least thousands of USD and requires a non-standard contract with a private proxy provider.
While experimenting, some Forgejo users were blocked on 26 April when attempting to block hundreds of IP ranges. Hundreds of IPs in the same range as them were being used for crawling, and they got caught in the middle. This is a concrete example of how effective residential proxy providers are.
Anubis has been deployed on https://try.next.forgejo.org/ and https://dev.next.forgejo.org/. The instances are victims of excessive crawling and this will reduce the pressure on the Forgejo infrastructure. This was made easy by leveraging new features of the latest Forgejo helm 12.3.0 release. The goal is to eventually do the same on https://code.forgejo.org/ after gaining devops experience running Anubis, if only to deny the data to the organization behind the excessive crawling.
If you want to know more or have insights to share, check out the discussion “Crawlers hitting Forgejo instances - global abuse trend”.
Other
Forgejo Contribution Workshop
On 27 April 2025, n0toose held a Forgejo Contribution Workshop at the Chaos Computer Club Aachen e.V. (CCCAC). The workshop was held in person and four people attended. It was a success and led to the discovery of some problems with the onboarding documentation. In addition, it resulted in two completed PRs and some others that are currently in progress. We welcome every new contributor to Forgejo!
There are plans to repeat such a workshop in the future at other locations.
Talk about Federation
Jerger will give a talk in German about the state of federation at the FSCK in Backnang, Germany. It will be a short talk giving insights into the engine room, security considerations, challenges and the way forward.
A recording should be available at https://media.ccc.de/ afterwards.
Recognition by the Git project
The official Git website, https://git-scm.com/, has undergone a recent update. The updated homepage now features a section highlighting various Forges. This change prominently showcases the logos of Forgejo and Codeberg, highlighting the contribution to the broader Git ecosystem.
We Forge
Forgejo is a community of people who contribute in an inclusive environment. We forge on an equal footing, by reporting a bug, voicing an idea in the chatroom or implementing a new feature. The following list of contributors is intended to reflect this diversity and to acknowledge all the contributions made over the past month. If you are missing, please ask for an update.
- https://codeberg.org/0ko
- https://codeberg.org/0x5f
- https://codeberg.org/achyrva
- https://codeberg.org/acidbong
- https://codeberg.org/adamu
- https://codeberg.org/aec
- https://codeberg.org/alperen
- https://codeberg.org/Aminda
- https://codeberg.org/Andre601
- https://codeberg.org/angelnu
- https://codeberg.org/antaanimosity
- https://codeberg.org/arija
- https://codeberg.org/artnay
- https://codeberg.org/Atalanttore
- https://codeberg.org/Atemu
- https://codeberg.org/awiteb
- https://codeberg.org/Baa
- https://codeberg.org/bbqrob
- https://codeberg.org/Beowulf
- https://codeberg.org/bjoernager
- https://codeberg.org/bkil
- https://codeberg.org/bmorel
- https://codeberg.org/brechtvl
- https://codeberg.org/bziemons
- https://codeberg.org/caesar
- https://codeberg.org/camillo
- https://codeberg.org/christopher-besch
- https://codeberg.org/cobak78
- https://codeberg.org/crystal
- https://codeberg.org/cstaky
- https://codeberg.org/darkswordreams
- https://codeberg.org/davrot
- https://codeberg.org/dbat
- https://codeberg.org/DD-P
- https://codeberg.org/dejan
- https://codeberg.org/devnewton
- https://codeberg.org/Dirk
- https://codeberg.org/dorianim
- https://codeberg.org/dovah-kiin
- https://codeberg.org/dploeger
- https://codeberg.org/dragast
- https://codeberg.org/earl-warren
- https://codeberg.org/Edgarsons
- https://codeberg.org/EkaterinePapava
- https://codeberg.org/el0n
- https://codeberg.org/eloy
- https://codeberg.org/Ember
- https://codeberg.org/famfo
- https://codeberg.org/faravah
- https://codeberg.org/Felitendo
- https://codeberg.org/FileX
- https://codeberg.org/fito
- https://codeberg.org/Fjuro
- https://codeberg.org/fkiraly
- https://codeberg.org/floss4good
- https://codeberg.org/fnetX
- https://codeberg.org/fogti
- https://codeberg.org/forestjohnson
- https://codeberg.org/frankm
- https://codeberg.org/fraschm98
- https://codeberg.org/frnmst
- https://codeberg.org/GewoonLeon
- https://codeberg.org/Gnaaarwhal
- https://codeberg.org/gridhead
- https://codeberg.org/gustavblass
- https://codeberg.org/Gusted
- https://codeberg.org/gwenya
- https://codeberg.org/gwhitney
- https://codeberg.org/hannesm
- https://codeberg.org/Hansi231
- https://codeberg.org/happwrap
- https://codeberg.org/harabat
- https://codeberg.org/HaraldToepfer
- https://codeberg.org/hexa
- https://codeberg.org/HinoByte
- https://codeberg.org/HJVT
- https://codeberg.org/hughrun
- https://codeberg.org/izzy
- https://codeberg.org/jackc
- https://codeberg.org/jak2k
- https://codeberg.org/JakobDev
- https://codeberg.org/jcheatum
- https://codeberg.org/Jengro
- https://codeberg.org/jerger
- https://codeberg.org/jexner
- https://codeberg.org/jmjl
- https://codeberg.org/JohnMoon-VTS
- https://codeberg.org/jonzuka
- https://codeberg.org/JSchlarb
- https://codeberg.org/justbispo
- https://codeberg.org/justknow
- https://codeberg.org/jutty
- https://codeberg.org/KaKi87
- https://codeberg.org/kayprish
- https://codeberg.org/kbruen
- https://codeberg.org/kibner
- https://codeberg.org/Kidswiss
- https://codeberg.org/kita
- https://codeberg.org/Kladky
- https://codeberg.org/klausfyhn
- https://codeberg.org/Kokomo
- https://codeberg.org/krisavi
- https://codeberg.org/kryptonian
- https://codeberg.org/Kwonunn
- https://codeberg.org/kwoot
- https://codeberg.org/Lachstec
- https://codeberg.org/lenikadali
- https://codeberg.org/LinuxUserGD
- https://codeberg.org/litchipi
- https://codeberg.org/ltning
- https://codeberg.org/lucasmz
- https://codeberg.org/maartenbeeckmans
- https://codeberg.org/mahlzahn
- https://codeberg.org/Maks1mS
- https://codeberg.org/markturney
- https://codeberg.org/martinwguy
- https://codeberg.org/Matlib
- https://codeberg.org/Mayaleeeeee
- https://codeberg.org/mdk
- https://codeberg.org/meaz
- https://codeberg.org/Menelion
- https://codeberg.org/mengzhuo
- https://codeberg.org/MerlijnWajer
- https://codeberg.org/Mez
- https://codeberg.org/mibar
- https://codeberg.org/Miguel_PL
- https://codeberg.org/mirkoperillo
- https://codeberg.org/mlncn
- https://codeberg.org/mmorais
- https://codeberg.org/mnalis
- https://codeberg.org/moonglum
- https://codeberg.org/mpsijm
- https://codeberg.org/msrd0
- https://codeberg.org/n0toose
- https://codeberg.org/nasmevka
- https://codeberg.org/Naxdy
- https://codeberg.org/neighborsbear
- https://codeberg.org/nekoedges
- https://codeberg.org/nikolayKeniston
- https://codeberg.org/Nordfriese
- https://codeberg.org/NormPlum
- https://codeberg.org/nostar
- https://codeberg.org/omenos
- https://codeberg.org/onitake
- https://codeberg.org/oscarotero
- https://codeberg.org/otf31
- https://codeberg.org/Outbreak2096
- https://codeberg.org/paspflue
- https://codeberg.org/pastk
- https://codeberg.org/pat-s
- https://codeberg.org/pboguslawski
- https://codeberg.org/pennersr
- https://codeberg.org/pep.
- https://codeberg.org/PeterDaveHello
- https://codeberg.org/pixelcode
- https://codeberg.org/Pneuma
- https://codeberg.org/polyfloyd
- https://codeberg.org/Porrumentzio
- https://codeberg.org/poVoq
- https://codeberg.org/programmerjake
- https://codeberg.org/ProjectPatatoe
- https://codeberg.org/puzzle-it-nu
- https://codeberg.org/radon512
- https://codeberg.org/rarogcmex
- https://codeberg.org/rfc2549
- https://codeberg.org/ringofstorms
- https://codeberg.org/ripple_prozac
- https://codeberg.org/RivenSkaye
- https://codeberg.org/RokeJulianLockhart
- https://codeberg.org/rvba
- https://codeberg.org/ryanlerch
- https://codeberg.org/Ryuno-Ki
- https://codeberg.org/sandro
- https://codeberg.org/sandwitch
- https://codeberg.org/sawntoe
- https://codeberg.org/sclu1034
- https://codeberg.org/senekor
- https://codeberg.org/ShadowPeo
- https://codeberg.org/Shuroii
- https://codeberg.org/Skyper
- https://codeberg.org/slatian
- https://codeberg.org/snematoda
- https://codeberg.org/Snoweuph
- https://codeberg.org/SomeTr
- https://codeberg.org/spiffyk
- https://codeberg.org/Squel
- https://codeberg.org/stephanm
- https://codeberg.org/stku
- https://codeberg.org/tacaly
- https://codeberg.org/tag
- https://codeberg.org/tebriel
- https://codeberg.org/tecosaur
- https://codeberg.org/themandalorian
- https://codeberg.org/toolforger
- https://codeberg.org/toras9000
- https://codeberg.org/tpoisseau
- https://codeberg.org/unikitty
- https://codeberg.org/ver4a
- https://codeberg.org/viceice
- https://codeberg.org/voltagex
- https://codeberg.org/vroyer
- https://codeberg.org/wangito33
- https://codeberg.org/webfly2
- https://codeberg.org/whytf
- https://codeberg.org/wolftune
- https://codeberg.org/Wuzzy
- https://codeberg.org/Xinayder
- https://codeberg.org/XOYZ69
- https://codeberg.org/xtex
- https://codeberg.org/yuki0iq
- https://codeberg.org/yurtpage
- https://codeberg.org/zerica
- https://codeberg.org/zeridon
- https://codeberg.org/zloidemon
- https://codeberg.org/zokki
- https://codeberg.org/Zughy