Source tree tags v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 were fixed

TL;DR: Skip this blog if you run a Forgejo instance using the binaries or container images downloaded from https://codeberg.org/forgejo

The symptoms

A Forgejo instance is:

  • unexpectedly running v1.19 despite the version string indicating v1.18
  • refusing to start after upgrading the package (Arch, etc.) to version >= v1.18.3-1

The problem before 11 February 2023

  • The v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 binaries and container images are v1.18 as advertised
  • The v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 tags did not match and were actually set on the development branch

Because of a bug in the release process, the tag of each release was actually set to the tip of the Forgejo development branch at the time of the release. Because Forgejo is a soft-fork, it matched the latest Gitea development branch and had all the Forgejo improvements on top of it. But the Forgejo binaries and container images were created using the v1.18 stable branch.

The problem is fixed on 11 February 2023

The problem was fixed 11 February 2023 ~7am UTC by force pushing the v1.18.1-0, v1.18.2-0, v1.18.2-1, v1.18.3-0 and v1.18.3-1 tags and nothing else. The binaries and container images were good and remain the same.

The consequences

A Forgejo binary built from sources by checking out one of the tags before 11 February 2023 will be running the development branch v1.19 instead of the expected v1.18 branch. Some packages, such as the official Arch Linux package, are also built from the tags and therefore contain Forgejo v1.19 instead of Forgejo v1.18.

The version displayed at the bottom left of the home page will incorrectly be v1.18* instead of v1.19* because it is created at build time using the name of the tag. But there is one difference in the home page that allows you to distinguish them, on the large logo in the center:

  • On v1.19 there is an alt="logo" attribute
  • On v1.18 there is no such attribute

A v1.19 Forgejo instance cannot be downgraded to v1.18. It will refuse to start because there is no safe downgrade path.

Forgejo admins: wait for v1.19

If your current instance unexpectedly runs v1.19, the recommendation is to not upgrade and wait.

Forgejo v1.19 will be released in a few months and the easiest workaround for a Forgejo admin running an instance built from source or installed from a package built from source is to wait until then before upgrading.

Forgejo instances installed from Arch package < v1.18.3.1-1

For those running Arch Linux (or a derivative such as Manjaro or Arch Linux ARM), our current recommendation is to add forgejo to the IgnorePkg list in /etc/pacman.conf until v1.19 is released.

For those who have already upgraded, we recommended downgrading your package to v1.18.3.0-1 as this is the latest version that exhibited this issue. You may be able to find the package for this version in your pacman cache. If the file is not in your pacman cache, it can be downloaded from the Arch Linux archive.

Alternatively, the downgrade utility from the AUR can help with installing an older version and adding Forgejo to IgnorePkg. This tool automates the process above with one command.

When Forgejo v1.19.0-0 is released, you should remove forgejo from IgnorePkg in /etc/pacman.conf and take the upgrade.

If you installed the package after v1.18.3.1-1 was released, your installation is not affected by this issue.

Package authors: (re)build v1.18.3-1

If a package was built using one of the tags above, build (or rebuild if the build happened before 11 February 2023) based on the current tag v1.18.3-1 with hash 4e5be58493 so that future installations are not affected by the problem.

It also means the existing installations will not be able to upgrade. Forgejo will refuse to start because they were really running v1.19 before upgrading. The package should be labeled as such with link to this blog post in the changelog to explain why it happens.

Get in touch, you will get help

If none of the recommended actions work for you, please get in touch at contact@forgejo.org or in the development chatrooom to get help. This was a bug and you deserve all the help you can get. Getting in touch will also help us improve this blog post if needed.