fix: anubis must not OG_PASSTHROUGH #481

earl-warren · 2025-05-29T20:26:34Z

earl-warren commented

2025-05-29 20:26:34 +00:00

First-time contributor

OG_PASSTHROUGH is self defeating in the patterns used to crawl
forgejo because there is a very wide variety of URLs. It is only
effective when a lot of requests are made in a short (24h) period
of time.

Since Matrix and all user agents in need of OpenGraph announce
themselves with proper user agents, it is enough to only route
user agents matching Mozilla|Opera to anubis and let the rest go
directly to Forgejo.

OG_PASSTHROUGH is self defeating in the patterns used to crawl forgejo because there is a very wide variety of URLs. It is only effective when a lot of requests are made in a short (24h) period of time. Since Matrix and all user agents in need of OpenGraph announce themselves with proper user agents, it is enough to only route user agents matching Mozilla|Opera to anubis and let the rest go directly to Forgejo.

earl-warren added 1 commit 2025-05-29 20:26:34 +00:00

fix: anubis must not OG_PASSTHROUGH

build / lint (pull_request) Has been cancelled

/ test (pull_request) Has been cancelled

build / lint (push) Has been cancelled

097a9d2071

OG_PASSTHROUGH is self defeating in the patterns used to crawl
forgejo because there is a very wide variety of URLs. It is only
effective when a lot of requests are made in a short (24h) period
of time.

Since Matrix and all user agents in need of OpenGraph announce
themselves with proper user agents, it is enough to only route
user agents matching Mozilla|Opera to anubis and let the rest go
directly to Forgejo.

earl-warren commented

2025-05-29 20:26:35 +00:00

First-time contributor

using https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/482

$ zstdcat traefik-2025-05-23* | grep forgejo-code | grep -v RunnerService | grep -v 'GET /v2/'  | sed -n -e 's|.*HTTP/..." \([0-9][0-9]*\) .*|\1|p' | sort | uniq -c
 171662 200
    439 201
 617884 202
...

The number of 200 requests is consistent with OpenGraph passthrough being deactivated. Now Anubis is effective. Previously it would have sent an additional ~600,000 requests, one for each challenge (code 202).

![image](/attachments/f478ed00-0c63-4bd3-aab3-f71adbccc74a) using https://invisible.forgejo.org/infrastructure/k8s-cluster/issues/482 ```sh $ zstdcat traefik-2025-05-23* | grep forgejo-code | grep -v RunnerService | grep -v 'GET /v2/' | sed -n -e 's|.*HTTP/..." $[0-9][0-9]*$ .*|\1|p' | sort | uniq -c 171662 200 439 201 617884 202 ... ``` The number of 200 requests is consistent with OpenGraph passthrough being deactivated. Now Anubis is effective. Previously it would have sent an additional ~600,000 requests, one for each challenge (code 202).

image.png

12 KiB

root closed this pull request

2025-05-29 20:26:35 +00:00

viceice reviewed 2025-05-29 20:26:35 +00:00

viceice left a comment

First-time contributor

added by F3

flux/apps/base/forgejo-next/forgejo/forgejo.yaml

					
				@ -107,9 +107,7 @@ spec:

				          - name: 'DIFFICULTY'

				            value: '1'

				          - name: 'OG_PASSTHROUGH'

				            value: 'true'

viceice commented

2025-05-29 20:26:35 +00:00

First-time contributor

we should explicit disable it for now

earl-warren commented

2025-05-29 20:26:35 +00:00

First-time contributor

c3ada98f2f..097a9d2071

agreed

https://invisible.forgejo.org/infrastructure/k8s-cluster/compare/c3ada98f2fb4d8123cc4385b295e21e89af2c8c1..097a9d207156ea01dc1a2366b49635871e704aa2 agreed