Learning about Services and how streams are proxy'd #28

New issue

Closed

opened 2025-12-08 12:51:45 +00:00 by Ghostinvisible-forgejo-org · 0 comments

Ghostinvisible-forgejo-org commented

2025-12-08 12:51:45 +00:00

Notes from the chatroom

Fri, Oct 25, 2024, 15:03:08 - * Earl Warren backtracking
Fri, Oct 25, 2024, 15:04:01 - Earl Warren: I'm quite sure there are a lot of things I do not need to worry about. Right now I'm just massively confused about the whole thing. But after bumping into a few walls, I'll get it.
Fri, Oct 25, 2024, 15:04:34 - Earl Warren: trying to separate traefik concepts from k8s concepts is good advice, that helps 🙏
Fri, Oct 25, 2024, 15:06:00 - Earl Warren: I should not be looking at kubectl -n kube-system describe svc traefik to understand flux/infrastructure/controllers/traefik.yaml because the output matches k8s concepts while the file matches traefik concepts
Fri, Oct 25, 2024, 15:06:56 - Michael Kriese: for traefik we tell it: hey listen on public port 2020 and name it ssh-next
Fri, Oct 25, 2024, 15:08:35 - Michael Kriese: then we configure the IngressRouteTCP which says: hey traefik forward traffic from ssh-next to forgejo ssh service
Fri, Oct 25, 2024, 15:09:01 - Earl Warren: the problem is that I perfectly understand the intent. But the details are confusing.
Fri, Oct 25, 2024, 15:09:03 - Michael Kriese: the forgejo-ssh service forwards to traefik pods
Fri, Oct 25, 2024, 15:09:44 - Michael Kriese: yeah, especially all involved objects can use different ports
Fri, Oct 25, 2024, 15:10:10 - Earl Warren: https://doc.traefik.io/traefik/v3.1/routing/entrypoints/ is what I would assume to explain https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L584 but none of the keys match the documentation.
Fri, Oct 25, 2024, 15:10:29 - Michael Kriese: so the kubernetes service is doing a kind of masquerading the service port to the pod port
Fri, Oct 25, 2024, 15:11:03 - Michael Kriese: so the service can use normal http and https ports while the pod needs to use higher ports when running as non root
Fri, Oct 25, 2024, 15:12:09 - Earl Warren: <@viceice:matrix.org "for traefik we tell it: hey list..."> so I assume "port: 2020" is the port that is public? Or is it exposedPort: 2020?
Fri, Oct 25, 2024, 15:12:10 - Michael Kriese: thats another cause why prefer to use named ports, so you don't need to remember to forward port x to port y. just say forward http to http or ssh to ssh
Fri, Oct 25, 2024, 15:12:25 - Michael Kriese: exposedPort: 2020
Fri, Oct 25, 2024, 15:12:47 - Michael Kriese: port: 2020 is the port where the traefik port listens on
Fri, Oct 25, 2024, 15:13:05 - Michael Kriese: so it can be different from exposedPort
Fri, Oct 25, 2024, 15:13:30 - Earl Warren: that's good
Fri, Oct 25, 2024, 15:13:53 - Earl Warren: https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L603-L604

    # -- The exposed port for this service
    exposedPort: 9000

what not helpful in that regard 😄
Fri, Oct 25, 2024, 15:13:58 - Michael Kriese: eg i use port 22 on my public loadbalancer ip and forward it via traefik to forgejo port 2222 (rootless)
Fri, Oct 25, 2024, 15:14:42 - Earl Warren: ok, that's sorted out 👍️
Fri, Oct 25, 2024, 15:15:27 - Michael Kriese: <@earl-warren:matrix.org "https://github.com/traefik/traef..."> but because of https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L602 that port is not published on traefik service
Fri, Oct 25, 2024, 15:15:47 - Michael Kriese: so it's not accessable from internet (if no firewall would block it)
Fri, Oct 25, 2024, 15:16:21 - Michael Kriese: that port should never be publised to internet!!!
Fri, Oct 25, 2024, 15:16:49 - Earl Warren: ```yaml
exposedPort: 2020
expose:
default: false

Fri, Oct 25, 2024, 15:16:55 - Earl Warren: would not be useful at all right?
Fri, Oct 25, 2024, 15:17:06 - Earl Warren: setting an exposed port only to **not** expose it
Fri, Oct 25, 2024, 15:17:35 - Michael Kriese: yes
Fri, Oct 25, 2024, 15:17:56 - Michael Kriese: you also need `port: xxx` to tell traefik to listen on something 
Fri, Oct 25, 2024, 15:18:20 - Earl Warren: now... if I was to query traefik to figure out what ports it has exposed, that would be with describe
Fri, Oct 25, 2024, 15:19:07 - Earl Warren: $ kubectl -n kube-system describe svc traefik | pastebinit 
https://paste.centos.org/view/60dc5349

Fri, Oct 25, 2024, 15:19:08 - Michael Kriese: this is what i do on traefik to listen on port 22:
```yaml
ports:
  ssh:
    expose:
      default: true
    exposedPort: 22
    port: 2222

Fri, Oct 25, 2024, 15:20:43 - Earl Warren: <@earl-warren:matrix.org "$ kubectl -n kube-system describ..."> I think I should not, too early.
Fri, Oct 25, 2024, 15:20:45 - Michael Kriese: you can ignore nodePort for now, it's basically the port which kubernetes uses to forward traffic from Port to targetPort via iptable rules
Fri, Oct 25, 2024, 15:21:25 - Earl Warren: <@viceice:matrix.org "this is what i do on traefik to ..."> now I wonder... what does it mean for traefik to listen to port 2222?
Fri, Oct 25, 2024, 15:21:54 - Earl Warren: or maybe it is the destination port
Fri, Oct 25, 2024, 15:22:15 - Earl Warren: no... I'm wrong
Fri, Oct 25, 2024, 15:22:27 - * Earl Warren scratching his head
Fri, Oct 25, 2024, 15:22:37 - Earl Warren: forgejo-next listens on 2222
Fri, Oct 25, 2024, 15:22:52 - Earl Warren: why would traefik need to... I miss something important
Fri, Oct 25, 2024, 15:23:22 - * Earl Warren reading again above
Fri, Oct 25, 2024, 15:23:25 - Michael Kriese: so for http traffic:

incomming traffic on port 80 is forwarded to port 31879 to kubeproxy process
there it's again forwarded to target port 80 on endpoints 10.42.0.105:80,10.42.0.106:80
Fri, Oct 25, 2024, 15:24:57 - Michael Kriese: <@earl-warren:matrix.org "now I wonder... what does it mea..."> this simply tells traefik to listen on port 2222 (can be any port, using this for dry, so i know it's the forgejo default ssh port for rootless)
Fri, Oct 25, 2024, 15:25:48 - Earl Warren: oh
Fri, Oct 25, 2024, 15:25:54 - Earl Warren: so it is not actually used?
Fri, Oct 25, 2024, 15:26:07 - Earl Warren: nobody connects to 2222 on traefik?
Fri, Oct 25, 2024, 15:26:37 - Earl Warren: it is for documentation purposes so it is used elsewhere?
Fri, Oct 25, 2024, 15:26:47 - Michael Kriese: not directly, you connect to port 22 and the kubernetes service is forwarding to traefik port 2222
Fri, Oct 25, 2024, 15:28:02 - Earl Warren: I'm afraid I'm wasting your time, I need to study this more carefully to get a mental image of what is going on 😊
Fri, Oct 25, 2024, 15:28:55 - Michael Kriese: so for traefik ports you configure two ports port: xxx where traefik listens with an entrypoint and exposedPort: yyy where public traffc needs to arrive (so for a loadbalancer service it can come from outside the cluster
Fri, Oct 25, 2024, 15:30:01 - Michael Kriese: thankfully i've a day off because the daycare is closed today
Fri, Oct 25, 2024, 15:30:15 - Michael Kriese: but i need to wake my daughter up now
Fri, Oct 25, 2024, 15:49:07 - * Earl Warren studies https://paste.centos.org/view/raw/163c149a, the output of helm install something --dry-run --debug -f traefik.yml oci://ghcr.io/traefik/helm/traefik with

deployment:
  replicas: 2
ports:
  web:
    port: 80
    redirectTo:
      port: websecure
      priority: 1
  ssh-next:
    port: 2020
    exposedPort: 2020
    # https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L611-L614
    expose:
      default: true
service:
  annotations:
    metallb.universe.tf/loadBalancerIPs: $failover_ipv4,$failover_ipv6
  spec:
    externalTrafficPolicy: Local
    ipFamilyPolicy: PreferDualStack
logs:
  general:
    level: DEBUG
  access:
    enabled: true
    fields:
      headers:
        # https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L365-L369
        names:
          User-Agent: keep

Fri, Oct 25, 2024, 15:54:35 - Earl Warren: So it creates a service of kind LoadBalancer which listens on all exposed ports and reverse proxy them to the "Deployment" which are pods with names starting with traefik
Fri, Oct 25, 2024, 15:54:37 - Earl Warren: $ kubectl -n kube-system describe pod traefik-7c47c4b4c-47vbx | pastebinit
https://paste.centos.org/view/734fefab

Fri, Oct 25, 2024, 15:57:40 - Earl Warren: each pod listens to ports Ports: 9100/TCP, 2020/TCP, 9000/TCP, 80/TCP, 8443/TCP as specified in the infrastructure/controllers/traefik.yaml and interpreted in https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L646-L673 (for websecure).
Fri, Oct 25, 2024, 16:08:06 - Earl Warren: A Deployment can then request from traefik to be routed whatever comes through this port, this is the purpose of IngressRouteTCP.

apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: forgejo-next-ssh
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - ssh-next # name from traefik port
  routes:
    - match: HostSNI(`*`)
      services:
        - name: forgejo-ssh
          port: ssh # forgejo ssh port on kubernetes service

Fri, Oct 25, 2024, 16:10:33 - Michael Kriese: 👍
Fri, Oct 25, 2024, 16:10:34 - Earl Warren: The entryPoints here matches that part of the traefik Deployment:

        args:
          - "--global.checknewversion"
          - "--global.sendanonymoususage"
          - "--entryPoints.metrics.address=:9100/tcp"
          - "--entryPoints.ssh-next.address=:2020/tcp"
          - "--entryPoints.traefik.address=:9000/tcp"
          - "--entryPoints.web.address=:80/tcp"
          - "--entryPoints.websecure.address=:8443/tcp"
          - "--api.dashboard=true"
          - "--ping=true"
          - "--metrics.prometheus=true"
          - "--metrics.prometheus.entrypoint=metrics"
          - "--providers.kubernetescrd"
          - "--providers.kubernetescrd.allowEmptyServices=true"
          - "--providers.kubernetesingress"
          - "--providers.kubernetesingress.allowEmptyServices=true"
          - "--entryPoints.web.http.redirections.entryPoint.to=:443"
          - "--entryPoints.web.http.redirections.entryPoint.scheme=https"
          - "--entryPoints.web.http.redirections.entryPoint.priority=1"
          - "--entryPoints.websecure.http.tls=true"
          - "--log.level=DEBUG"
          - "--accesslog=true"
          - "--accesslog.fields.defaultmode=keep"
          - "--accesslog.fields.headers.defaultmode=drop"
          - "--accesslog.fields.headers.names.User-Agent=keep"

This is really the only way traefik can know that the container it lives in receives something on a given port.
Fri, Oct 25, 2024, 16:11:21 - Earl Warren: And this is also how it knows to match an entrypoint name to a port.
Fri, Oct 25, 2024, 16:17:47 - Earl Warren: The documentation of the IngressRouteTCP are in https://doc.traefik.io/traefik/v3.1/routing/providers/kubernetes-crd/#kind-ingressroutetcp and quite terse but there is nothing better.
Fri, Oct 25, 2024, 16:18:34 - Earl Warren: match: HostSNI(*) makes sense for SSH because there is no notion of the host from which the request originates.
Fri, Oct 25, 2024, 16:22:36 - Earl Warren: The service part:

      services:
        - name: forgejo-ssh
          port: ssh # forgejo ssh port on kubernetes service

tells it to reverse proxy whatever it gets to the service forgejo-ssh. The documentation is not helpful as it does not give a hint about what is done (reverse proxying) and just

routes[n].services List of Kubernetes service definitions (See below for ExternalName Service setup)

$ kubectl -n forgejo-next get svc forgejo-ssh
NAME          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
forgejo-ssh   ClusterIP   10.43.7.152   <none>        22/TCP    3h22m

Fri, Oct 25, 2024, 16:24:23 - Michael Kriese: <@earl-warren:matrix.org "The service part:"> it connects traefik entry point to kubernetes service and port
Fri, Oct 25, 2024, 16:26:12 - Earl Warren: connects as in proxying the stream from the traefik Deployment to the forgejo-ssh Service? (trying to sort out names in my head) 😁
Fri, Oct 25, 2024, 16:34:38 - * Earl Warren studying the output of

~/k8s-cluster/flux/apps/forgejo-next helm install forgejo --dry-run --debug -f forgejo-next.yaml oci://code.forgejo.org/forgejo-helm/forgejo | pastebinit
https://paste.centos.org/view/275ac368
Fri, Oct 25, 2024, 16:36:54 - Earl Warren: ```yaml

Source: forgejo/templates/gitea/ssh-svc.yaml

apiVersion: v1
kind: Service
metadata:
name: forgejo-ssh
namespace: "default"
labels:
helm.sh/chart: forgejo-10.0.0
app: forgejo
app.kubernetes.io/name: forgejo
app.kubernetes.io/instance: forgejo
app.kubernetes.io/version: "9.0.0"
version: "9.0.0"
app.kubernetes.io/managed-by: Helm
annotations:
{}
spec:
type: ClusterIP
clusterIP: None
ports:

name: ssh
port: 22
targetPort: 2222
protocol: TCP
selector:
app.kubernetes.io/name: forgejo
app.kubernetes.io/instance: forgejo


is what defines the forgejo-ssh service and names the `ssh` port to be listening on 22. And it forwards the stream to port 2222. 
Fri, Oct 25, 2024, 16:38:51 - Earl Warren: This is the only purpose of a service really https://kubernetes.io/docs/concepts/services-networking/service/
Fri, Oct 25, 2024, 16:40:41 - Earl Warren: I suppose Michael Kriese you use reverse-proxy / proxying / connecting (i.e. what a Service) does, interchangeably.
Fri, Oct 25, 2024, 16:41:28 - Earl Warren: And now the packet has reach its destination and that makes logical sense. ✨️
Fri, Oct 25, 2024, 16:43:34 - Earl Warren: I think a key factor to my confusion is the lack of understanding that a Service has a single role which is to lisen to incoming cnx and forward them to a designated destination.
Fri, Oct 25, 2024, 16:44:40 - Earl Warren: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
Fri, Oct 25, 2024, 16:44:50 - Earl Warren: the selector is the destination
Fri, Oct 25, 2024, 16:45:05 - Earl Warren: the port is what to listen to
Fri, Oct 25, 2024, 16:46:39 - Earl Warren: another key confusion factor is the lack of understanding that the chart will create both transparently and I assumed somehow that the Service and the Deployment were one thing. They are two.
Fri, Oct 25, 2024, 16:47:28 - Earl Warren: So now I get that there actually are hoops from Service => Deployment => Service => Deployment etc. 
Fri, Oct 25, 2024, 16:48:24 - Earl Warren: Ok, that's good enough for now. I'm sure I'll get confused again... but maybe not completely lost.
Fri, Oct 25, 2024, 16:48:30 - * Earl Warren archiving notes

Notes [from the chatroom](https://matrix.to/#/!NdTYAXrlSgIkGNiPgQ:matrix.org/$4f6wJ0S3_we3aV4cnmvNLRnjEGpbtWzJzH_JS_vmmi4?via=schinas.net&via=matrix.org&via=mozilla.org) Fri, Oct 25, 2024, 15:03:08 - * Earl Warren backtracking Fri, Oct 25, 2024, 15:04:01 - Earl Warren: I'm quite sure there are a lot of things I do not need to worry about. Right now I'm just massively confused about the whole thing. But after bumping into a few walls, I'll get it. Fri, Oct 25, 2024, 15:04:34 - Earl Warren: trying to separate traefik concepts from k8s concepts is good advice, that helps 🙏 Fri, Oct 25, 2024, 15:06:00 - Earl Warren: I should not be looking at `kubectl -n kube-system describe svc traefik` to understand `flux/infrastructure/controllers/traefik.yaml` because the output matches k8s concepts while the file matches traefik concepts Fri, Oct 25, 2024, 15:06:56 - Michael Kriese: for traefik we tell it: hey listen on public port 2020 and name it `ssh-next` Fri, Oct 25, 2024, 15:08:35 - Michael Kriese: then we configure the `IngressRouteTCP` which says: hey traefik forward traffic from `ssh-next` to forgejo ssh service Fri, Oct 25, 2024, 15:09:01 - Earl Warren: the problem is that I perfectly understand the intent. But the details are confusing. Fri, Oct 25, 2024, 15:09:03 - Michael Kriese: the `forgejo-ssh` service forwards to traefik pods Fri, Oct 25, 2024, 15:09:44 - Michael Kriese: yeah, especially all involved objects can use different ports Fri, Oct 25, 2024, 15:10:10 - Earl Warren: https://doc.traefik.io/traefik/v3.1/routing/entrypoints/ is what I would assume to explain https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L584 but none of the keys match the documentation. Fri, Oct 25, 2024, 15:10:29 - Michael Kriese: so the kubernetes service is doing a kind of masquerading the service port to the pod port Fri, Oct 25, 2024, 15:11:03 - Michael Kriese: so the service can use normal http and https ports while the pod needs to use higher ports when running as non root Fri, Oct 25, 2024, 15:12:09 - Earl Warren: <@viceice:matrix.org "for traefik we tell it: hey list..."> so I assume "port: 2020" is the port that is public? Or is it exposedPort: 2020? Fri, Oct 25, 2024, 15:12:10 - Michael Kriese: thats another cause why prefer to use named ports, so you don't need to remember to forward port x to port y. just say forward http to http or ssh to ssh Fri, Oct 25, 2024, 15:12:25 - Michael Kriese: `exposedPort: 2020` Fri, Oct 25, 2024, 15:12:47 - Michael Kriese: `port: 2020` is the port where the traefik port listens on Fri, Oct 25, 2024, 15:13:05 - Michael Kriese: so it can be different from exposedPort Fri, Oct 25, 2024, 15:13:30 - Earl Warren: that's good Fri, Oct 25, 2024, 15:13:53 - Earl Warren: https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L603-L604 ```yaml # -- The exposed port for this service exposedPort: 9000 ``` what not helpful in that regard 😄 Fri, Oct 25, 2024, 15:13:58 - Michael Kriese: eg i use port 22 on my public loadbalancer ip and forward it via traefik to forgejo port 2222 (rootless) Fri, Oct 25, 2024, 15:14:42 - Earl Warren: ok, that's sorted out 👍️ Fri, Oct 25, 2024, 15:15:27 - Michael Kriese: <@earl-warren:matrix.org "https://github.com/traefik/traef..."> but because of https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L602 that port is not published on `traefik` service Fri, Oct 25, 2024, 15:15:47 - Michael Kriese: so it's not accessable from internet (if no firewall would block it) Fri, Oct 25, 2024, 15:16:21 - Michael Kriese: that port should never be publised to internet!!! Fri, Oct 25, 2024, 15:16:49 - Earl Warren: ```yaml exposedPort: 2020 expose: default: false ``` Fri, Oct 25, 2024, 15:16:55 - Earl Warren: would not be useful at all right? Fri, Oct 25, 2024, 15:17:06 - Earl Warren: setting an exposed port only to **not** expose it Fri, Oct 25, 2024, 15:17:35 - Michael Kriese: yes Fri, Oct 25, 2024, 15:17:56 - Michael Kriese: you also need `port: xxx` to tell traefik to listen on something Fri, Oct 25, 2024, 15:18:20 - Earl Warren: now... if I was to query traefik to figure out what ports it has exposed, that would be with describe Fri, Oct 25, 2024, 15:19:07 - Earl Warren: $ kubectl -n kube-system describe svc traefik | pastebinit https://paste.centos.org/view/60dc5349 Fri, Oct 25, 2024, 15:19:08 - Michael Kriese: this is what i do on traefik to listen on port 22: ```yaml ports: ssh: expose: default: true exposedPort: 22 port: 2222 ``` Fri, Oct 25, 2024, 15:20:43 - Earl Warren: <@earl-warren:matrix.org "$ kubectl -n kube-system describ..."> I think I should not, too early. Fri, Oct 25, 2024, 15:20:45 - Michael Kriese: you can ignore `nodePort` for now, it's basically the port which kubernetes uses to forward traffic from `Port` to `targetPort` via iptable rules Fri, Oct 25, 2024, 15:21:25 - Earl Warren: <@viceice:matrix.org "this is what i do on traefik to ..."> now I wonder... what does it mean for traefik to listen to port 2222? Fri, Oct 25, 2024, 15:21:54 - Earl Warren: or maybe it is the destination port Fri, Oct 25, 2024, 15:22:15 - Earl Warren: no... I'm wrong Fri, Oct 25, 2024, 15:22:27 - * Earl Warren scratching his head Fri, Oct 25, 2024, 15:22:37 - Earl Warren: forgejo-next listens on 2222 Fri, Oct 25, 2024, 15:22:52 - Earl Warren: why would traefik need to... I miss something important Fri, Oct 25, 2024, 15:23:22 - * Earl Warren reading again above Fri, Oct 25, 2024, 15:23:25 - Michael Kriese: so for http traffic: - incomming traffic on port 80 is forwarded to port 31879 to kubeproxy process - there it's again forwarded to target port 80 on endpoints `10.42.0.105:80,10.42.0.106:80` Fri, Oct 25, 2024, 15:24:57 - Michael Kriese: <@earl-warren:matrix.org "now I wonder... what does it mea..."> this simply tells traefik to listen on port 2222 (can be any port, using this for dry, so i know it's the forgejo default ssh port for rootless) Fri, Oct 25, 2024, 15:25:48 - Earl Warren: oh Fri, Oct 25, 2024, 15:25:54 - Earl Warren: so it is not actually used? Fri, Oct 25, 2024, 15:26:07 - Earl Warren: nobody connects to 2222 on traefik? Fri, Oct 25, 2024, 15:26:37 - Earl Warren: it is for documentation purposes so it is used elsewhere? Fri, Oct 25, 2024, 15:26:47 - Michael Kriese: not directly, you connect to port 22 and the kubernetes service is forwarding to traefik port 2222 Fri, Oct 25, 2024, 15:28:02 - Earl Warren: I'm afraid I'm wasting your time, I need to study this more carefully to get a mental image of what is going on 😊 Fri, Oct 25, 2024, 15:28:55 - Michael Kriese: so for traefik ports you configure two ports `port: xxx` where traefik listens with an entrypoint and `exposedPort: yyy` where public traffc needs to arrive (so for a loadbalancer service it can come from outside the cluster Fri, Oct 25, 2024, 15:30:01 - Michael Kriese: thankfully i've a day off because the daycare is closed today Fri, Oct 25, 2024, 15:30:15 - Michael Kriese: but i need to wake my daughter up now Fri, Oct 25, 2024, 15:49:07 - * Earl Warren studies https://paste.centos.org/view/raw/163c149a, the output of `helm install something --dry-run --debug -f traefik.yml oci://ghcr.io/traefik/helm/traefik` with ```yaml deployment: replicas: 2 ports: web: port: 80 redirectTo: port: websecure priority: 1 ssh-next: port: 2020 exposedPort: 2020 # https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L611-L614 expose: default: true service: annotations: metallb.universe.tf/loadBalancerIPs: $failover_ipv4,$failover_ipv6 spec: externalTrafficPolicy: Local ipFamilyPolicy: PreferDualStack logs: general: level: DEBUG access: enabled: true fields: headers: # https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L365-L369 names: User-Agent: keep ``` Fri, Oct 25, 2024, 15:54:35 - Earl Warren: So it creates a service of kind LoadBalancer which listens on all exposed ports and reverse proxy them to the "Deployment" which are pods with names starting with traefik Fri, Oct 25, 2024, 15:54:37 - Earl Warren: $ kubectl -n kube-system describe pod traefik-7c47c4b4c-47vbx | pastebinit https://paste.centos.org/view/734fefab Fri, Oct 25, 2024, 15:57:40 - Earl Warren: each pod listens to ports `Ports: 9100/TCP, 2020/TCP, 9000/TCP, 80/TCP, 8443/TCP` as specified in the [infrastructure/controllers/traefik.yaml](https://code.forgejo.org/infrastructure/k8s-cluster/src/branch/main/flux/infrastructure/controllers/traefik.yaml#L30-L41) and interpreted in https://github.com/traefik/traefik-helm-chart/blob/v32.1.1/traefik/values.yaml#L646-L673 (for websecure). Fri, Oct 25, 2024, 16:08:06 - Earl Warren: A Deployment can then request from traefik to be routed whatever comes through this port, this is the purpose of [IngressRouteTCP](https://code.forgejo.org/infrastructure/k8s-cluster/src/branch/main/flux/apps/forgejo-next/forgejo-route.yaml). ```yaml apiVersion: traefik.io/v1alpha1 kind: IngressRouteTCP metadata: name: forgejo-next-ssh annotations: kubernetes.io/ingress.class: traefik spec: entryPoints: - ssh-next # name from traefik port routes: - match: HostSNI(`*`) services: - name: forgejo-ssh port: ssh # forgejo ssh port on kubernetes service ``` Fri, Oct 25, 2024, 16:10:33 - Michael Kriese: 👍 Fri, Oct 25, 2024, 16:10:34 - Earl Warren: The `entryPoints` here matches that part of the [traefik Deployment](https://paste.centos.org/view/734fefab): ```yaml args: - "--global.checknewversion" - "--global.sendanonymoususage" - "--entryPoints.metrics.address=:9100/tcp" - "--entryPoints.ssh-next.address=:2020/tcp" - "--entryPoints.traefik.address=:9000/tcp" - "--entryPoints.web.address=:80/tcp" - "--entryPoints.websecure.address=:8443/tcp" - "--api.dashboard=true" - "--ping=true" - "--metrics.prometheus=true" - "--metrics.prometheus.entrypoint=metrics" - "--providers.kubernetescrd" - "--providers.kubernetescrd.allowEmptyServices=true" - "--providers.kubernetesingress" - "--providers.kubernetesingress.allowEmptyServices=true" - "--entryPoints.web.http.redirections.entryPoint.to=:443" - "--entryPoints.web.http.redirections.entryPoint.scheme=https" - "--entryPoints.web.http.redirections.entryPoint.priority=1" - "--entryPoints.websecure.http.tls=true" - "--log.level=DEBUG" - "--accesslog=true" - "--accesslog.fields.defaultmode=keep" - "--accesslog.fields.headers.defaultmode=drop" - "--accesslog.fields.headers.names.User-Agent=keep" ``` This is really the only way traefik can know that the container it lives in receives something on a given port. Fri, Oct 25, 2024, 16:11:21 - Earl Warren: And this is also how it knows to match an entrypoint name to a port. Fri, Oct 25, 2024, 16:17:47 - Earl Warren: The documentation of the [IngressRouteTCP](https://code.forgejo.org/infrastructure/k8s-cluster/src/branch/main/flux/apps/forgejo-next/forgejo-route.yaml) are in https://doc.traefik.io/traefik/v3.1/routing/providers/kubernetes-crd/#kind-ingressroutetcp and quite terse but there is nothing better. Fri, Oct 25, 2024, 16:18:34 - Earl Warren: `match: HostSNI(`\*`) ` makes sense for SSH because there is no notion of the host from which the request originates. Fri, Oct 25, 2024, 16:22:36 - Earl Warren: The service part: ```yaml services: - name: forgejo-ssh port: ssh # forgejo ssh port on kubernetes service ``` tells it to reverse proxy whatever it gets to the service `forgejo-ssh`. The documentation is not helpful as it does not give a hint about what is done (reverse proxying) and just > routes[n].services List of Kubernetes service definitions (See below for ExternalName Service setup) ``` $ kubectl -n forgejo-next get svc forgejo-ssh NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE forgejo-ssh ClusterIP 10.43.7.152 <none> 22/TCP 3h22m ``` Fri, Oct 25, 2024, 16:24:23 - Michael Kriese: <@earl-warren:matrix.org "The service part:"> it connects traefik entry point to kubernetes service and port Fri, Oct 25, 2024, 16:26:12 - Earl Warren: connects as in proxying the stream from the traefik Deployment to the forgejo-ssh Service? (trying to sort out names in my head) 😁 Fri, Oct 25, 2024, 16:34:38 - * Earl Warren studying the output of $ ~/k8s-cluster/flux/apps/forgejo-next$ helm install forgejo --dry-run --debug -f forgejo-next.yaml oci://code.forgejo.org/forgejo-helm/forgejo | pastebinit https://paste.centos.org/view/275ac368 Fri, Oct 25, 2024, 16:36:54 - Earl Warren: ```yaml # Source: forgejo/templates/gitea/ssh-svc.yaml apiVersion: v1 kind: Service metadata: name: forgejo-ssh namespace: "default" labels: helm.sh/chart: forgejo-10.0.0 app: forgejo app.kubernetes.io/name: forgejo app.kubernetes.io/instance: forgejo app.kubernetes.io/version: "9.0.0" version: "9.0.0" app.kubernetes.io/managed-by: Helm annotations: {} spec: type: ClusterIP clusterIP: None ports: - name: ssh port: 22 targetPort: 2222 protocol: TCP selector: app.kubernetes.io/name: forgejo app.kubernetes.io/instance: forgejo ``` is what defines the forgejo-ssh service and names the `ssh` port to be listening on 22. And it forwards the stream to port 2222. Fri, Oct 25, 2024, 16:38:51 - Earl Warren: This is the only purpose of a service really https://kubernetes.io/docs/concepts/services-networking/service/ Fri, Oct 25, 2024, 16:40:41 - Earl Warren: I suppose Michael Kriese you use reverse-proxy / proxying / connecting (i.e. what a Service) does, interchangeably. Fri, Oct 25, 2024, 16:41:28 - Earl Warren: And now the packet has reach its destination and that makes logical sense. ✨️ Fri, Oct 25, 2024, 16:43:34 - Earl Warren: I think a key factor to my confusion is the lack of understanding that a Service has a single role which is to lisen to incoming cnx and forward them to a designated destination. Fri, Oct 25, 2024, 16:44:40 - Earl Warren: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service Fri, Oct 25, 2024, 16:44:50 - Earl Warren: the selector is the destination Fri, Oct 25, 2024, 16:45:05 - Earl Warren: the port is what to listen to Fri, Oct 25, 2024, 16:46:39 - Earl Warren: another key confusion factor is the lack of understanding that the chart will create both transparently and I assumed somehow that the Service and the Deployment were one thing. They are two. Fri, Oct 25, 2024, 16:47:28 - Earl Warren: So now I get that there actually are hoops from Service => Deployment => Service => Deployment etc. Fri, Oct 25, 2024, 16:48:24 - Earl Warren: Ok, that's good enough for now. I'm sure I'll get confused again... but maybe not completely lost. Fri, Oct 25, 2024, 16:48:30 - * Earl Warren archiving notes