RKE2 K8s Cluster: Capsule, Ceph & OIDC HA (ICSC)

Cluster Kubernetes RKE2 con Capsule, Ceph e OIDC – Progetto ICSC

Implementazione HA, multi-tenancy e autenticazione IAM INFN

• 3 nodi controller in HA come VM su Proxmox VE in anti affinity
• 4 nodi worker bare metal ad alte performance
• Installazione RKE2 tramite modulo Puppet backbone INFN

The slide covers production Ceph storage deployed at Tier-2 Roma. It details the ceph-rbd-csi StorageClass, dynamic provisioning via CSI plugin, and a PVC example requesting 10Gi of RBD storage.

Storage (Ceph)

• Production Ceph storage at Tier-2 Roma
• StorageClass: ceph-rbd-csi
• Dynamic provisioning with CSI plugin
• PVC example: 10Gi RBD storage request

Source: Presentazione tecnica aggiornata – Cluster Kubernetes RKE2 con Capsule, Ceph e OIDC (ICSC)

Native Kubernetes provides limited multi-tenancy via Namespaces and RBAC, with manual resource isolation, complex policies, and risks of team/IAM conflicts. Capsule enhances this by isolating tenants with dedicated namespaces, policies, and resources, plus automatic IAM group authorizations for simpler, secure team management.

Capsule: Multi-Tenancy e Tenant

Source: apiVersion: capsule.clastix.io/v1beta2 kind: Tenant ... owners: users [Schema: IAM → Capsule → Tenant → Namespace]

This slide presents a four-step workflow for OIDC authentication and authorization in an RKE2 Kubernetes cluster. It covers configuring the OIDC provider on the API server, generating user OIDC config, obtaining a short-lived token, and accessing resources via kubectl with Capsule.

OIDC Authentication & Authorization

{ "headers": [ "Step", "Description", "Command" ], "rows": [ [ "1. Configure OIDC", "Configure OIDC provider on RKE2 API Server", "Set API Server flags: --oidc-issuer-url=https://iam.cloud.infn.it --oidc-client-id=icsc-k8s-roma1 etc." ], [ "2. Generate OIDC Config", "Create OIDC configuration for user", "oidc-gen <user> --iss https://iam.cloud.infn.it" ], [ "3. Obtain Token", "Generate short-lived token for cluster audience", "oidc-token --aud icsc-k8s-roma1 <user>" ], [ "4. Access Cluster", "Authenticate to cluster and access resources (via Capsule)", "kubectl --token=$token get pods" ] ] }

Source: Cluster Kubernetes RKE2 con Capsule, Ceph e OIDC (ICSC)

Kube-VIP is deployed as a DaemonSet on controller nodes to provide a shared VIP, ensuring high availability (HA) for the Kubernetes API Server. It uses leader election with ARP and services enabled, via the command $KUBEVIPCMD manifest daemonset --interface eth0 --address 192.168.x.x --controlplane --services --arp --leaderElection.

Kube-VIP e Alta Affidabilità

• Kube-VIP DaemonSet su controller per VIP condiviso
• Garantisce HA dell'API Server Kubernetes
• Leader election con ARP e services abilitati
• $KUBEVIPCMD manifest daemonset --interface eth0 --address 192.168.x.x --controlplane --services --arp --leaderElection

The slide "Risultati e Test" highlights 1 tenant capsule managed and active for 1 day. It reports 100% success across differentiated IAM access with proper user isolation, limited visibility to tenant resources only, and tested failover with confirmed HA controller.

Risultati e Test

• 1: Tenant Capsule Gestiti

Attivi da 1 giorno

• 100%: Accesso IAM Differenziato

Utenti isolati correttamente

• 100%: Visibilità Limitata

Solo risorse tenant

• 100%: Failover Testato

Controller HA confermato Source: Cluster Kubernetes RKE2 ICSC

The conclusion slide lists successfully achieved objectives: a high-availability RKE2 cluster, dynamic Ceph-RBD storage, multi-tenancy with OIDC, and scalable INFN infrastructure. It affirms that the cluster is fully operational and scalable.

Conclusioni

✅ Cluster RKE2 HA ✅ Storage Ceph-RBD dinamico ✅ Multi-tenancy e OIDC ✅ Infra scalabile INFN

Cluster operativo e scalabile!

Obiettivi raggiunti con successo.

Source: Progetto ICSC