从零编写 Kubernetes Controller

发表于 更新于 分类于 ，

image

控制器，就是监控kubernetes中各种资源的变化，并把当前状态变成期望状态。

Overview

我们在这个章节中来实现一个基于用户的 Quota 配额的控制器，在社区提供的 资源配额 中仅仅能够根据命名空间进行资源配额，但是实际上我们可以根据用户（跨多个Namespace）进行资源配额。那么我们就需要进行一些定制化的开发。

这里简化，我们仅针对 CPU 和 MEM 进行限制。

那我们进行需求分析

  1. 需要能够定义用户的 ResourceQuota 配额
  2. 能够针对用户的 ResourceQuota 配额，进行资源配额的限制
    1. 需要计算用户当前已经使用的配额量
    2. 针对超限的用户禁止其继续创建

根据我们的需求，我们在 Kubernetes 提供的一系列能力中可以找到我们相对需要使用的工具

  • CRD 定制资源: 申明我们自定义的数据结构来储存对用户的资源配额的信息
  • Controller 动态准入控制: 通过 Reconcile，获得用户当前所有的资源配额。
  • ValidatingAdmissionWebhook: 通过回调接口，获得用户对应的信息

那么我们系统的简化架构如下

Github 最终源码

CRD

首先设计我们的数据结构，这里需要使用 使用 CustomResourceDefinition 扩展 Kubernetes API 我们可以直接编写 Yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # 名字必需与下面的 spec 字段匹配，并且格式为 '<名称的复数形式>.<组名>'
  name: crontabs.stable.example.com
spec:
  # 组名称，用于 REST API: /apis/<组>/<版本>
  group: stable.example.com
  # 列举此 CustomResourceDefinition 所支持的版本
  versions:
    - name: v1
      # 每个版本都可以通过 served 标志来独立启用或禁止
      served: true
      # 其中一个且只有一个版本必需被标记为存储版本
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # 可以是 Namespaced 或 Cluster
  scope: Namespaced
  names:
    # 名称的复数形式，用于 URL：/apis/<组>/<版本>/<名称的复数形式>
    plural: crontabs
    # 名称的单数形式，作为命令行使用时和显示时的别名
    singular: crontab
    # kind 通常是单数形式的驼峰命名（CamelCased）形式。你的资源清单会使用这一形式。
    kind: CronTab
    # shortNames 允许你在命令行使用较短的字符串来匹配资源
    shortNames:
    - ct

但是这样太难以维护了，我们一般使用另外一种方式 kubebuilder 通过代码生成的方式来生成这个 Yaml 和一些基本的代码

builder

首先安装 kubebuilder

1
2
3
# download kubebuilder and install locally.
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

那我们开始初始化一个项目

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ mkdir quota-limit
$ cd quota-limit
$ kubebuilder init --domain tutorial.controller.io --repo tutorial.controller.io/quota-limit

INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.0
go: downloading sigs.k8s.io/controller-runtime v0.17.0
go: downloading k8s.io/apimachinery v0.29.0
go: downloading k8s.io/client-go v0.29.0
go: downloading github.com/go-logr/logr v1.4.1
go: downloading k8s.io/klog/v2 v2.110.1
go: downloading k8s.io/api v0.29.0
go: downloading k8s.io/component-base v0.29.0
go: downloading github.com/evanphx/json-patch/v5 v5.8.0
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1
go: downloading k8s.io/apiextensions-apiserver v0.29.0
go: downloading github.com/prometheus/client_golang v1.18.0
go: downloading k8s.io/kube-openapi v0.0.0-20231010175941-2dd684a91f00
go: downloading golang.org/x/net v0.19.0

最终生成的目录如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
$ tree .
.
├── Dockerfile
├── Makefile # 一些常见的脚本命令
├── PROJECT
├── README.md
├── cmd  # 函数命令
│   └── main.go  # 入口函数
├── config  # 各种配置文件
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   └── rbac
│       ├── auth_proxy_client_clusterrole.yaml
│       ├── auth_proxy_role.yaml
│       ├── auth_proxy_role_binding.yaml
│       ├── auth_proxy_service.yaml
│       ├── kustomization.yaml
│       ├── leader_election_role.yaml
│       ├── leader_election_role_binding.yaml
│       ├── role.yaml
│       ├── role_binding.yaml
│       └── service_account.yaml
├── go.mod
├── go.sum
├── hack  # hack 文件
│   └── boilerplate.go.txt 
└── test  # 生成的测试用例目录
    ├── e2e
    │   ├── e2e_suite_test.go # 统一入口
    │   └── e2e_test.go  # 端到端测试目录
    └── utils  # 单侧的一些帮助函数
        └── utils.go

生成文件

go.mod

1
2
3
4
5
6
7
8
9
10
11
12
// 这里就是我们的模块名称
module tutorial.controller.io/quota-limit

go 1.21

require (
	github.com/onsi/ginkgo/v2 v2.14.0
	github.com/onsi/gomega v1.30.0
	k8s.io/apimachinery v0.29.0
	k8s.io/client-go v0.29.0
	sigs.k8s.io/controller-runtime v0.17.0
)

从 go.mod 中我们就看到了所有的依赖，相对非常的干净

  • onsi/gomega 和 onsi/ginkgo： 提供测试相关能力
  • apimachinery 和 client-go： 社区提供的 k8s 的标准的 API 接口，就和 Kubectel 的能力类似
  • controller-runtime：kubernetes相关的一个开源项目，用于构建k8s控制器。通常可与kubebuilder一起使用构建CRD以对应的自定义控制器。

e2e_test

这里我们生成一个 e2e_test.go 文件，这里使用了 ginkgo 作为测试框架。

不过这里其实在实际的生成中我们并不会使用，因为往往有大量的其他依赖，端到端的测试相较于单元测试来说并没有太多的优势，反而有很多麻烦之处。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

// e2e_test.go 
var _ = Describe("controller", Ordered, func() {

  // 在 controller 测试之前，会执行 BeforeAll
	BeforeAll(func() {
		By("installing prometheus operator")
		Expect(utils.InstallPrometheusOperator()).To(Succeed())

		By("installing the cert-manager")
		Expect(utils.InstallCertManager()).To(Succeed())

		By("creating manager namespace")
		cmd := exec.Command("kubectl", "create", "ns", namespace)
		_, _ = utils.Run(cmd)
	})

  // 在 controller 测试之后，会执行 AfterAll
  AfterAll(func() {
		By("uninstalling the Prometheus manager bundle")
		utils.UninstallPrometheusOperator()

		By("uninstalling the cert-manager bundle")
		utils.UninstallCertManager()

		By("removing manager namespace")
		cmd := exec.Command("kubectl", "delete", "ns", namespace)
		_, _ = utils.Run(cmd)
	})

  // 其中一个测试用例
	Context("Operator", func() {
    // 测试 Cares
		It("should run successfully", func() {
			var controllerPodName string
			var err error

			// projectimage stores the name of the image used in the example
			var projectimage = "example.com/quota-limit:v0.0.1"

      // 这里构建镜像
			By("building the manager(Operator) image")
			cmd := exec.Command("make", "docker-build", fmt.Sprintf("IMG=%s", projectimage))
			_, err = utils.Run(cmd)
			ExpectWithOffset(1, err).NotTo(HaveOccurred())

      
      // 通过将镜像导入 Kind，Kind 是一个虚拟 K8s 的工具 https://kind.sigs.k8s.io/
			By("loading the the manager(Operator) image on Kind")
			err = utils.LoadImageToKindClusterWithName(projectimage)
			ExpectWithOffset(1, err).NotTo(HaveOccurred())

      // 安装 CRD
			By("installing CRDs")
			cmd = exec.Command("make", "install")
			_, err = utils.Run(cmd)

      // 部署 controller
			By("deploying the controller-manager")
			cmd = exec.Command("make", "deploy", fmt.Sprintf("IMG=%s", projectimage))
			_, err = utils.Run(cmd)
			ExpectWithOffset(1, err).NotTo(HaveOccurred())

      // 判断 POD 是否运行
			By("validating that the controller-manager pod is running as expected")
			verifyControllerUp := func() error {
				// Get pod name

				cmd = exec.Command("kubectl", "get",
					"pods", "-l", "control-plane=controller-manager",
					"-o", "go-template={{ range .items }}"+
						"{{ if not .metadata.deletionTimestamp }}"+
						"{{ .metadata.name }}"+
						"{{ \"\\n\" }}{{ end }}{{ end }}",
					"-n", namespace,
				)

				podOutput, err := utils.Run(cmd)
				ExpectWithOffset(2, err).NotTo(HaveOccurred())
				podNames := utils.GetNonEmptyLines(string(podOutput))
				if len(podNames) != 1 {
					return fmt.Errorf("expect 1 controller pods running, but got %d", len(podNames))
				}
				controllerPodName = podNames[0]
				ExpectWithOffset(2, controllerPodName).Should(ContainSubstring("controller-manager"))

				// 验证 POD 状态
				cmd = exec.Command("kubectl", "get",
					"pods", controllerPodName, "-o", "jsonpath={.status.phase}",
					"-n", namespace,
				)
				status, err := utils.Run(cmd)
				ExpectWithOffset(2, err).NotTo(HaveOccurred())
				if string(status) != "Running" {
					return fmt.Errorf("controller pod in %s status", status)
				}
				return nil
			}
			EventuallyWithOffset(1, verifyControllerUp, time.Minute, time.Second).Should(Succeed())

		})
	})
})


// e2e_suite_test.go
// 这里就相对简单了，因为我们可能会写多个单侧文件，这里有一个统一的入口，也是 Ginko 的惯用法之一。
// Run e2e tests using the Ginkgo runner.
func TestE2E(t *testing.T) {
	RegisterFailHandler(Fail)
	fmt.Fprintf(GinkgoWriter, "Starting quota-limit suite\n")
	RunSpecs(t, "e2e suite")
}

boilerplate.go.txt

boilerplate.go.txt 实际上就是申明文件的 Licensed 在开源社区，每个项目都应该申明自己的协议类型，这里默认使用了 apache 协议，这里就只是定义了协议模板。

cmd

在 go 语言中，我们一般会生成一个可执行文件

1
go build -o bin/main cmd/main.go

一个项目可能会生成多个可执行文件，一般都放在 CMD 目录之中。可以参考 Standard Go Project Layout

main.go

main.go 作为我们系统的入口，也生成了非常多的内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
func main() {
	var metricsAddr string
	var enableLeaderElection bool
	var probeAddr string
	var secureMetrics bool
	var enableHTTP2 bool

  // flag 是 go 的 SDK，用于解析命令行参数，比如 --metrics-bind-address=:8081 就可以覆盖默认值 8080
	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")
	flag.BoolVar(&secureMetrics, "metrics-secure", false,
		"If set the metrics endpoint is served securely")
	flag.BoolVar(&enableHTTP2, "enable-http2", false,
		"If set, HTTP/2 will be enabled for the metrics and webhook servers")
	opts := zap.Options{
		Development: true,
	}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

  // ctrl 就是 controller runtime，这里设置了 Logger
  // 就和 Spring 中 Logger 一样，定义了接口，需要具体的 logger 库来实现，这里使用了 zap log 库
	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

	// if the enable-http2 flag is false (the default), http/2 should be disabled
	// due to its vulnerabilities. More specifically, disabling http/2 will
	// prevent from being vulnerable to the HTTP/2 Stream Cancelation and
	// Rapid Reset CVEs. For more information see:
	// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
	// - https://github.com/advisories/GHSA-4374-p667-p6c8
	disableHTTP2 := func(c *tls.Config) {
		setupLog.Info("disabling http/2")
		c.NextProtos = []string{"http/1.1"}
	}

	tlsOpts := []func(*tls.Config){}
	if !enableHTTP2 {
		tlsOpts = append(tlsOpts, disableHTTP2)
	}

  // 初始化 webhook
	webhookServer := webhook.NewServer(webhook.Options{
		TLSOpts: tlsOpts,
	})

  // 初始化 manager，这是 controller runtime 的惯用法
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme: scheme,
		Metrics: metricsserver.Options{
			BindAddress:   metricsAddr,
			SecureServing: secureMetrics,
			TLSOpts:       tlsOpts,
		},
		WebhookServer:          webhookServer,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "48ab7dee.tutorial.controller.io",
		// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
		// when the Manager ends. This requires the binary to immediately end when the
		// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
		// speeds up voluntary leader transitions as the new leader don't have to wait
		// LeaseDuration time first.
		//
		// In the default scaffold provided, the program ends immediately after
		// the manager stops, so would be fine to enable this option. However,
		// if you are doing or is intended to do any operation such as perform cleanups
		// after the manager stops then its usage might be unsafe.
		// LeaderElectionReleaseOnCancel: true,
	})
	if err != nil {
		setupLog.Error(err, "unable to start manager")
		os.Exit(1)
	}

	//+kubebuilder:scaffold:builder

	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up health check")
		os.Exit(1)
	}
	if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up ready check")
		os.Exit(1)
	}

  // 启动 manager
	setupLog.Info("starting manager")
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}

config

这里就是我们大头部分了，因为和 k8s 打交道，我们知道我们和 k8s 之间的交互是通过 各种 YAML 文件来进行的。这里将各种文件都拆分到不同的目录了。
不过在看具体的文件之前，我们需要了解下 使用 Kustomize 对 Kubernetes 对象进行声明式管理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
.
├── default
│   ├── kustomization.yaml
│   ├── manager_auth_proxy_patch.yaml # kube-rbac-proxy，下文介绍
│   └── manager_config_patch.yaml
├── manager
│   ├── kustomization.yaml
│   └── manager.yaml  # 最终部署的 Deployment 文件
├── prometheus
│   ├── kustomization.yaml
│   └── monitor.yaml # Prometheus 的监控配置
└── rbac # 各种类型配置
    ├── auth_proxy_client_clusterrole.yaml
    ├── auth_proxy_role_binding.yaml
    ├── auth_proxy_role.yaml
    ├── auth_proxy_service.yaml
    ├── kustomization.yaml
    ├── leader_election_role_binding.yaml
    ├── leader_election_role.yaml
    ├── role_binding.yaml
    ├── role.yaml # 角色配置
    └── service_account.yaml  # ServiceAccount 配置
Kustomize

Kustomize 是一款 Kubernetes 原生的配置管理工具，其核心理念是允许用户自定义 Kubernetes 资源配置，而无需直接修改原始的 YAML 文件。这在很大程度上提高了配置的可维护性和可重用性。Kustomize 使用声明式的方式来定制资源，通过一系列预定义的指令和规则，用户可以对基础资源进行修改、添加或删除。Kustomize 与其他模板技术相比，具有以下优势：

  1. 易于理解：Kustomize 使用简单的 YAML 语法，与 Kubernetes 资源本身的定义方式保持一致，易于学习和理解。
  2. 原子性：Kustomize 的覆盖策略允许用户精确地修改特定部分的配置，而不会影响其他部分。这样可以确保修改的原子性，避免出现意外的副作用。
  3. 可组合性：Kustomize 支持将多个覆盖层叠加在一起，从而形成一个完整的定制资源。这使得用户可以在多个环境和场景之间复用同一套基础配置，降低了维护成本。
  4. 集成 Kubernetes：Kustomize 自 Kubernetes v1.14 起已经被集成在 kubectl 中，用户无需安装额外的工具即可使用 Kustomize。

这个其实和 helm chart 类似的定位，只是写法不同，但是其核心思想是一致的，就是通用一些模板文件来覆盖定制

可以参考 kustomize 入门 等文章入门，不过这类型的东西一般都是在具体场景中使用的时候再多去了解即可。

在项目的 Makefile 中

1
2
3
4
5
6
7
8
9
.PHONY: deploy
deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default | $(KUBECTL) apply -f -

.PHONY: undeploy
undeploy: kustomize ## Undeploy controller from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.
	$(KUSTOMIZE) build config/default | $(KUBECTL) delete --ignore-not-found=$(ignore-not-found) -f -

我们可以看到实际在部署的时候，调用了 deploy， 就是去修改了 Deployment 中的镜像，然后通过 Kustomize 构建了所有的 YAML 文件，最后通过 kubectl apply -f 将所有的 YAML 文件部署到 K8s 中。

我们增加一个 view

1
2
3
view: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
	$(KUSTOMIZE) build config/default | cat

执行一下 make view 我们就可以看到最终的输出了，实际上就是生成了大量的 yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ make view
/home/yanick/codes/go/quota-limit/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /home/yanick/codes/go/quota-limit/bin/kustomize-v5.3.0 edit set image controller=controller:latest
/home/yanick/codes/go/quota-limit/bin/kustomize-v5.3.0 build config/default | cat
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/component: manager
    app.kubernetes.io/created-by: quota-limit
    app.kubernetes.io/instance: system
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: namespace
    app.kubernetes.io/part-of: quota-limit
    control-plane: controller-manager
  name: quota-limit-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: rbac
    app.kubernetes.io/created-by: quota-limit
    app.kubernetes.io/instance: controller-manager-sa
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: serviceaccount
    app.kubernetes.io/part-of: quota-limit
  name: quota-limit-controller-manager
  namespace: quota-limit-system
---

kube-rbac-proxy

这里组件相对少见，在 manager_auth_proxy_patch.yaml 中定义了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: kube-rbac-proxy
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - "ALL"
        image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
        args:
        - "--secure-listen-address=0.0.0.0:8443"
        - "--upstream=http://127.0.0.1:8080/"
        - "--logtostderr=true"
        - "--v=0"
        ports:
        - containerPort: 8443
          protocol: TCP
          name: https
        resources:
          limits:
            cpu: 500m
            memory: 128Mi
          requests:
            cpu: 5m
            memory: 64Mi
      - name: manager
        args:
        - "--health-probe-bind-address=:8081"
        - "--metrics-bind-address=127.0.0.1:8080"
        - "--leader-elect"

实际上就是在 Controller 中挂入了一个 Sidecar, 其作用是

代理是为了能够保护普罗米修斯指标的端点,在一个场景中，攻击者可能获得对Pod的完全控制，攻击者将能够发现大量关于工作负载以及相关工作负载的当前负载的信息

原理是

1.kube-rbac-proxy首先确定哪个用户在执行请求, 支持客户端使用TLS或者token(也就是prometheus访问的时候应带上)
2. 对于客户端证书，只需根据配置的CA对证书进行验证。（TLS逻辑）
3. 如果提交了一个承载令牌，则authentication.k8s.io用于执行一个标记检查。（TOKEN逻辑）
4. 一旦用户通过了身份验证，再一次authentication.k8s.io用于执行一个SubjectAccessReview（确保这个用户有访问的权限）为了对各个请求进行授权，需要确保经过身份验证的用户具有所需的RBAC角色。

实际上很少使用，一般可以去掉，这里是一个非常高阶的安全需求，假设系统中有一些POD被攻陷，通过 prometheus 的指标接口获得很多业务相关的敏感信息

创建 API

接下来我们创建 CRD 定义

1
2
3
4
5
6
7
8
9
10
11
12
13
$ kubebuilder create api --group resource --version v1 --kind Quota
INFO Create Resource [y/n]                        
y
INFO Create Controller [y/n]                      
y
INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO api/v1/quota_types.go                        
INFO api/v1/groupversion_info.go                  
INFO internal/controller/suite_test.go            
INFO internal/controller/quota_controller.go      
INFO internal/controller/quota_controller_test.go 
INFO Update dependencies:

执行完成之后，在我们的目录下就生成了 api 的定义文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ tree .
.
├── Dockerfile
├── Makefile
├── PROJECT
├── README.md
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── quota_types.go  <---- 定义文件
│       └── zz_generated.deepcopy.go
├── internal
│   └── controller
│       ├── quota_controller.go   <--- 模板代码
│       ├── quota_controller_test.go
│       └── suite_test.go

$ git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   PROJECT
        new file:   api/v1/groupversion_info.go  # 增加了 groupversion_info 文件，这里定义了 CRD 的 KIND 和 Version
        new file:   api/v1/quota_types.go        # Quota CRD 的定义
        new file:   api/v1/zz_generated.deepcopy.go  # 一些自动生成的代码
        modified:   cmd/main.go                    # main.go 新增了一些内容
        new file:   config/crd/kustomization.yaml
        new file:   config/crd/kustomizeconfig.yaml
        modified:   config/default/kustomization.yaml
        new file:   config/rbac/quota_editor_role.yaml   # 给 CRD 生成了 编辑的角色
        new file:   config/rbac/quota_viewer_role.yaml   # 给 CRD 生成了 只读的角色
        new file:   config/samples/kustomization.yaml    # 增加了一个 sample 文件夹
        new file:   config/samples/resource_v1_quota.yaml
        new file:   internal/controller/quota_controller.go         # 增加 CRD 的 Controller 文件
        new file:   internal/controller/quota_controller_test.go   # 增加 CRD 的 Controller 测试文件
        new file:   internal/controller/suite_test.go    # 增加 Controller 的 集成测试文件

增加文件解析

api/v1 目录

因我们刚刚创建了 CRD 定义，所以我们需要增加一些定义文件，就是都在这里

quota_controller.go 文件

生成了 controller 的模板文件，我们只需要去填充 Reconcile 的逻辑即可

quota_controller.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// QuotaReconciler reconciles a Quota object
type QuotaReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=resource.tutorial.controller.io,resources=quotas,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=resource.tutorial.controller.io,resources=quotas/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=resource.tutorial.controller.io,resources=quotas/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Quota object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.17.0/pkg/reconcile
func (r *QuotaReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	// TODO(user): your logic here

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *QuotaReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&resourcev1.Quota{}).
		Complete(r)
}
quota_controller_test.go

就和之前的 e2e 测试一样，生成了一个测试用例，我们需要去编写一些测试逻辑

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
var _ = Describe("Quota Controller", func() {
	Context("When reconciling a resource", func() {
		const resourceName = "test-resource"

		ctx := context.Background()

		typeNamespacedName := types.NamespacedName{
			Name:      resourceName,
			Namespace: "default", // TODO(user):Modify as needed
		}
		quota := &resourcev1.Quota{}

		BeforeEach(func() {
			By("creating the custom resource for the Kind Quota")
			err := k8sClient.Get(ctx, typeNamespacedName, quota)
			if err != nil && errors.IsNotFound(err) {
				resource := &resourcev1.Quota{
					ObjectMeta: metav1.ObjectMeta{
						Name:      resourceName,
						Namespace: "default",
					},
					// TODO(user): Specify other spec details if needed.
				}
				Expect(k8sClient.Create(ctx, resource)).To(Succeed())
			}
		})

		AfterEach(func() {
			// TODO(user): Cleanup logic after each test, like removing the resource instance.
			resource := &resourcev1.Quota{}
			err := k8sClient.Get(ctx, typeNamespacedName, resource)
			Expect(err).NotTo(HaveOccurred())

			By("Cleanup the specific resource instance Quota")
			Expect(k8sClient.Delete(ctx, resource)).To(Succeed())
		})
		It("should successfully reconcile the resource", func() {
			By("Reconciling the created resource")
			controllerReconciler := &QuotaReconciler{
				Client: k8sClient,
				Scheme: k8sClient.Scheme(),
			}

			_, err := controllerReconciler.Reconcile(ctx, reconcile.Request{
				NamespacedName: typeNamespacedName,
			})
			Expect(err).NotTo(HaveOccurred())
			// TODO(user): Add more specific assertions depending on your controller's reconciliation logic.
			// Example: If you expect a certain status condition after reconciliation, verify it here.
		})
	})
})

定义 Spec

我们看完了新增的部分，我们直接对 Spec 进行定义

quota_types.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// Quota is the Schema for the quotas API
type Quota struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   QuotaSpec   `json:"spec,omitempty"`
	Status QuotaStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// QuotaList contains a list of Quota
type QuotaList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Quota `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Quota{}, &QuotaList{})
}

包含了一些必要的实现，那么我们就来定义我们的 API，我们就简单的定义一些字段

1
2
3
4
5
6
// QuotaSpec defines the desired state of Quota
type QuotaSpec struct {

    // 这直接借鉴 Pod 中的定义
	Limits v1.ResourceList `json:"limits,omitempty" protobuf:"bytes,1,rep,name=limits,casttype=ResourceList,castkey=ResourceName"`
}

我们需要让用户来定义 CPU 和 MEM 的资源，我们直接抄袭 Pod 中的定义的部分。

我们执行 make manifests 就可以生成 crd 文件了。

1
2
$ make manifests 
/quota-limit/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

我们就可以生成 crd 文件了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── resource.tutorial.controller.io_quotas.yaml  <- 生成的文件
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml

$ cat config/crd/bases/resource.tutorial.controller.io_quotas.yaml
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.14.0
  name: quotas.resource.tutorial.controller.io
spec:
  group: resource.tutorial.controller.io
  names:
    kind: Quota
    listKind: QuotaList
    plural: quotas
    singular: quota
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: Quota is the Schema for the quotas API
        properties:
          apiVersion:
            description: |-
              APIVersion defines the versioned schema of this representation of an object.
              Servers should convert recognized schemas to the latest internal value, and
              may reject unrecognized values.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            type: string
          kind:
            description: |-
              Kind is a string value representing the REST resource this object represents.
              Servers may infer this from the endpoint the client submits requests to.
              Cannot be updated.
              In CamelCase.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            type: string
          metadata:
            type: object
          spec:
            description: QuotaSpec defines the desired state of Quota
            properties:
              limits:
                additionalProperties:
                  anyOf:
                  - type: integer
                  - type: string
                  pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                  x-kubernetes-int-or-string: true
                description: ResourceList is a set of (resource name, quantity) pairs.
                type: object
            type: object
          status:
            description: QuotaStatus defines the observed state of Quota
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

到这里我们就完成了基本的 CRD 定义，那我们可以运行一下看看

1
2
3
4
5
6
# 指定的 kubeconfig
$ export KUBECONFIG=$KUBES/k3s.config 

# 将 CRD Apply 到集群中
$ k apply -f config/crd/bases/resource.tutorial.controller.io_quotas.yaml 
customresourcedefinition.apiextensions.k8s.io/quotas.resource.tutorial.controller.io created

创建一个 CR 试试

1
2
3
4
5
6
7
8
9
$ cat limit.yaml
apiVersion: resource.tutorial.controller.io/v1
kind: Quota
metadata:
  name: test
spec:
  limits:
    cpu: "1"
    mem: "100Mi"

将其 apply 到系统里面

1
2
3
# apply
$ k apply -f limit.yaml
quota.resource.tutorial.controller.io/test-limit created

这样就可以从 kubectl 获得

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# get
$ k get quotas.resource.tutorial.controller.io test-limit -oyaml
apiVersion: resource.tutorial.controller.io/v1
kind: Quota
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"resource.tutorial.controller.io/v1","kind":"Quota","metadata":{"annotations":{},"name":"test","namespace":"default"},"spec":{"limits":{"cpu":"1","mem":"100Mi"}}}
  creationTimestamp: "2024-03-26T07:17:38Z"
  generation: 1
  name: test
  namespace: default
  resourceVersion: "25148"
  uid: a899cf34-7c44-4ea7-a557-5039f97520b6
spec:
  limits:
    cpu: "1"
    mem: 100Mi

Quota Controller & Webhook

为 Quota 创建一个 ValidatingWebhook

1
2
3
4
5
6
7
8
9
$ kubebuilder create webhook --group resource --version v1 --kind Quota --defaulting --programmatic-validation

INFO Writing kustomize manifests for you to edit... 
INFO Writing scaffold for you to edit...          
INFO api/v1/quota_webhook.go                      
INFO api/v1/quota_webhook_test.go                 
INFO api/v1/webhook_suite_test.go                 
INFO Update dependencies:

这里其实对于我们系统来说是可有可无的，但是因为 Kubebuilder 依赖于 webhook 修改 Deployment等信息，所以这里最好是用 CLI 创建一个我们 CRD 的 Validate 顺带可以帮助我们构建后面对于 POD 的 Validate Webhook。

新增文件解析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   PROJECT
        new file:   api/v1/quota_webhook.go             # 新增了 Webhook 的实现函数
        new file:   api/v1/quota_webhook_test.go
        new file:   api/v1/webhook_suite_test.go    
        modified:   api/v1/zz_generated.deepcopy.go
        modified:   cmd/main.go
        new file:   config/certmanager/certificate.yaml  # 新增了 certmanager 组件，用于处理 Webhook 的证书问题，下面会解释
        new file:   config/certmanager/kustomization.yaml
        new file:   config/certmanager/kustomizeconfig.yaml
        modified:   config/crd/kustomization.yaml
        new file:   config/crd/patches/cainjection_in_quotas.yaml
        new file:   config/crd/patches/webhook_in_quotas.yaml
        modified:   config/default/kustomization.yaml
        new file:   config/default/manager_webhook_patch.yaml  # 为 manager Deployment 增加了 webhook 的配置
        new file:   config/default/webhookcainjection_patch.yaml
        new file:   config/webhook/kustomization.yaml
        new file:   config/webhook/kustomizeconfig.yaml
        new file:   config/webhook/service.yaml              # 新增了 Service 用于 Webhook 服务
        modified:   go.mod

执行一下 make manifests 我们就能找到最终生成得了 webhook 定义 config/webhook/manifests.yaml，这里生成了2个 webhook，分别是 Mutating 和 Validating

  • Mutating: 用于修改 Quota CRD
  • Validating: 用于验证 Quota CRD
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
---
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook-service
      namespace: system
      path: /mutate-resource-tutorial-controller-io-v1-quota
  failurePolicy: Fail
  name: mquota.kb.io
  rules:
  - apiGroups:
    - resource.tutorial.controller.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - quotas
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook-service
      namespace: system
      path: /validate-resource-tutorial-controller-io-v1-quota
  failurePolicy: Fail
  name: vquota.kb.io
  rules:
  - apiGroups:
    - resource.tutorial.controller.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - quotas
  sideEffects: None

你肯定很疑惑，为什么我们没有创建 mutating 但是会有呢，因为在我们的命令中 --defaulting 就是使用 mutating 来实现的，这里功能就是用户如果创建一个 Quota CR 但是有很多值没有填写，我们可以为其填充一些默认值。
我们不需要就直接将其不生成，如下即可。

1
kubebuilder create webhook --group resource --version v1 --kind Quota --programmatic-validation

main 文件的改动

执行之后，main函数里多了一个 if 来判断是否启用了 Webhook，如果启用了，则就会去注册 Webhook 。

1
2
3
4
5
6
7
8
if os.Getenv("ENABLE_WEBHOOKS") != "false" {
    // 注册 Validate 的 Webhook
    if err = (&resourcev1.Quota{}).SetupWebhookWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create webhook", "webhook", "Quota")
        os.Exit(1)
    }
    setupLog.Info("quota webhook inited")
}

好奇的你肯定还有一个疑问，为什么路径是 /validate-resource-tutorial-controller-io-v1-quota 和 /mutate-resource-tutorial-controller-io-v1-quota，是因为 kubebuilder 想要简化这个流程，就是用了特定的规则来生成这样的 API 地址，在我们自己的代码里面是没有找到相关的实现的。这点也是 Go 社区很奇怪的设计之一。

config/default/manager_webhook_patch.yaml

新增了如下内容，其实就是给 controller 挂了一个 webhook-server-cert 让他可以访问这个 cert，这个 cert 其实 certmanager 生成的 TLS 证书，下文解释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
apiVersion: apps/v1
kind: Deployment
metadata:
  name: controller-manager
  namespace: system
spec:
  template:
    spec:
      containers:
      - name: manager
        ports:
        - containerPort: 9443
          name: webhook-server
          protocol: TCP
        volumeMounts:
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: webhook-server-cert

quota_webhook.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

// log is for logging in this package.
var quotalog = logf.Log.WithName("quota-resource")

// SetupWebhookWithManager 提供初始化 Webhook 的接口
func (r *Quota) SetupWebhookWithManager(mgr ctrl.Manager) error {
	return ctrl.NewWebhookManagedBy(mgr).
		For(r).
		Complete()
}

// TODO(user): EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!

// 这里给 kubebuilder 一个默认的注解，帮助其生成 webhook 配置
//+kubebuilder:webhook:path=/mutate-resource-tutorial-controller-io-v1-quota,mutating=true,failurePolicy=fail,sideEffects=None,groups=resource.tutorial.controller.io,resources=quotas,verbs=create;update,versions=v1,name=mquota.kb.io,admissionReviewVersions=v1

var _ webhook.Defaulter = &Quota{}

// Default implements webhook.Defaulter so a webhook will be registered for the type
// 实现 --defaulting 的功能，这里就是一个 mutating wenhook 的主题逻辑，所以上面可以看到 mutating=true
func (r *Quota) Default() {
	quotalog.Info("default", "name", r.Name)

	// TODO(user): fill in your defaulting logic.
}

// TODO(user): change verbs to "verbs=create;update;delete" if you want to enable deletion validation.
//+kubebuilder:webhook:path=/validate-resource-tutorial-controller-io-v1-quota,mutating=false,failurePolicy=fail,sideEffects=None,groups=resource.tutorial.controller.io,resources=quotas,verbs=create;update,versions=v1,name=vquota.kb.io,admissionReviewVersions=v1

// 实现 --programmatic-validation 的功能，也就一个 validating 的能力
var _ webhook.Validator = &Quota{}

// ValidateCreate implements webhook.Validator so a webhook will be registered for the type 
// 创建的函数调用这里验证
func (r *Quota) ValidateCreate() (admission.Warnings, error) {
	quotalog.Info("validate create", "name", r.Name)

	// TODO(user): fill in your validation logic upon object creation.
	return nil, nil
}

// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
// 更新这里函数
func (r *Quota) ValidateUpdate(old runtime.Object) (admission.Warnings, error) {
	quotalog.Info("validate update", "name", r.Name)

	// TODO(user): fill in your validation logic upon object update.
	return nil, nil
}

// ValidateDelete implements webhook.Validator so a webhook will be registered for the type
// 删除这里验证
func (r *Quota) ValidateDelete() (admission.Warnings, error) {
	quotalog.Info("validate delete", "name", r.Name)

	// TODO(user): fill in your validation logic upon object deletion.
	return nil, nil
}

给好奇宝宝的章节

那上文的究竟是怎么生成的呢，答案就是在 SetupWebhookWithManager 中的 Complete() 函数，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// Register webhook(s) for type
blder.registerDefaultingWebhook()
blder.registerValidatingWebhook()



// 注册 Default 函数
func (blder *WebhookBuilder) registerDefaultingWebhook() {
	mwh := blder.getDefaultingWebhook()
	if mwh != nil {
		mwh.LogConstructor = blder.logConstructor
		path := generateMutatePath(blder.gvk)

		// Checking if the path is already registered.
		// If so, just skip it.
		if !blder.isAlreadyHandled(path) {
			log.Info("Registering a mutating webhook",
				"GVK", blder.gvk,
				"path", path)
      
      // 实际上就是注册一个 mutating webhook 服务，路径就是下面生成的，mvh 就是逻辑，就是我们上面编写的 Default()
			blder.mgr.GetWebhookServer().Register(path, mwh)
		}
	}
}

func generateMutatePath(gvk schema.GroupVersionKind) string {
	return "/mutate-" + strings.ReplaceAll(gvk.Group, ".", "-") + "-" +
		gvk.Version + "-" + strings.ToLower(gvk.Kind)
}

func generateValidatePath(gvk schema.GroupVersionKind) string {
	return "/validate-" + strings.ReplaceAll(gvk.Group, ".", "-") + "-" +
		gvk.Version + "-" + strings.ToLower(gvk.Kind)
}

运行起来

直到这里，我们已经有什么东西呢

  • Quota 的 CRD
  • Quota 的 Controller
  • Quota 的 Mutating Webhook
  • Quota 的 Validating Webhook

那我们尝试把这些跑起来吧。

修改点

在运行之前，我们需要修改一些内容，不用担心并不是太难

  1. Dockefile
    1
    2
    3
    4
    5
    6
    FROM gcr.io/distroless/static:nonroot
    WORKDIR /
    COPY --from=builder /workspace/manager .
    USER 65532:65532

    ENTRYPOINT ["/manager"]
    这里的 gcr.io/distroless/static:nonroot 在国内无法拉取，我们直接换成 debain
1
FROM debain
  1. 禁用 kube-rbac-proxy sidecar

修改

1
2
3
4
5
6
7
8
9
10
11
12
# config/default/kustomization.yaml

patches:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
- path: manager_auth_proxy_patch.yaml

# 我们直接将这里注释即可

# - path: manager_auth_proxy_patch.yaml

这里刚好体现了 kustomize 的优越，我们只要注释这里就可以了

  1. 修改支持 CERTMANAGER
    还是刚刚的文件
1
2
3
4
5
6
7
# config/default/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
#- ../certmanager

# 取消注释，即可
../certmanager

看注释，其实还有很多 [CERTMANAGER] 部分都需要打开，最终效果如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# config/default/kustomization.yaml
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- path: webhookcainjection_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
# Uncomment the following replacements to add the cert-manager CA injection annotations
replacements:
  - source: # Add cert-manager annotation to ValidatingWebhookConfiguration, MutatingWebhookConfiguration and CRDs
      kind: Certificate
      group: cert-manager.io
      version: v1
      name: serving-cert # this name should match the one in certificate.yaml
      fieldPath: .metadata.namespace # namespace of the certificate CR
    targets:
      - select:
          kind: ValidatingWebhookConfiguration
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 0
          create: true
      - select:
          kind: MutatingWebhookConfiguration
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 0
          create: true
      - select:
          kind: CustomResourceDefinition
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 0
          create: true
  - source:
      kind: Certificate
      group: cert-manager.io
      version: v1
      name: serving-cert # this name should match the one in certificate.yaml
      fieldPath: .metadata.name
    targets:
      - select:
          kind: ValidatingWebhookConfiguration
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 1
          create: true
      - select:
          kind: MutatingWebhookConfiguration
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 1
          create: true
      - select:
          kind: CustomResourceDefinition
        fieldPaths:
          - .metadata.annotations.[cert-manager.io/inject-ca-from]
        options:
          delimiter: '/'
          index: 1
          create: true
  - source: # Add cert-manager annotation to the webhook Service
      kind: Service
      version: v1
      name: webhook-service
      fieldPath: .metadata.name # namespace of the service
    targets:
      - select:
          kind: Certificate
          group: cert-manager.io
          version: v1
        fieldPaths:
          - .spec.dnsNames.0
          - .spec.dnsNames.1
        options:
          delimiter: '.'
          index: 0
          create: true
  - source:
      kind: Service
      version: v1
      name: webhook-service
      fieldPath: .metadata.namespace # namespace of the service
    targets:
      - select:
          kind: Certificate
          group: cert-manager.io
          version: v1
        fieldPaths:
          - .spec.dnsNames.0
          - .spec.dnsNames.1
        options:
          delimiter: '.'
          index: 1
          create: true

为什么需要 certmanager

我们这里解释一下，我们以 isito 的 sidecar 注入为例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
$ k get mutatingwebhookconfigurations.admissionregistration.k8s.io istio-sidecar-injector -oyaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: "2024-02-29T06:38:03Z"
  generation: 3
  labels:
    app: sidecar-injector
    install.operator.istio.io/owning-resource: installed-state
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio.io/rev: default
    operator.istio.io/component: IstiodRemote
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.19.4
    release: istio
  name: istio-sidecar-injector
  resourceVersion: "9653"
  uid: f1b24001-94a4-42ad-bddf-f716a8041428
webhooks:
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZDVENDQXZHZ0F3SUJBZ0lKQUxXaE9JaW16T3NwTUEwR0NTcUdTSWIzRFFFQkN3VUFNQ0l4RGpBTUJnTlYKQkFvTUJVbHpkR2x2TVJBd0RnWURWUVFEREFkU2IyOTBJRU5CTUI0WERUSXpNRFV3TmpBNE5UYzBPRm9YRFRNegpNRFV3TXpBNE5UYzBPRm93SWpFT01Bd0dBMVVFQ2d3RlNYTjBhVzh4RURBT0JnTlZCQU1NQjFKdmIzUWdRMEV3CmdnSWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUNEd0F3Z2dJS0FvSUNBUUREZkF1OEUycUd5TFdTR1hKVm0xTXEKNDNYVlFzYkd0RW40MGhjZ2ZUeXBnbnU3NDVzVEE2blhWUGRraldnUHRzZFJ4MkJDSEU1RXdld0UxaTdwSzRiago3VjdkZmQ5aTExNjNMVms1VHNHMmN5SWtGMEc5MFczUC9MR3Q4eEI5MDdwTVZVL1B2dEdyRjRvQ3Z0Z3FNT2J0CjhOOUNwV0VOWUVPbEF0NGxXbEtZdHZGZVpaR08wSDNENGIxMmpoK1ZqWHA1V0tYamJzN0twblVTRHZRUEcyR3IKTldoUDlnTit4RE1mMm9JcFpUVVAvV2dtSC80cVRHV3IzM1lGVFNTdUQxbG1qRVcwaXFGd2xxQTIwNDFHeHZ6NwozTTdTSWhaYzhlUld5eTV1RGdyMUlNSHViaWJQSFZjTjJpNWtlOWN3WXlxRUNlQ1NGQTRoeXRITWJUcUtHZ0VFClE3a2dhanlNbUdVeGxoN0lUU3dZWWRWL1hFaDQ5dlg4M2VZc0F6Mk51SWZWRzRVYWpvS0dDaUM5cFZYa2pNSUIKOTRPN2FmYmU0SVdhcU14QUhQTEpMTE11cXU0RE9VU2MyQXV1SE5YSnJiNHluSVIwQjVOOHdjYVBGQmVEc3pDTwpkUnFTY1pwVE5uZ1M0eTBQTXRObG5HazRaTzQzK3g3YkFrdUxKR2JOWFJOdEpVdUIzTlU1NEYrbVI5MDF5UEo3CklYZXdPMjF1V3ZlQXM1czNuTXI4SHZSZ2Q4czQ3eFllTWNGWDJ3M0RmN0R4ZXVsSlFpYTNvbGFGbUk3T1JUazAKbmR5ZjE4NUIyWnFxMjdDSTk4MHV2RnBIYkJNT3FMTG5RazhndFg0WTFPNTMrcHQ0cGdtRWg1OFVucGRrc2tKaQpuZi9GTlR4U3ZSNDMxV2R1Ujhyc1J3SURBUUFCbzBJd1FEQWRCZ05WSFE0RUZnUVVtbDVLeitWMFEvOGtwUW52ClVHbzlpRXdxQ3Zjd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBT0JnTlZIUThCQWY4RUJBTUNBdVF3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dJQkFBQ1QzNFlibzh5MjhxNTdpQ2tsKzR4c2s5ck54YXRRajAwU1ZQTEpnNVBEWmRiZAorY1F0T0JrVDYyQzYwNDBxWERZQ2xsbC9mWGs5T0E1ekgxa29XVEJnYkJabkxhV1JVSDhsb0NjQ0RGekJkOFFxCjBjV3hCajRzVFJFUHVjREFySVRoek00K3NiZXIxVjJPMy95Sk1iQ0loeFhCc09UTFI5K2lZTXhtWm4yZlltTDcKZkZjOGh6RHVPT3NQUGVmQnVvbVYvNW1wNnNSTzVrRHF0UXlkZHhLZk9JU0xoeUVxV3M5SnE1RHZ5bzlkMWZDSAo0MW9ZZnJCN3hOa2c2cm9VYjA2cUlobVlWUVJpaDNlOEJBTGx5amtaS1RyNElGWHZkdTRZYmNxd042OTdRL00xCnRUeEdlbUFiczJXbS9vOExsTFFteUdWMEVJSm9ycjJGUWdzbkx3WVIvM216WndsQ2xJYjh6WmhlMlE2N0lmZm4KdUVYRWNXZ0lBSnVkNU5VRjlQK0RlOFVNaHhqaldNSnFFSU5sSWpyUDZLM2VFODRMLy9ISzA0NHM0c2dOUWw2ZgpZWS9BK0VCTy81TnBLSGpXMzNnWW9rWXcrMXNWSDdtU051MHljamtqNVA3d0JJUVo0SE5BMXZyazVaTndmajZRClQzMDh6NllFTGM3MlQ0MVVjR1A2VWxUWHRhTUpqTzBEUGI2NHMwSU5na0hpaTAxS3k2SExxZ1pjbVl3MzJZaG4KR2dLanNDOVhOQWN4dGJlc0JLVjh1RGVZY2htYkl6a3NubG9Ca283bHpib3hvdkt5ckx1ckYrWGxMaFU1bFN1OQpOcDJIL0xGOFFCYTdlRHJ1SEorenF4bDh1dzAxTHNwNzQ0ck9kTi9QNStJQXFZc3pIVWJXUndEUFBiM3IKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: istiod
      namespace: istio-system
      path: /inject/cluster/cluster2/net/network2
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rev.namespace.sidecar-injector.istio.io
  namespaceSelector:
    matchLabels:
      istio.io/deactivated: never-match
  objectSelector:
    matchLabels:
      istio.io/deactivated: never-match
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

相较于我们当前的 Webhook 定义主要有个 caBundle的区别，这个是做什么的呢？
webhook 需要有一定的安全性保证，最简单的就是 TLS了， 我们提供的一个 HTTPS 的 Webhook Server 服务，因我们的证书大多数时候是自签名的，那么我们需要告知 Apiserver 我们的服务器证书是什么，这样来保证通讯的安全（验证服务端是不是真实的）。

那么这里就需要几个问题

  1. Controller Server: 服务端证书怎么生成
  2. Webhook Server: caBundle 谁来填入
  3. 这两个证书必须是成对的

这里就利用了社区的 certmanager 来解决这个问题。

certmanager 可以帮助我们生成一个 TLS 的 组件，这个组件也很复杂，我们大概知道能做这么久好了。

Let it run

  1. 是安装 certmanager
1
2
3
4
5
6
7
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

$ k get pod -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-56d77974fd-2q7jq              1/1     Running   0          3m37s
cert-manager-cainjector-855c6869b7-c65nb   1/1     Running   0          3m37s
cert-manager-webhook-69c88dbbd7-2klv8      1/1     Running   0          3m37s
  1. 构建我们的系统
1
$ make docker-build docker-push IMG=manager:0.0.1
  1. 部署
1
2
3
4
$ make deploy IMG=manager:0.0.1
$ k get pod -n quota-limit-system
NAME                                              READY   STATUS    RESTARTS   AGE
quota-limit-controller-manager-5cf66977db-4wntp   1/1     Running   0          13s

我们尝试 apply 一个 quota，就可以看到日志

1
2
2024-03-27T03:53:52Z    INFO    quota-resource  default {"name": "test"}
2024-03-27T03:53:52Z    INFO    quota-resource  validate create {"name": "test"}

再看看我们的 validateing webhook，我们的 CA 也被正确的填充了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
$ k get validatingwebhookconfigurations.admissionregistration.k8s.io quota-limit-validating-webhook-configuration -oyaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from: quota-limit-system/quota-limit-serving-cert
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"admissionregistration.k8s.io/v1","kind":"ValidatingWebhookConfiguration","metadata":{"annotations":{"cert-manager.io/inject-ca-from":"quota-limit-system/quota-limit-serving-cert"},"labels":{"app.kubernetes.io/component":"webhook","app.kubernetes.io/created-by":"quota-limit","app.kubernetes.io/instance":"validating-webhook-configuration","app.kubernetes.io/managed-by":"kustomize","app.kubernetes.io/name":"validatingwebhookconfiguration","app.kubernetes.io/part-of":"quota-limit"},"name":"quota-limit-validating-webhook-configuration"},"webhooks":[{"admissionReviewVersions":["v1"],"clientConfig":{"service":{"name":"quota-limit-webhook-service","namespace":"quota-limit-system","path":"/validate-resource-tutorial-controller-io-v1-quota"}},"failurePolicy":"Fail","name":"vquota.kb.io","rules":[{"apiGroups":["resource.tutorial.controller.io"],"apiVersions":["v1"],"operations":["CREATE","UPDATE"],"resources":["quotas"]}],"sideEffects":"None"}]}
  creationTimestamp: "2024-03-27T03:50:28Z"
  generation: 2
  labels:
    app.kubernetes.io/component: webhook
    app.kubernetes.io/created-by: quota-limit
    app.kubernetes.io/instance: validating-webhook-configuration
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: validatingwebhookconfiguration
    app.kubernetes.io/part-of: quota-limit
  name: quota-limit-validating-webhook-configuration
  resourceVersion: "1596180"
  uid: cb31bc67-9a0c-4f66-a2a5-8fcaaf2775fc
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0lSQU1QaUt2eVU1ZFVKZFVRdDNIZ0F6eEV3RFFZSktvWklodmNOQVFFTEJRQXcKQURBZUZ3MHlOREF6TWpjd016VXdNamRhRncweU5EQTJNalV3TXpVd01qZGFNQUF3Z2dFaU1BMEdDU3FHU0liMwpEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUUMrMGp4Qkk3RGN2WkVMcTVQTjlqR1AySHRFTE4rUmtaQ0c1N2MwCkxmSkYrOW1HdmtHc3hKOHR0eFZIaU1qcUV4VlFOdWNpL0JqWWJxY0NWVUlzOG5YNE9vMk1Gc0RGeWI3VjVGZ0UKYndPdWxBSGsyQjFuWFdCL2VYL05QN2NXSS9uYXdxall5ck5IbEI1TWN6RTgzR1BDUnFWVWczM1MvbXZMNzlkQgpUdTZzTWhLS0NWemg1OXM4UmRpVktBWFZMZ2hrT3R4RU9sWXhIV2JBRHV4S1dEQVNuclVJclB1NTViTmtXWUt6CjVwcEl5VERjZDJvb1FvVkxEQ2ZMdEVOaWhKanM1TUNEVHhMa2JxQVExaGlMaVF6SjR6VEMycitZVThoSGdyZHUKb1c1Sm43T3U0c09NYzZObVNZa21ybEtLQnVhYkxsS1IwTk94cWFYOHlWd1JsUWNqQWdNQkFBR2pnYVl3Z2FNdwpEZ1lEVlIwUEFRSC9CQVFEQWdXZ01Bd0dBMVVkRXdFQi93UUNNQUF3Z1lJR0ExVWRFUUVCL3dSNE1IYUNNbkYxCmIzUmhMV3hwYldsMExYZGxZbWh2YjJzdGMyVnlkbWxqWlM1eGRXOTBZUzFzYVcxcGRDMXplWE4wWlcwdWMzWmoKZ2tCeGRXOTBZUzFzYVcxcGRDMTNaV0pvYjI5ckxYTmxjblpwWTJVdWNYVnZkR0V0YkdsdGFYUXRjM2x6ZEdWdApMbk4yWXk1amJIVnpkR1Z5TG14dlkyRnNNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJpS09rcCtiZlU5a2RpCkFlQW5yQzBCeU5hQk1BSkp1bEdTL3NvQkVNVjJReU9wYTg2d3Z4ZndSKzF0QUhyNU8zZm9PNC9Td1pCK2lqbHEKdG1rSFl6OXAxQkVKQWRZUnR3S1lNSFJzTFRWRWJQK1YraGYvUzRvZm9iNDRoSklSMUJ2VGkwK2sxdW1rNTNHSwo1U1ZsVzBiZjhKdzhQa0VhdmdsaDh4enBjWTBvaXJ6Q245VUtKRFVKRVZ2VWc4K2hrMG1KMkcwZjc2WHpYNThsCkNnZHUzZnJGT25lRjg0UjdZbEE4VmlRRmc0azVEazN0YU52Qyt5aXdSQ1c1Yk9TSGZGWHhpU3FxWHlxanU3ZlMKU1oyTXBycjBjdUQyYk9vM1hpdkZpVUpNQm5tUytUeUk4NDhjeVR5VldGckVPc3picWo2S0JrWUJRYW8wQmtkSgp3NGszRmlrLwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    service:
      name: quota-limit-webhook-service
      namespace: quota-limit-system
      path: /validate-resource-tutorial-controller-io-v1-quota
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: vquota.kb.io
  namespaceSelector: {}
  objectSelector: {}
  rules:
  - apiGroups:
    - resource.tutorial.controller.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - quotas
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

More

我们这里已经看了太多的东西，那么如果我们不使用 kubebuilder 怎么可以搞定这个事情呢？

虽然用了 kubebuilder 实际上就和用 spring 一样，虽然我们写的少了，但是理解的事情变的更多了，那么我们简化一下，如果我们根本不用 kubebuilder 怎么解决。

  1. 我们手写 CRD
    我们找社区标准的 CRD 自己编写，然后自己编写一些 CRD 的结构体给程序使用

  2. 我们自己手写 Webhook

webhook 本质就是一个 https 服务，那我们就需要写一个 TLS 服务，然后启动的时候将我们服务器的公钥回填到 webhook cr 的 caBundle 字段里面去。

其实我们就发现了，其实 kubebuilder 那些不让我们做的，才是业务的核心，其他的慢慢理解即可。

Pod ValiatingWebhook

直到现在，我们已经将环境部署完成，可以直接 validating webhook 了。那就可以来看看我们核心的本体逻辑了。

获得已有的 Limit 信息

对于 Controller 来说，我们需要知道所有的 POD 当前已经使用的 CPU 和 MEM，这里我们准备一个测试的 Deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo-web
  template:
    metadata:
      labels:
        app: echo-web
        user: "test"
    spec:
      containers:
      - name: echo-web
        image: ealen/echo-server
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 128Mi

那我们第一步显然就是感知到当前用户已用的 Limits

最简单的方式，就是我们每次都直接查询，如果开启了 Client 的 Cache 模式，这个成本也不会太高。当然我们在后面也可以进行一些优化。

1
2
3
4
podList := &v1.PodList{}
_ := r.List(ctx, podList, client.HasLabels{
    "user", "<USER_ID>",
})

通过 Pod 的信息就可以获得 Limit 信息了。那我们现在需要 ValidatingAdmissionWebhook 部分。

构建 AdmissionWebhook

我们要为 Pod 创建一个 Webhook，参考 Admission Webhook for Core Types，这里和标准的不太一样，因为社区并不支持之前的命令模式。

我们准备代码的基本模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
// 新增文件 internal/controller/pod_validating.go
package controller

import (
	"context"
	"fmt"

	v1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	logf "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
	quotav1 "tutorial.controller.io/quota-limit/api/v1"
)

type PodValidator struct {
	Client  client.Client
	Decoder *admission.Decoder
}

func (v *PodValidator) validate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
	log := logf.FromContext(ctx)
	pod, ok := obj.(*v1.Pod)
	if !ok {
		return nil, fmt.Errorf("expected a Pod but got a %T", obj)
	}

	log.Info("Validating Pod")
	key := "user"

	// 获得用户的 "user" 信息
	userName, found := pod.Labels[key]
	if !found {
		return nil, fmt.Errorf("missing label %s", key)
	}

	// 获得用户配置的 Quota
	quota := &quotav1.Quota{
		ObjectMeta: metav1.ObjectMeta{
			Name:      userName,
			Namespace: "default", // 我们假设用户的 Quota 都放在 default，和用户名同名
		},
		Spec: quotav1.QuotaSpec{
			Limits: v1.ResourceList{
				v1.ResourceCPU:    resource.MustParse("2"),
				v1.ResourceMemory: resource.MustParse("4Gi"),
			},
		},
	}
	err := v.Client.Get(ctx, client.ObjectKeyFromObject(quota), quota)
	// 没有配置 Quota，就是采用默认值
	if err != nil && !apierrors.IsNotFound(err) {
		return nil, fmt.Errorf("get quota failed")
	}

	cpuResource := resource.MustParse("0")
	memResource := resource.MustParse("0Mi")

	// 获得用户的所有的 Pod
	podList := &v1.PodList{}
	if err := v.Client.List(ctx, podList, client.MatchingLabels{
		"user": userName,
	}); err != nil {
		return nil, err
	}

	// 累计当前所有的
	for i := range podList.Items {
		pod := podList.Items[i]
		for j := range pod.Spec.Containers {
			container := pod.Spec.Containers[j]
			cpu := container.Resources.Limits.Cpu().DeepCopy()
			cpuResource.Add(cpu)
			mem := container.Resources.Limits.Memory().DeepCopy()
			memResource.Add(mem)
		}
	}

	if cpuResource.Cmp(quota.Spec.Limits.Cpu().DeepCopy()) < 0 ||
		memResource.Cmp(quota.Spec.Limits.Memory().DeepCopy()) < 0 {
		return nil, fmt.Errorf("user %s, limit out of quota", userName)
	}

	return nil, nil
}

func (v *PodValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (warnings admission.Warnings, err error) {
	return v.validate(ctx, obj);
}

func (v *PodValidator) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (warnings admission.Warnings, err error) {
	return v.validate(ctx, newObj);
}

func (v *PodValidator) ValidateDelete(ctx context.Context, obj runtime.Object) (warnings admission.Warnings, err error) {
	return v.validate(ctx, obj);
}

注册函数 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// main.go 中
	if os.Getenv("ENABLE_WEBHOOKS") != "false" {
		if err = (&resourcev1.Quota{}).SetupWebhookWithManager(mgr); err != nil {
			setupLog.Error(err, "unable to create webhook", "webhook", "Quota")
			os.Exit(1)
		}

		if err := builder.WebhookManagedBy(mgr).
			For(&v1.Pod{}).
			WithValidator(&controller.PodValidator{
				Client:  mgr.GetClient(),
				Decoder: admission.NewDecoder(mgr.GetScheme()),
			}).
			Complete(); err != nil {
			os.Exit(1)
		}
	}

给生成器标记生成 Webhook，这里的路径名是特殊的规则，上文已经提及。

1
2
3
4
5
6
// +kubebuilder:webhook:path=/validate--v1-pod,mutating=false,failurePolicy=fail,groups="",resources=pods,verbs=create;update,versions=v1,name=vpod.kb.io,admissionReviewVersions=v1,sideEffects=None

// Quota is the Schema for the quotas API
type Quota struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

最终会生成如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-webhook-configuration
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook-service
      namespace: system
      path: /validate--v1-pod
  failurePolicy: Fail
  name: vpod.kb.io
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
  sideEffects: None

构建镜像

1
$ make docker-build docker-push IMG=manager:0.0.2

部署

  1. 部署应用

    1
    $ make deploy IMG=manager:0.0.2

  2. 检查 POD 运行

1
2
3
$ k get pod -n quota-limit-system                                             
NAME                                              READY   STATUS    RESTARTS   AGE
quota-limit-controller-manager-756d65d46f-6cqx6   1/1     Running   0          6m2s

Demo Show

我们输入一个 demo

1
2
$ k apply -f example/limit.yaml  
quota.resource.tutorial.controller.io/test created

我们就可以从日志中看到

1
2
2024-03-26T09:59:33Z    INFO    quota-resource  default {"name": "test"}
2024-03-26T09:59:33Z    INFO    quota-resource  validate create {"name": "test"}

这里执行了 default 和 validate 2 个 hook 逻辑

那我们继续创建其他 POD

1
2024-03-26T10:08:00Z    INFO    admission       Validating Pod  {"webhookGroup": "", "webhookKind": "Pod", "Pod": {"name":"echo-web-956d664c9-4t486","namespace":"default"}, "namespace": "default", "name": "echo-web-956d664c9-4t486", "resource": {"group":"","version":"v1","resource":"pods"}, "user": "system:serviceaccount:kube-system:replicaset-controller", "requestID": "2b9afee4-618e-4680-844a-5a96d5d035a7"}

我们就可以从日志发现这个逻辑了，这样就满足了，我们需求了。

遗留

这里其实有个很糟糕的遗留问题，我们在 apply Pod 的 Validating Webhook 的时候，因为很有可能我们自己的 Controller 也没有启动起来，但是这个 Controller 无法启动，这就变成了一个死循环了，其实更好的方式是应该，一开始情况下 failurePolicy 应该是 Ingrone 忽略错误，当我们的组件启动之后，再把这个改成 Fail 即可。

1
2
3
4
5
6
7
8
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: webhook-service
      namespace: system
      path: /validate--v1-pod
  failurePolicy: Fail

参考