本文参考:https://zhuanlan.zhihu.com/p/404764407

仅做记录

1、介绍

Webhook就是一种HTTP回调,用于在某种情况下执行某些动作,Webhook不是K8S独有的,很多场景下都可以进行Webhook,比如在提交完代码后调用一个Webhook自动构建docker镜像。

Admission Webhook 是 api-server 对外提供的一个扩展能力,api-server 作为 kubernetes 的核心,几乎所有组件都需要跟他打交道,基本可以说掌控了 k8s 的 api-server,你就可以控制 k8s 的行为。

在早期的版本 api-server 并没有提供 admissionresgistration 的能力(v1.9之前),当我们要对 k8s 进行控制的时候,只能重新编译 api-server。比如你想阻止某个控制器的行为,或拦截某个控制器的资源修改。admission webhook 就是提供了这样的能力,比如你希望某个特定 label 标签的 pod 再创建的时候都注入 sidercar,或者阻止不合规的资源。

(1)K8S中提供了自定义资源类型自定义控制器来扩展功能:

Admission Webhook 包涵两种 CRD:mutatingwebhookconfigurationvalidatingwebhookconfiguration

下面是一个 mutatingwebhookconfiguration 的CRD文件:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-test.shikanon.com
webhooks:
- admissionReviewVersions: # admissionReviewVersions 请求的版本
  - v1beta1
  - v1
  clientConfig: # 客户端配置
    caBundle: # ca证书
    service: # 调用服务相关配置,这里是一个k8s的service,访问地址是<name>.<namespace>.svc:<port>/<path>
      name: mutating-test 
      namespace: testing-tools
      path: /mutation-deployment
      port: 8000
  failurePolicy: Ignore # 调用失败策略,Ignore为忽略错误, failed表示admission会处理错误
  matchPolicy: Exact
  name: mutating-test.shikanon.com # webhook名称
  namespaceSelector: {} # 命名空间过滤条件
  objectSelector: # 对象过滤条件
    matchExpressions:
    - key: mutating-test-webhook
      operator: In
      values:
      - enabled
      - "true"
  # reinvocationPolicy表示再调度策略,因为webhook本身没有顺序性,因此每个修改后可能又被其他webhook修改,所以提供
  # 一个策略表示是否需要被多次调用,Never 表示只会调度一次,IfNeeded 表示资源被修改后会再调度这个webhook
  reinvocationPolicy: Never 
  rules: # 规则
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments
    scope: '*' # 匹配范围,"*" 匹配所有资源,但不包括子资源,"*/*" 匹配所有资源,包括子资源
  sideEffects: None # 这个表示webhook是否存在副作用,主要针对 dryRun 的请求
  timeoutSeconds: 30

(2)还提供了动态准入控制,其实就是通过Webhook来实现准入控制,分为两种:

Admission Webhook有哪些使用场景?如下

  • 在资源持久化到ETCD之前进行修改(Mutating Webhook),比如增加init Container或者sidecar Container

  • 在资源持久化到ETCD之前进行校验(Validating Webhook),不满足条件的资源直接拒绝并给出相应信息

istio就是通过 mutating webhooks 来自动将Envoy这个 sidecar 容器注入到 Pod 中去的。

上面提到K8S的动态准入控制是通过Webhook来实现的,那么它到底是在哪个环节执行的?请看下图

从图中可以看到,先执行的是Mutating Webhook,它可以对资源进行修改,然后执行的是Validating Webhook,它可以拒绝或者接受请求,但是它不能修改请求。

api-server 通过读取 mutatingwebhookconfiguration 和 validatingwebhookconfiguration 的 CR 文件的目标地址,然后回调用户自定义的服务。

                                            ┌──────────────────────────────────┐
             ┌─────────────────┐            │                                  │
    apply    │                 │    read    │  validatingwebhookconfiguration  │
────────────►│    api-server   │◄───────────┤                                  │
             │                 │            │  mutatingwebhookconfiguration    │
             └────────┬────────┘            │                                  │
                      │                     └──────────────────────────────────┘
                      │
                      │  回调
                      │
                      │
             ┌────────▼────────┐
             │                 │
             │  webhookservice │
             │                 │
             └─────────────────┘

工作原理和步骤

  • 用户发送一个 Kubernetes 资源创建、更新或删除的请求;

  • 请求到达 Kubernetes API Server;

  • 经过认证、授权;

  • Kubernetes API Server 将请求发送给Mutating Admission Controller 进行处理;

  • Admission Webhook 对请求进行审查和修改,并返回处理结果给 Admission Controller;

  • Kubernetes API Server 将请求发送给Validating Admission Controller 进行处理;

  • Admission Webhook 对请求进行校验,并返回处理结果给 Admission Controller;

  • Admission Controller 根据 Admission Webhook 的处理结果决定是否允许请求通过。

2、Admission Webhook列表

根据k8s官网,列举部分准入控制器:

名字

类别

作用

AlwaysAdmit

验证

该准入控制器允许所有的 Pod 进入集群。此插件已被弃用,因其行为与没有准入控制器一样。

AlwaysDeny

验证

拒绝所有的请求。由于它没有实际意义,已被弃用

CertificateApproval

验证

此准入控制器获取审批 CertificateSigningRequest 资源的请求并执行额外的鉴权检查, 以确保针对设置了 spec.signerName 的 CertificateSigningRequest 资源而言, 审批请求的用户有权限对证书请求执行 审批 操作。

CertificateSigning

验证

此准入控制器监视对 CertificateSigningRequest 资源的 status.certificate 字段的更新请求, 并执行额外的鉴权检查,以确保针对设置了 spec.signerName 的 CertificateSigningRequest 资源而言, 签发证书的用户有权限对证书请求执行 签发 操作。

CertificateSubjectRestriction

验证

此准入控制器监视 spec.signerName 被设置为 kubernetes.io/kube-apiserver-client 的 CertificateSigningRequest 资源创建请求,并拒绝所有将 “group”(或 “organization attribute”) 设置为 system:masters 的请求。

DefaultIngressClass

变更

该准入控制器监测没有请求任何特定 Ingress 类的 Ingress 对象创建请求,并自动向其添加默认 Ingress 类。 这样,没有任何特殊 Ingress 类需求的用户根本不需要关心它们,他们将被设置为默认 Ingress 类。

DefaultStorageClass

变更

此准入控制器监测没有请求任何特定存储类的 PersistentVolumeClaim 对象的创建请求, 并自动向其添加默认存储类。 这样,没有任何特殊存储类需求的用户根本不需要关心它们,它们将被设置为使用默认存储类。

PodTopologyLabels

变更

PodTopologyLabels 准入控制器变更所有绑定到节点的 pod 的 pods/binding 子资源, 添加与所绑定节点相匹配的拓扑标签。 这使得节点拓扑标签可以作为 Pod 标签使用,并且可以通过 Downward API 提供给运行中的容器。 由于此控制器而可用的标签是 topology.kubernetes.io/regiontopology.kubernetes.io/zone 标签。

ExtendedResourceToleration

变更

此插件有助于创建带有扩展资源的专用节点。 如果运维人员想要创建带有扩展资源(如 GPU、FPGA 等)的专用节点,他们应该以扩展资源名称作为键名, 为节点设置污点。 如果启用了此准入控制器,会将此类污点的容忍度自动添加到请求扩展资源的 Pod 中, 用户不必再手动添加这些容忍度。

ImagePolicyWebhook

验证

ImagePolicyWebhook 准入控制器允许使用后端 Webhook 做出准入决策。

NamespaceAutoProvision

变更

此准入控制器会检查针对名字空间域资源的所有传入请求,并检查所引用的名字空间是否确实存在。 如果找不到所引用的名字空间,控制器将创建一个名字空间。 此准入控制器对于不想要求名字空间必须先创建后使用的集群部署很有用。

NamespaceExists

验证

此准入控制器检查针对名字空间作用域的资源(除 Namespace 自身)的所有请求。 如果请求引用的名字空间不存在,则拒绝该请求。

NodeRestriction

验证

该准入控制器限制了某 kubelet 可以修改的 NodePod 对象。 为了受到这个准入控制器的限制,kubelet 必须使用在 system:nodes 组中的凭证, 并使用 system:node:<nodeName> 形式的用户名。 这样,kubelet 只可修改自己的 Node API 对象,只能修改绑定到自身节点的 Pod 对象。

3、开发K8S Webhook最佳实践

我们以一个简单的Webhook作为例子,该Webhook会在创建Deployment资源的时候检查它是否有相应的标签,如果没有的话,则加上(Mutating Webhook),然后在检验它是否有相应的标签(Validating Webhook),有则创建该Deployment,否则拒绝并给出相应错误提示。

K8S中Webhook的调用原理为首先向K8S集群中注册一个Admission Webhook(Validating / Mutating),所谓注册是向K8S集群注册一个地址,而实际Webhook服务可能跑在Pod里,也可能跑在开发机上;当创建资源的时候会调用这些Webhook进行修改或验证,最后持久化到ETCD中。

3.1、检查是否开启了动态准入控制

查看APIServer是否开启了MutatingAdmissionWebhook和
ValidatingAdmissionWebhook

# 获取apiserver pod名字
apiserver_pod_name=`kubectl get --no-headers=true po -n kube-system | grep kube-apiserver | awk '{ print $1 }'`
# 查看api server的启动参数plugin
kubectl get po $apiserver_pod_name -n kube-system -o yaml | grep plugin

如果输出如下,说明已经开启

- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

3.2、webhook简单实例

https://github.com/scriptwang/admission-webhook-example

文件列表如下

|-- Dockerfile
|-- LICENSE
|-- README.md
|-- build
|-- debug               # debug相关K8S配置
|   |-- mutatingwebhook.yaml
|   |-- service.yaml
|   |-- sleep-no-validation.yaml
|   |-- sleep-with-labels.yaml
|   |-- sleep.yaml
|   |-- sshportforward.sh
|   |-- validatingwebhook.yaml
|   |-- webhook-create-signed-cert.sh
|   `-- webhook-patch-ca-bundle.sh
|-- deployment           # 部署相关K8S配置
|   |-- deployment.yaml
|   |-- mutatingwebhook.yaml
|   |-- rbac.yaml
|   |-- service.yaml
|   |-- sleep-no-validation.yaml
|   |-- sleep-with-labels.yaml
|   |-- sleep.yaml
|   |-- validatingwebhook.yaml
|   |-- webhook-create-signed-cert.sh
|   `-- webhook-patch-ca-bundle.sh
|-- go.mod
|-- go.sum
|-- main.go     # webhook核心文件,启动server,监听端口
|-- mod.sh
|-- webhook.go  # webhook核心文件,处理业务逻辑

其中main.go和webhook.go是整个webhook的核心,前者用于启动Server,监听端口,后者用于实现核心业务逻辑。

(1)webhook.go

其核心在serve方法,根据传进来的path判断是mutate还是validate,然后执行相应的操作,这个path是自己在MutatingWebhookConfiguration或者ValidatingWebhookConfiguration中定义的

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"strings"
	"time"

	"github.com/golang/glog"
	"k8s.io/api/admission/v1beta1"
	admissionregistrationv1beta1 "k8s.io/api/admissionregistration/v1beta1"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/serializer"
	v1 "k8s.io/kubernetes/pkg/apis/core/v1"
)

var (
	runtimeScheme = runtime.NewScheme()
	codecs        = serializer.NewCodecFactory(runtimeScheme)
	deserializer  = codecs.UniversalDeserializer()

	// (https://github.com/kubernetes/kubernetes/issues/57982)
	defaulter = runtime.ObjectDefaulter(runtimeScheme)
)

var (
	ignoredNamespaces = []string{
		metav1.NamespaceSystem,
		metav1.NamespacePublic,
	}
	requiredLabels = []string{
		nameLabel,
		instanceLabel,
		versionLabel,
		componentLabel,
		partOfLabel,
		managedByLabel,
	}
	addLabels = map[string]string{
		nameLabel:      NA,
		instanceLabel:  NA,
		versionLabel:   NA,
		componentLabel: NA,
		partOfLabel:    NA,
		managedByLabel: NA,
	}
)

const (
	admissionWebhookAnnotationValidateKey = "admission-webhook-example.qikqiak.com/validate"
	admissionWebhookAnnotationMutateKey   = "admission-webhook-example.qikqiak.com/mutate"
	admissionWebhookAnnotationStatusKey   = "admission-webhook-example.qikqiak.com/status"

	nameLabel      = "app.kubernetes.io/name"
	instanceLabel  = "app.kubernetes.io/instance"
	versionLabel   = "app.kubernetes.io/version"
	componentLabel = "app.kubernetes.io/component"
	partOfLabel    = "app.kubernetes.io/part-of"
	managedByLabel = "app.kubernetes.io/managed-by"

	NA = "not_available"
)

type WebhookServer struct {
	server *http.Server
}

// Webhook Server parameters
type WhSvrParameters struct {
	port           int    // webhook server port
	certFile       string // path to the x509 certificate for https
	keyFile        string // path to the x509 private key matching `CertFile`
	sidecarCfgFile string // path to sidecar injector configuration file
}

type patchOperation struct {
	Op    string      `json:"op"`
	Path  string      `json:"path"`
	Value interface{} `json:"value,omitempty"`
}

func init() {
	_ = corev1.AddToScheme(runtimeScheme)
	_ = admissionregistrationv1beta1.AddToScheme(runtimeScheme)
	// defaulting with webhooks:
	// https://github.com/kubernetes/kubernetes/issues/57982
	_ = v1.AddToScheme(runtimeScheme)
}

func admissionRequired(ignoredList []string, admissionAnnotationKey string, metadata *metav1.ObjectMeta) bool {
	// skip special kubernetes system namespaces
	for _, namespace := range ignoredList {
		if metadata.Namespace == namespace {
			glog.Infof("Skip validation for %v for it's in special namespace:%v", metadata.Name, metadata.Namespace)
			return false
		}
	}

	annotations := metadata.GetAnnotations()
	if annotations == nil {
		annotations = map[string]string{}
	}

	var required bool
	switch strings.ToLower(annotations[admissionAnnotationKey]) {
	default:
		required = true
	case "n", "no", "false", "off":
		required = false
	}
	return required
}

func mutationRequired(ignoredList []string, metadata *metav1.ObjectMeta) bool {
	required := admissionRequired(ignoredList, admissionWebhookAnnotationMutateKey, metadata)
	annotations := metadata.GetAnnotations()
	if annotations == nil {
		annotations = map[string]string{}
	}
	status := annotations[admissionWebhookAnnotationStatusKey]

	if strings.ToLower(status) == "mutated" {
		required = false
	}

	glog.Infof("Mutation policy for %v/%v: required:%v", metadata.Namespace, metadata.Name, required)
	return required
}

func validationRequired(ignoredList []string, metadata *metav1.ObjectMeta) bool {
	required := admissionRequired(ignoredList, admissionWebhookAnnotationValidateKey, metadata)
	glog.Infof("Validation policy for %v/%v: required:%v", metadata.Namespace, metadata.Name, required)
	return required
}

func updateAnnotation(target map[string]string, added map[string]string) (patch []patchOperation) {
	for key, value := range added {
		if target == nil || target[key] == "" {
			target = map[string]string{}
			patch = append(patch, patchOperation{
				Op:   "add",
				Path: "/metadata/annotations",
				Value: map[string]string{
					key: value,
				},
			})
		} else {
			patch = append(patch, patchOperation{
				Op:    "replace",
				Path:  "/metadata/annotations/" + key,
				Value: value,
			})
		}
	}
	return patch
}

func updateLabels(target map[string]string, added map[string]string) (patch []patchOperation) {
	values := make(map[string]string)
	for key, value := range added {
		if target == nil || target[key] == "" {
			values[key] = value
		}
	}
	patch = append(patch, patchOperation{
		Op:    "add",
		Path:  "/metadata/labels",
		Value: values,
	})
	return patch
}

func createPatch(availableAnnotations map[string]string, annotations map[string]string, availableLabels map[string]string, labels map[string]string) ([]byte, error) {
	var patch []patchOperation

	patch = append(patch, updateAnnotation(availableAnnotations, annotations)...)
	patch = append(patch, updateLabels(availableLabels, labels)...)

	return json.Marshal(patch)
}

// validate deployments and services
func (whsvr *WebhookServer) validate(ar *v1beta1.AdmissionReview, log *bytes.Buffer) *v1beta1.AdmissionResponse {
	req := ar.Request
	var (
		availableLabels                 map[string]string
		objectMeta                      *metav1.ObjectMeta
		resourceNamespace, resourceName string
	)

	log.WriteString(fmt.Sprintf("\n======begin Admission for Namespace=[%v], Kind=[%v], Name=[%v]======", req.Namespace, req.Kind.Kind, req.Name))

	switch req.Kind.Kind {
	case "Deployment":
		var deployment appsv1.Deployment
		if err := json.Unmarshal(req.Object.Raw, &deployment); err != nil {
			log.WriteString(fmt.Sprintf("\nCould not unmarshal raw object: %v", err))
			glog.Errorf(log.String())
			return &v1beta1.AdmissionResponse{
				Result: &metav1.Status{
					Message: err.Error(),
				},
			}
		}
		resourceName, resourceNamespace, objectMeta = deployment.Name, deployment.Namespace, &deployment.ObjectMeta
		availableLabels = deployment.Labels
	case "Service":
		var service corev1.Service
		if err := json.Unmarshal(req.Object.Raw, &service); err != nil {
			log.WriteString(fmt.Sprintf("\nCould not unmarshal raw object: %v", err))
			glog.Errorf(log.String())
			return &v1beta1.AdmissionResponse{
				Result: &metav1.Status{
					Message: err.Error(),
				},
			}
		}
		resourceName, resourceNamespace, objectMeta = service.Name, service.Namespace, &service.ObjectMeta
		availableLabels = service.Labels
	//其他不支持的类型
	default:
		msg := fmt.Sprintf("\nNot support for this Kind of resource  %v", req.Kind.Kind)
		log.WriteString(msg)
		return &v1beta1.AdmissionResponse{
			Result: &metav1.Status{
				Message: msg,
			},
		}
	}

	if !validationRequired(ignoredNamespaces, objectMeta) {
		log.WriteString(fmt.Sprintf("Skipping validation for %s/%s due to policy check", resourceNamespace, resourceName))
		return &v1beta1.AdmissionResponse{
			Allowed: true,
		}
	}

	allowed := true
	var result *metav1.Status
	log.WriteString(fmt.Sprintf("available labels: %s ", availableLabels))
	log.WriteString(fmt.Sprintf("required labels: %s", requiredLabels))
	for _, rl := range requiredLabels {
		if _, ok := availableLabels[rl]; !ok {
			allowed = false
			result = &metav1.Status{
				Reason: "required labels are not set",
			}
			break
		}
	}

	return &v1beta1.AdmissionResponse{
		Allowed: allowed,
		Result:  result,
	}
}

// main mutation process
func (whsvr *WebhookServer) mutate(ar *v1beta1.AdmissionReview, log *bytes.Buffer) *v1beta1.AdmissionResponse {
	req := ar.Request
	var (
		availableLabels, availableAnnotations map[string]string
		objectMeta                            *metav1.ObjectMeta
		resourceNamespace, resourceName       string
	)

	log.WriteString(fmt.Sprintf("\n======begin Admission for Namespace=[%v], Kind=[%v], Name=[%v]======", req.Namespace, req.Kind.Kind, req.Name))
	log.WriteString("\n>>>>>>" + req.Kind.Kind)

	switch req.Kind.Kind {
	case "Deployment":
		var deployment appsv1.Deployment
		if err := json.Unmarshal(req.Object.Raw, &deployment); err != nil {
			log.WriteString(fmt.Sprintf("\nCould not unmarshal raw object: %v", err))
			glog.Errorf(log.String())
			return &v1beta1.AdmissionResponse{
				Result: &metav1.Status{
					Message: err.Error(),
				},
			}
		}
		resourceName, resourceNamespace, objectMeta = deployment.Name, deployment.Namespace, &deployment.ObjectMeta
		availableLabels = deployment.Labels
	case "Service":
		var service corev1.Service
		if err := json.Unmarshal(req.Object.Raw, &service); err != nil {
			log.WriteString(fmt.Sprintf("\nCould not unmarshal raw object: %v", err))
			glog.Errorf(log.String())
			return &v1beta1.AdmissionResponse{
				Result: &metav1.Status{
					Message: err.Error(),
				},
			}
		}
		resourceName, resourceNamespace, objectMeta = service.Name, service.Namespace, &service.ObjectMeta
		availableLabels = service.Labels
	//其他不支持的类型
	default:
		msg := fmt.Sprintf("\nNot support for this Kind of resource  %v", req.Kind.Kind)
		log.WriteString(msg)
		return &v1beta1.AdmissionResponse{
			Result: &metav1.Status{
				Message: msg,
			},
		}
	}

	if !mutationRequired(ignoredNamespaces, objectMeta) {
		log.WriteString(fmt.Sprintf("Skipping validation for %s/%s due to policy check", resourceNamespace, resourceName))
		return &v1beta1.AdmissionResponse{
			Allowed: true,
		}
	}

	annotations := map[string]string{admissionWebhookAnnotationStatusKey: "mutated"}
	patchBytes, err := createPatch(availableAnnotations, annotations, availableLabels, addLabels)
	if err != nil {
		return &v1beta1.AdmissionResponse{
			Result: &metav1.Status{
				Message: err.Error(),
			},
		}
	}

	log.WriteString(fmt.Sprintf("AdmissionResponse: patch=%v\n", string(patchBytes)))
	return &v1beta1.AdmissionResponse{
		Allowed: true,
		Patch:   patchBytes,
		PatchType: func() *v1beta1.PatchType {
			pt := v1beta1.PatchTypeJSONPatch
			return &pt
		}(),
	}
}

// Serve method for webhook server
func (whsvr *WebhookServer) serve(w http.ResponseWriter, r *http.Request) {
	//记录日志
	var log bytes.Buffer

	//读取从ApiServer过来的数据放到body
	var body []byte
	if r.Body != nil {
		if data, err := ioutil.ReadAll(r.Body); err == nil {
			body = data
		}
	}
	if len(body) == 0 {
		log.WriteString("empty body")
		glog.Info(log.String())
		//返回状态码400
		//如果在Apiserver调用此Webhook返回是400,说明APIServer自己传过来的数据是空
		http.Error(w, log.String(), http.StatusBadRequest)
		return
	}

	// verify the content type is accurate
	contentType := r.Header.Get("Content-Type")
	if contentType != "application/json" {
		log.WriteString(fmt.Sprintf("Content-Type=%s, expect `application/json`", contentType))
		glog.Errorf(log.String())
		//如果在Apiserver调用此Webhook返回是415,说明APIServer自己传过来的数据不是json格式,处理不了
		http.Error(w, log.String(), http.StatusUnsupportedMediaType)
		return
	}

	var admissionResponse *v1beta1.AdmissionResponse
	ar := v1beta1.AdmissionReview{}
	if _, _, err := deserializer.Decode(body, nil, &ar); err != nil {
		//组装错误信息
		log.WriteString(fmt.Sprintf("\nCan't decode body,error info is :  %s", err.Error()))
		glog.Errorln(log.String())
		//返回错误信息,形式表现为资源创建会失败,
		admissionResponse = &v1beta1.AdmissionResponse{
			Result: &metav1.Status{
				Message: log.String(),
			},
		}
	} else {
		fmt.Println(r.URL.Path)
		if r.URL.Path == "/mutate" {
			admissionResponse = whsvr.mutate(&ar, &log)
		} else if r.URL.Path == "/validate" {
			admissionResponse = whsvr.validate(&ar, &log)
		}
	}

	admissionReview := v1beta1.AdmissionReview{}
	if admissionResponse != nil {
		admissionReview.Response = admissionResponse
		if ar.Request != nil {
			admissionReview.Response.UID = ar.Request.UID
		}
	}

	resp, err := json.Marshal(admissionReview)
	if err != nil {
		log.WriteString(fmt.Sprintf("\nCan't encode response: %v", err))
		http.Error(w, log.String(), http.StatusInternalServerError)
	}
	glog.Infof("Ready to write reponse ...")
	if _, err := w.Write(resp); err != nil {
		log.WriteString(fmt.Sprintf("\nCan't write response: %v", err))
		http.Error(w, log.String(), http.StatusInternalServerError)
	}

	log.WriteString("\n======ended Admission already writed to reponse======")
	//东八区时间
	datetime := time.Now().In(time.FixedZone("GMT", 8*3600)).Format("2006-01-02 15:04:05")
	//最后打印日志
	glog.Infof(datetime + " " + log.String())
}

(2)main.go

启动服务,监听在443端口

package main

import (
	"context"
	"crypto/tls"
	"flag"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"syscall"

	"github.com/golang/glog"
)

func main() {
	var parameters WhSvrParameters

	// get command line parameters
	flag.IntVar(&parameters.port, "port", 443, "Webhook server port.")
	flag.StringVar(&parameters.certFile, "tlsCertFile", "/etc/webhook/certs/cert.pem", "File containing the x509 Certificate for HTTPS.")
	flag.StringVar(&parameters.keyFile, "tlsKeyFile", "/etc/webhook/certs/key.pem", "File containing the x509 private key to --tlsCertFile.")
	flag.Parse()

	pair, err := tls.LoadX509KeyPair(parameters.certFile, parameters.keyFile)
	if err != nil {
		glog.Errorf("Failed to load key pair: %v", err)
	}

	whsvr := &WebhookServer{
		server: &http.Server{
			Addr:      fmt.Sprintf(":%v", parameters.port),
			TLSConfig: &tls.Config{Certificates: []tls.Certificate{pair}},
		},
	}

	// define http server and server handler
	mux := http.NewServeMux()
	mux.HandleFunc("/mutate", whsvr.serve)
	mux.HandleFunc("/validate", whsvr.serve)
	whsvr.server.Handler = mux

	// start webhook server in new routine
	go func() {
		if err := whsvr.server.ListenAndServeTLS("", ""); err != nil {
			glog.Errorf("Failed to listen and serve webhook server: %v", err)
		}
	}()

	glog.Info("Server started")

	// listening OS shutdown singal
	signalChan := make(chan os.Signal, 1)
	signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
	<-signalChan

	glog.Infof("Got OS shutdown signal, shutting down webhook server gracefully...")
	whsvr.server.Shutdown(context.Background())
}

3.3、部署

所谓部署,是将webhook服务编译好并创建镜像,然后推送到远程仓库,再将这个镜像部署成Pod,当然镜像这里已经准备好了,不需要创建,这里想表达的是先部署起来看看能不能跑,能跑在说后面的创建镜像和debug等

(1)创建RBAC

由于我们的webhook会对资源进行修改,所以需要单独给一个ServiceAccount,在K8S集群中直接创建即可

kubectl apply -f deployment/rbac.yaml

(2)证书认证

K8S集群默认是HTTPS通信的,所以APiserver调用webhook的过程也是HTTPS的,所以需要进行证书认证,证书认证相当于是给Service的域名进行认证(Service后面会创建),将Service域名放到认证请求server.csr文件中,然后创建一个K8S证书签署请求资源CertificateSigningRequest,APIServer签署该证书后生成server-cert.pem,再将最初创建的私钥server-key.pem和签署好的证书server-cert.pem放到Secret中供Deployment调用,详细过程看脚本
webhook-create-signed-cert.sh

认证很简单,执行该脚本即可,会创建一个名为
admission-webhook-example-certs的Secret

./deployment/webhook-create-signed-cert.sh

这一步顺便把Service创建了,因为证书是给该Service的域名颁发的

kubectl apply -f deployment/service.yaml

(3)部署deployment

看一下Deployment的编排文件,serviceAccount和secret依次是上面两步创建的

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook-example-deployment
  labels:
    app: admission-webhook-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-webhook-example
  template:
    metadata:
      labels:
        app: admission-webhook-example
    spec:
      # 之前在RBAC中创建的serviceAccount
      serviceAccount: admission-webhook-example-sa
      containers:
        - name: admission-webhook-example
          # 该镜像已经存在
          image: kimoqi/admission-webhook-example:v1
          imagePullPolicy: Always
          args:
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=4
            - 2>&1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            # 第二步中创建的Secret,用于证书认证
            secretName: admission-webhook-example-certs

部署Deployment

kubectl apply -f deployment/deployment.yaml

稍等片刻如果有类似如下输出说明Pod已经运行

kubectl get po 
NAME                                       READY   STATUS 
admission-webhook-example-deployment-xxx   1/1     Running

可以新开一个窗口查看对应日志配合验证

(4)部署validatingWebhook

查看编排文件
deployment/validatingwebhook.yaml,里面有一个占位符${CA_BUNDLE}

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: required-labels.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/validate"
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

这个是什么呢?webhook是APIServer调用的,此时APIServer相当于是一个客服端,webhook是一个服务端,可以对比下平时上网,打开https网站时是谁在验证域名的证书?是内置在浏览器里面的根证书在做验证,所以这里的CA_BUNDLE就类似于APIServer调用webhook的根证书,它去验证webhook证书。

所以先填充这个CA_BUNDLE后再执行

# 填充占位符
cat deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > /tmp/validatingwebhook.yaml

# 部署
kubectl apply -f /tmp/validatingwebhook.yaml

(5)部署MutatingWebhook

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: mutating-example.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/mutate"
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

部署流程和ValidatingWebhook一致,需要注意的是validate的逻辑上没有对应的标签拒绝创建,而mutate的逻辑没有对应的标签会加上对应的标签,而且执行顺序是先执行mutate再执行validate,所以有以下情况

  • 只部署ValidatingWebhook:没有对应标签的资源会被拒绝创建

  • 只部署MutatingWebhook:没有对应标签的资源会加上对应标签,然后成功创建

  • 两者都部署:没有对应标签的资源会加上对应标签,也会通过ValidatingWebhook的验证,最终成功创建