k8s 定制开发(kubebuilder)

curl -LO "https://dl.k8s.io/release/v1.23.0/bin/windows/amd64/kubectl.exe"

$ kubectl.exe get ns
NAME              STATUS   AGE
default           Active   111d
kube-node-lease   Active   111d
kube-public       Active   111d
kube-system       Active   111d
mpaas             Active   30d

curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)
chmod +x kubebuilder && mv kubebuilder /usr/local/bin/

# 确保Proxy已经设置ok
export GOPROXY=https://goproxy.io,direct
# clone kubebuilder镜像仓库, 注意我直接使用的master
git clone git@github.com:kubernetes-sigs/kubebuilder.git
# 注意 提前搞定你的make命令安装
cd kubebuilder
make build
# 编译完成后, kubebuilder的可执行文件在 当前目录的bin下

$ kubebuilder -h
CLI tool for building Kubernetes extensions and tools.

Usage:
  kubebuilder [flags]
  kubebuilder [command]

Examples:
The first step is to initialize your project:
    kubebuilder init [--plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]]

<PLUGIN KEYS> is a comma-separated list of plugin keys from the following table     
and <PROJECT VERSION> a supported project version for these plugins.

                        Plugin keys | Supported project versions
------------------------------------+----------------------------
          base.go.kubebuilder.io/v3 |                          3
   declarative.go.kubebuilder.io/v1 |                       2, 3
               go.kubebuilder.io/v2 |                       2, 3
               go.kubebuilder.io/v3 |                          3
 kustomize.common.kubebuilder.io/v1 |                          3

For more specific help for the init command of a certain plugins and project version
configuration please run:
    kubebuilder init --help --plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]

Default plugin keys: "go.kubebuilder.io/v3"
Default project version: "3"


Available Commands:
  alpha       Alpha-stage subcommands
  completion  Load completions for the specified shell
  create      Scaffold a Kubernetes API or webhook
  edit        Update the project configuration
  help        Help about any command
  init        Initialize a new project
  version     Print the kubebuilder version

Flags:
  -h, --help                     help for kubebuilder
      --plugins strings          plugin keys to be used for this subcommand execution
      --project-version string   project version (default "3")

Use "kubebuilder [command] --help" for more information about a command.

$ kubebuilder init -h
Initialize a new project including the following files:
  - a "go.mod" with project dependencies
  - a "PROJECT" file that stores project configuration
  - a "Makefile" with several useful make targets for the project
  - several YAML files for project deployment under the "config" directory
  - a "main.go" file that creates the manager that will run the project controllers

Usage:
  kubebuilder init [flags]

Examples:
  # Initialize a new project with your domain and name in copyright
  kubebuilder init --plugins go/v3 --domain example.org --owner "Your name"

  # Initialize a new project defining a specific project version
  kubebuilder init --plugins go/v3 --project-version 3


Flags:
      --component-config         create a versioned ComponentConfig file, may be 'true' or 'false'
      --domain string            domain for groups (default "my.domain")
      --fetch-deps               ensure dependencies are downloaded (default true)
  -h, --help                     help for init
      --license string           license to use to boilerplate, may be one of 'apache2', 'none' (default "apache2")
      --owner string             owner to add to the copyright
      --project-name string      name of this project
      --project-version string   project version (default "3")
      --repo string              name to use for go module (e.g., github.com/user/repo), defaults to the go package of the current working directory.
      --skip-go-version-check    if specified, skip checking the Go version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

$ kubebuilder init --domain magedu.com --repo gitee.com/go-course/k8s-operator

Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.0
go: downloading k8s.io/client-go v0.23.0
go: downloading github.com/evanphx/json-patch v4.12.0+incompatible
go: downloading gomodules.xyz/jsonpatch/v2 v2.2.0
go: downloading github.com/prometheus/common v0.28.0
go: downloading k8s.io/component-base v0.23.0
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369
Update dependencies:
$ go mod tidy
go: downloading github.com/go-logr/zapr v1.2.0
go: downloading github.com/onsi/gomega v1.17.0
go: downloading go.uber.org/zap v1.19.1
go: downloading github.com/Azure/go-autorest/autorest v0.11.18
go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.13
go: downloading go.uber.org/goleak v1.1.12
go: downloading cloud.google.com/go v0.81.0
go: downloading gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f
go: downloading github.com/Azure/go-autorest/logger v0.2.1
go: downloading github.com/form3tech-oss/jwt-go v3.2.3+incompatible
Next: define a resource with:
$ kubebuilder create api

$ kubebuilder create api -h
Scaffold a Kubernetes API by writing a Resource definition and/or a Controller.

If information about whether the resource and controller should be scaffolded  
was not explicitly provided, it will prompt the user if they should be.        

After the scaffold is written, the dependencies will be updated and
make generate will be run.

Usage:
  kubebuilder create api [flags]

Examples:
  # Create a frigates API with Group: ship, Version: v1beta1 and Kind: Frigate
  kubebuilder create api --group ship --version v1beta1 --kind Frigate

  # Edit the API Scheme
  nano api/v1beta1/frigate_types.go

  # Edit the Controller
  nano controllers/frigate/frigate_controller.go

  # Edit the Controller Test
  nano controllers/frigate/frigate_controller_test.go

  # Generate the manifests
  make manifests

  # Install CRDs into the Kubernetes cluster using kubectl apply
  make install

  # Regenerate code and run against the Kubernetes cluster configured by ~/.kube/config
  make run


Flags:
      --controller           if set, generate the controller without prompting the user (default true)
      --force                attempt to create resource even if it already exists
      --group string         resource Group
  -h, --help                 help for api
      --kind string          resource Kind
      --make make generate   if true, run make generate after generating files (default true)
      --namespaced           resource is namespaced (default true)
      --plural string        resource irregular plural form
      --resource             if set, generate the resource without prompting the user (default true)
      --version string       resource Version

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution

kubebuilder create api --group traefik --version v1 --kind TraefikService

Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api\v1\traefikservice_types.go
controllers\traefikservice_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
/e/Projects/Golang/go-course-projects/k8s-operator/bin/controller-gen object:headerFile="hack\\boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests

<etcd_prefix>/<entry_point>/services/loadBalancer/servers/<index>/url   <url_value>

traefik etcd配置的前缀, provider配置时 有设置
services: 表示 web entrypoint的 services配置
loadBalancer: cmdb 服务loadBalancer配置
servers: loadBalancer 下的实例配置
0(变量): index

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// TraefikServiceSpec defines the desired state of TraefikService
type TraefikServiceSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of TraefikService. Edit traefikservice_types.go to remove/update
	Entrypoint string `json:"entrypoint"`
	URL        string `json:"url"`
}

// TraefikServiceStatus defines the observed state of TraefikService
type TraefikServiceStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// 是否被激活
	Active bool `json:"active,omitempty"`
	// 最新更新时间
	LastUpdateTime *metav1.Time `json:"lastUpdateTime,omitempty"`
	// 如果更新失败, 失败的原因
	Message string `json:"message,omitempty"`
}

$ make manifests generate

/e/Projects/Golang/go-course-projects/k8s-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/e/Projects/Golang/go-course-projects/k8s-operator/bin/controller-gen object:headerFile="hack\\boilerplate.go.txt" paths="./..."

Deployment
  install          Install CRDs into the K8s cluster specified in ~/.kube/config.
  uninstall        Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion.

$ make install
/e/Projects/Golang/go-course-projects/k8s-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/e/Projects/Golang/go-course-projects/k8s-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/traefikservices.traefik.magedu.com created

$ kubectl.exe get crd
NAME                                             CREATED AT
tkeserviceconfigs.cloud.tencent.com              2021-11-19T05:58:43Z
traefikservices.traefik.magedu.com               2022-03-10T13:51:06Z
volumesnapshotclasses.snapshot.storage.k8s.io    2021-11-19T05:59:24Z
volumesnapshotcontents.snapshot.storage.k8s.io   2021-11-19T05:59:24Z
volumesnapshots.snapshot.storage.k8s.io          2021-11-19T05:59:24Z

apiVersion: traefik.magedu.com/v1
kind: TraefikService
metadata:
  name: traefikservice-sample
spec:
  # TODO(user): Add fields here
  entrypoint: web
  url: https://www.baidu.com

# 创建资源
$ kubectl.exe apply -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com/traefikservice-sample created

# 查看资源
$ kubectl.exe get TraefikService
NAME                    AGE
traefikservice-sample   13m

# 查看资源yaml
$ kubectl.exe get TraefikService -o yaml
apiVersion: v1
items:
- apiVersion: traefik.magedu.com/v1
  kind: TraefikService
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"traefik.magedu.com/v1","kind":"TraefikService","metadata":{"annotations":{},"name":"traefikservice-sample","namespace":"default"},"spec":{"entrypoint":"web","url":"https://www.baidu.com"}}
    creationTimestamp: "2022-03-10T14:29:48Z"
    generation: 1
    name: traefikservice-sample
    namespace: default
    resourceVersion: "6043243202"
    selfLink: /apis/traefik.magedu.com/v1/namespaces/default/traefikservices/traefikservice-sample
    uid: aac22af5-6130-4d85-a5b0-dd4b4ff6d1de
  spec:
    entrypoint: web
    url: https://www.baidu.com
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ kubectl.exe delete -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com "traefikservice-sample" deleted

// TraefikServiceReconciler reconciles a TraefikService object
type TraefikServiceReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

// 每当traefikv1.TraefikService{}对象有变化时，我们就会收到一个请求
func (r *TraefikServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 获取日志对象
	l := log.FromContext(ctx, "namespace", req.Namespace)

	// TODO(user): your logic here

	// 1.通过名称获取TraefikService对象, 并打印
	var obj traefikv1.TraefikService
	if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
		// 如果Not Found则表示该资源已经删除, 需要做删除处理
		if apierrors.IsNotFound(err) {
			l.Info("delete service ...")
			err = nil
		} else {
      l.Error(err, "unable to fetch TraefikService")
    }

		return ctrl.Result{}, err
	}

	l.Info("get TraefikService object",
		"name", obj.Name,
		"url", obj.Spec.URL,
		"entrypoint", obj.Spec.Entrypoint)

	// 2. 更加状态 排除已经同步完成的对象
	if obj.Status.Active {
		l.Info("traefik service is active, skip ...")
		return ctrl.Result{}, nil
	}

	// 3. 注册服务到Traefik service中, 比如写入到etcd provider中
	l.Info("set traefik service ...")

	// 4. 修改成功调整对象的状态
	obj.Status.Active = true
	obj.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}
	if err := r.Status().Update(ctx, &obj); err != nil {
		l.Info("unable to update status", "reason", err.Error())
	}

	return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *TraefikServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&traefikv1.TraefikService{}).
		Complete(r)
}

make manifests && make install

make run

# 创建资源
$ kubectl.exe apply -f config/samples/traefik_v1_traefikservice.yaml 
traefikservice.traefik.magedu.com/traefikservice-sample created

...
1.6478685486381395e+09  INFO    controller.traefikservice       get TraefikService object       {"reconciler group": "traefik.magedu.com", "reconciler kind": "TraefikService", "name": "traefikservice-sample", "namespace": "default", "namespace": "default", "name": "traefikservice-sample", "url": "https://www.baidu.com", "entrypoint": "web"}
1.6478685486386724e+09  INFO    controller.traefikservice       set traefik service ... {"reconciler group": "traefik.magedu.com", "reconciler kind": "TraefikService", "name": "traefikservice-sample", "namespace": "default", "namespace": "default"}
1.6478685486955843e+09  INFO    controller.traefikservice       get TraefikService object       {"reconciler group": "traefik.magedu.com", "reconciler kind": "TraefikService", "name": "traefikservice-sample", "namespace": "default", "namespace": "default", "name": "traefikservice-sample", "url": "https://www.baidu.com", "entrypoint": "web"}
1.6478685486961432e+09  INFO    controller.traefikservice       traefik service is active, skip ...     {"reconciler group": "traefik.magedu.com", "reconciler kind": "TraefikService", "name": "traefikservice-sample", 
"namespace": "default", "namespace": "default"}

// SetupWithManager sets up the controller with the Manager.
func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&v1.Pod{}).
		Complete(r)
}

func (r *EndpointReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	// 获取日志对象
	l := log.FromContext(ctx)

	// TODO(user): your logic here

	// 1.通过名称获取Pods对象, 并打印
	var obj v1.Pod
	if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
		// 如果Not Found则表示该资源已经删除, 需要做删除处理
		if apierrors.IsNotFound(err) {
			l.Info("delete Pods ...",
				"namespace", req.Namespace,
				"name", req.Name)
			err = nil
		} else {
			l.Error(err, "unable to fetch Pods")
		}
	}

	eps, _ := json.Marshal(obj)
	l.Info(string(eps))

	return ctrl.Result{}, nil
}

func main() {
	var metricsAddr string
	var enableLeaderElection bool
	var probeAddr string
	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")
	opts := zap.Options{
		Development: true,
	}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))

	// 默认配置
	options := ctrl.Options{
		Scheme:                 scheme,
		MetricsBindAddress:     metricsAddr,
		Port:                   9443,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "d18673dd.magedu.com",
	}

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), options)
	if err != nil {
		setupLog.Error(err, "unable to start manager")
		os.Exit(1)
	}

	if err = (&controllers.TraefikServiceReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "TraefikService")
		os.Exit(1)
	}
	//+kubebuilder:scaffold:builder

	// Pods Controller
	if err = (&controllers.PodReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "Pods")
		os.Exit(1)
	}

	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up health check")
		os.Exit(1)
	}
	if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up ready check")
		os.Exit(1)
	}

	setupLog.Info("starting manager")
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}

1.6479577314173741e+09  INFO    controller.pod  nginx-86bd55b966-hwhtz  {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default", "namespace": "default", "labels": {"k8s-app":"nginx","pod-template-hash":"86bd55b966","qcloud-app":"nginx"}}
1.647957731418373e+09   INFO    controller.pod  172.16.0.69     {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default"}
1.647957731419115e+09   INFO    controller.pod  test    {"reconciler group": "", "reconciler kind": "Pod", "name": "nginx-86bd55b966-hwhtz", "namespace": "default", "pod_id": null}