# kubebuilder-tutorial

**Repository Path**: wilds/kubebuilder-tutorial

## Basic Information

- **Project Name**: kubebuilder-tutorial
- **Description**: kubebuilder-tutorial-test
- **Primary Language**: Go
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-05-11
- **Last Updated**: 2024-05-16

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# kube-builder
## version
```
sigs.k8s.io/kubebuilder/v3 v3.14.1
  sigs.k8s.io/controller-tools v0.14.0(k8s.io/api v0.29.0)
  sigs.k8s.io/controller-runtime v0.17.2
    k8s.io/client-go v0.29.0
	  k8s.io/kubernetes 1.29.0
	  k8s.io/code-generator v0.29.0
```
```
git checkout -b v3.14.1 tags/v3.14.1
```
## init
```
[root@openfuyao-0004 ccc]# mkdir kubebuilder-tutorial
[root@openfuyao-0004 ccc]# cd kubebuilder-tutorial/
[root@openfuyao-0004 kubebuilder-tutorial]# go mod init kubebuilder-tutorial
go: creating new go.mod: module kubebuilder-tutorial
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder init --domain wilds.com
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.17.2
INFO Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder edit --multigroup=true
[root@openfuyao-0004 kubebuilder-tutorial]# make kustomize
mkdir -p /opt/paul/code/wilds/ccc/kubebuilder-tutorial/bin
Downloading sigs.k8s.io/kustomize/kustomize/v5@v5.3.0
[root@openfuyao-0004 kubebuilder-tutorial]# make controller-gen
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.14.0
[root@openfuyao-0004 kubebuilder-tutorial]#

```
## create api
### 创建资源api/project/v1/User
```
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder create api --group project --version v1 --kind User
INFO Create Resource [y/n]
y
INFO Create Controller [y/n]
y
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO api/project/v1/user_types.go
INFO api/project/v1/groupversion_info.go
INFO internal/controller/project/suite_test.go
INFO internal/controller/project/user_controller.go
INFO internal/controller/project/user_controller_test.go
INFO Update dependencies:
$ go mod tidy
INFO Running make:
$ make generate
/opt/paul/code/wilds/bbb/kubebuilder-tutorial/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
```
### 设置资源api/project/v1/User存储版本`//+kubebuilder:storageversion`
```
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:storageversion

// User is the Schema for the users API
type User struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   UserSpec   `json:"spec,omitempty"`
	Status UserStatus `json:"status,omitempty"`
}
```
### 创建资源api/project/v2/User
```
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder create api --group project --version v2 --kind User
INFO Create Resource [y/n]
y
INFO Create Controller [y/n]
n
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO api/project/v2/user_types.go
INFO api/project/v2/groupversion_info.go
INFO Update dependencies:
$ go mod tidy
INFO Running make:
$ make generate
/opt/paul/code/wilds/bbb/kubebuilder-tutorial/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
```
### 创建资源api/project/v1/UserGroup
```
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder create api --group project --version v1 --kind UserGroup
INFO Create Resource [y/n]
y
INFO Create Controller [y/n]
y
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO api/project/v1/usergroup_types.go
INFO api/project/v1/groupversion_info.go
INFO internal/controller/project/suite_test.go
INFO internal/controller/project/usergroup_controller.go
INFO internal/controller/project/usergroup_controller_test.go
INFO Update dependencies:
$ go mod tidy
INFO Running make:
$ make generate
/opt/paul/code/wilds/bbb/kubebuilder-tutorial/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
```
### 设置资源api/project/v1/UserGroup存储版本`//+kubebuilder:storageversion`
```
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:storageversion

// UserGroup is the Schema for the users API
type UserGroup struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   UserGroupSpec   `json:"spec,omitempty"`
	Status UserGroupStatus `json:"status,omitempty"`
}
```
### 创建资源api/project/v2/UserGroup
```
[root@openfuyao-0004 kubebuilder-tutorial]# kubebuilder create api --group project --version v2 --kind UserGroup
INFO Create Resource [y/n]
y
INFO Create Controller [y/n]
n
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO api/project/v2/usergroup_types.go
INFO api/project/v2/groupversion_info.go
INFO Update dependencies:
$ go mod tidy
INFO Running make:
$ make generate
/opt/paul/code/wilds/ccc/kubebuilder-tutorial/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
```
### 创建内部资源api/project/User与api/project/UserGroup
#### 创建内部资源
```
cp api/project/v1/user_types.go api/project
cp api/project/v1/usergroup_types.go api/project
cp api/project/v1/groupversion_info.go api/project
```
- 删除groupversion_info.go中的包comments `// +groupName=project.wilds.com`,不然执行`make manifests`会报错?
- 修改package为`package proejct`
- 删除User、UserGroup中的`//+kubebuilder:storageversion`comments
- 删除各资源中的tag
- 修改groupversion_info.go版本为runtime.APIVersionInternal `GroupVersion = schema.GroupVersion{Group: "project.wilds.com", Version: runtime.APIVersionInternal}`
#### install
```
mkdir -p api/project/install

cat<<EOF>api/project/install/install.go
package install

import (
  "k8s.io/apimachinery/pkg/runtime"
  utilruntime "k8s.io/apimachinery/pkg/util/runtime"
  "k8s.io/kubernetes/pkg/api/legacyscheme"
  "tutorial/api/project"
  "tutorial/api/project/v1"
  "tutorial/api/project/v2"
)

func init() {
  Install(legacyscheme.Scheme)
}

// Install registers the API group and adds types to a scheme
func Install(scheme *runtime.Scheme) {
  utilruntime.Must(project.AddToScheme(scheme))
  utilruntime.Must(v1.AddToScheme(scheme))
  utilruntime.Must(v2.AddToScheme(scheme))
  utilruntime.Must(scheme.SetVersionPriority(v2.SchemeGroupVersion, v1.SchemeGroupVersion))
}
EOF
```
#### 增加SchemeGroupVersion变量(供clientset使用)/增加localSchemeBuilder变量(供conversion使用)
- 增加register.go文件，增加`SchemeGroupVersion = GroupVersion`
- 增加register.go文件，增加`localSchemeBuilder = SchemeBuilder.SchemeBuilder`
```
cat<<EOF>api/project/register.go
package project

var (
	// 增加SchemeGroupVersion变量
	SchemeGroupVersion = GroupVersion
)
EOF

cat<<EOF>api/project/v1/register.go
package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

var (
// 增加SchemeGroupVersion变量
SchemeGroupVersion = GroupVersion
// 增加localSchemeBuilder变量
localSchemeBuilder = SchemeBuilder.SchemeBuilder
)

func init() {
	// We only register manually written functions here. The registration of the
	// generated functions takes place in the generated files. The separation
	// makes the code compile even when the generated files are missing.
	localSchemeBuilder.Register(addKnownTypes)
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

// Adds the list of known types to api.Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&User{},
		&UserList{},
		&UserGroup{},
		&UserGroupList{},
	)

	scheme.AddKnownTypes(SchemeGroupVersion,
		&metav1.Status{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}
EOF

cat<<EOF>api/project/v2/register.go
package v2

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

var (
// 增加SchemeGroupVersion变量
SchemeGroupVersion = GroupVersion
// 增加localSchemeBuilder变量
localSchemeBuilder = SchemeBuilder.SchemeBuilder
)

func init() {
	// We only register manually written functions here. The registration of the
	// generated functions takes place in the generated files. The separation
	// makes the code compile even when the generated files are missing.
	localSchemeBuilder.Register(addKnownTypes)
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

// Adds the list of known types to api.Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&User{},
		&UserList{},
		&UserGroup{},
		&UserGroupList{},
	)

	scheme.AddKnownTypes(SchemeGroupVersion,
		&metav1.Status{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}
EOF
```
> - `kubebuilder create api`与cliengen的生成规则不同,需要手动添加SchemeGroupVersion/localSchemeBuilder变量
> - `listergen的生成代码中依赖Resource函数
### make
```
[root@openfuyao-0004 kubebuilder-tutorial]# go mod tidy
[root@openfuyao-0004 kubebuilder-tutorial]# go mod vendor
[root@openfuyao-0004 kubebuilder-tutorial]# make generate
/opt/paul/code/wilds/bbb/kubebuilder-tutorial/bin/controller-gen-v0.14.0 object:headerFile="hack/boilerplate.go.txt" paths="./..."
[root@openfuyao-0004 kubebuilder-tutorial]# make manifests
/opt/paul/code/wilds/bbb/kubebuilder-tutorial/bin/controller-gen-v0.14.0 rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
```
## code-gen(client-gen/lister-gen/informer-gen)
### 在各Resource增加`//+genclient`comments(XxxList资源不用添加)
```
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:storageversion
//+genclient
//+genclient:method=Scale,verb=update,subresource=scale,input=k8s.io/api/extensions/v1beta1.Scale,result=k8s.io/api/extensions/v1beta1.Scale

// User is the Schema for the users API
type User struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   UserSpec   `json:"spec,omitempty"`
	Status UserStatus `json:"status,omitempty"`
}
```
### 在doc.go中增加groupName(register-gen使用)
```
cat<<EOF>api/project/v1/doc.go
//+k8s:conversion-gen=kubebuilder-tutorial/api/project
//+k8s:conversion-gen-external-types=kubebuilder-tutorial/api/project/v1
//+k8s:defaulter-gen=TypeMeta
//+k8s:defaulter-gen-input=kubebuilder-tutorial/api/project/v1

// Package v1 is the v1 version of the API.
// +groupName=project.wilds.com
package v1 // import "kubebuilder-tutorial/api/project/v1"
```

```
cat<<EOF>api/project/v1/doc.go
//+k8s:conversion-gen=kubebuilder-tutorial/api/project
//+k8s:conversion-gen-external-types=kubebuilder-tutorial/api/project/v2
//+k8s:defaulter-gen=TypeMeta
//+k8s:defaulter-gen-input=kubebuilder-tutorial/api/project/v2

// Package v1 is the v1 version of the API.
// +groupName=project.wilds.com
package v2 // import "kubebuilder-tutorial/api/project/v2"
```
### 执行client代码生成(最新版本：v.0.30.0)
```
client-gen --clientset-api-path=/apis --clientset-name=clientset --input-base="kubebuilder-tutorial/api/" --input="project/v1,project/v2,project" --output-dir="./client" --output-pkg="tutorial/client" -v 9

client-gen --clientset-api-path=/apis --clientset-name=internalclientset --input-base="kubebuilder-tutorial/pkg/api/" --input="project" --output-dir="./client" --output-pkg="tutorial/client" -v 9

lister-gen --output-dir="./client/listers" --output-pkg="tutorial/client/listers" -v 9 kubebuilder-tutorial/api/project/v1 kubebuilder-tutorial/api/project/v2 kubebuilder-tutorial/api/project

informer-gen --listers-package="kubebuilder-tutorial/client/listers" --internal-clientset-package="kubebuilder-tutorial/client/clientset" --versioned-clientset-package="kubebuilder-tutorial/client/clientset" --output-dir="./client/informers" --output-pkg="kubebuilder-tutorial/client/informers" -v 9 kubebuilder-tutorial/api/project/v1 tutorial/api/project/v2 tutorial/api/project
```
#### 执行client代码生成(v.0.29.0)
```
mkdir hack
cat<<EOF>hack/boilerplate.go.txt
/*
Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
EOF
```
```
#register-gen --go-header-file=hack/boilerplate.go.txt --input-dirs="kubebuilder-tutorial/api/project/v1,kubebuilder-tutorial/api/project/v2" --output-base=".." -v 9

client-gen --go-header-file=hack/boilerplate.go.txt --clientset-api-path=/apis --clientset-name=clientset --input-base="kubebuilder-tutorial/api/" --input="project/v1,project/v2,project" --output-base=".." --output-package="kubebuilder-tutorial/client" -v 9

client-gen --go-header-file=hack/boilerplate.go.txt --clientset-api-path=/apis --clientset-name=internalclientset --input-base="kubebuilder-tutorial/api/" --input="project" --output-base=".." --output-package="kubebuilder-tutorial/client" -v 9

lister-gen --go-header-file=hack/boilerplate.go.txt --output-base=".." --output-file-base="./client/listers" --output-package="kubebuilder-tutorial/client/listers" -v 9 --input-dirs=kubebuilder-tutorial/api/project/v1,kubebuilder-tutorial/api/project/v2,kubebuilder-tutorial/api/project

informer-gen --go-header-file=hack/boilerplate.go.txt --listers-package="kubebuilder-tutorial/client/listers" --internal-clientset-package="kubebuilder-tutorial/client/clientset" --versioned-clientset-package="kubebuilder-tutorial/client/clientset" --output-dir="./client/informers" --output-pkg="kubebuilder-tutorial/client/informers" -v 9 kubebuilder-tutorial/api/project/v1 kubebuilder-tutorial/api/project/v2 kubebuilder-tutorial/api/project
```
### 依赖更新
```
go mod tidy
go mod vendor
```
## code-gen(defaulter-gen/conversion-gen)
### 增加defaulter marker
- (可选)在User类型中增加`//+k8s:defaulter-gen=true`
- 在UserSpec field里增加`//+default="foo"`
### 增加增加defaulter/conversion marker
在api/project/v1目录,增加doc.go文件
```
cat<<EOF>api/project/v1/doc.go
//+k8s:conversion-gen=kubebuilder-tutorial/api/project
//+k8s:conversion-gen-external-types=kubebuilder-tutorial/api/project/v1
//+k8s:defaulter-gen=TypeMeta
//+k8s:defaulter-gen-input=kubebuilder-tutorial/api/project/v1

// Package v1 is the v1 version of the API.
package v1 // import "kubebuilder-tutorial/api/project/v1"
EOF
```
在api/project/v2目录,增加doc.go文件
```
cat<<EOF>api/project/v2/doc.go
//+k8s:conversion-gen=kubebuilder-tutorial/api/project
//+k8s:conversion-gen-external-types=kubebuilder-tutorial/api/project/v2
//+k8s:defaulter-gen=TypeMeta
//+k8s:defaulter-gen-input=kubebuilder-tutorial/api/project/v2

// Package v2 is the v2 version of the API.
package v2 // import "kubebuilder-tutorial/api/project/v2"
EOF
```
### defaulter-gen与conversion-gen代码生成
```
defaulter-gen --go-header-file=hack/boilerplate.go.txt --output-base=".." --output-file-base=generated.defaults.go -v 9 --input-dirs=kubebuilder-tutorial/api/project/v1,kubebuilder-tutorial/api/project/v2
conversion-gen --go-header-file=hack/boilerplate.go.txt --output-base=".." --output-file-base=generated.conversion.go -v 9 --input-dirs=kubebuilder-tutorial/api/project/v1,kubebuilder-tutorial/api/project/v2
```
v0.30.0版本
```
# defaulter-gen --output-file=generated.defaults.go -v 9 kubebuilder-tutorial/api/project/v1
#conversion-gen --output-file=generated.conversion.go -v 9 kubebuilder-tutorial/api/project/v1
```
### 查看
```
cat api/project/v1/generated.defaults.go
cat api/project/v1/generated.conversion.go
```
## code-gen(controller-gen产生webhook)
### 在UserSpec.Foo中增加如下comments `//+kubebuilder:validation:MinLength=2`
```
// UserSpec defines the desired state of User
type UserSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file

	// Foo is an example field of User. Edit user_types.go to remove/update
    //+kubebuilder:validation:MinLength=2
    //+default="foo"
	Foo string `json:"foo,omitempty"`
}
```
### project/v1实现Hub接口
```
cat<<EOF>api/project/v1/user_conversion.go
package v1
import ( 
  _ "sigs.k8s.io/controller-runtime/pkg/conversion"
)

// Hub marks this type as a conversion hub.
func (*User) Hub() {}
EOF
```
### project/v2实现Convertible接口
```
cat<<EOF>api/project/v2/user_conversion.go
package v2
import (
  "kubebuilder-tutorial/api/project/v1"  
  "sigs.k8s.io/controller-runtime/pkg/conversion"
)

// ConvertTo converts this CronJob to the Hub version (v1).
func (src *User) ConvertTo(dstRaw conversion.Hub) error {
    dst := dstRaw.(*v1.User)
    dst.Spec.Foo = src.Spec.Foo
    return nil
}

// ConvertFrom converts from the Hub version (v1) to this version.
func (dst *User) ConvertFrom(srcRaw conversion.Hub) error {
    src := srcRaw.(*v1.User)
    dst.Spec.Foo = src.Spec.Foo
    return nil
}
EOF
```
### 生成webhook(default/conversion/validation)代码
```
kubebuilder create webhook --group project --version v1 --kind User --defaulting --force
kubebuilder create webhook --group project --version v1 --kind User --defaulting --conversion --programmatic-validation --force
kubebuilder create webhook --group project --version v2 --kind User --defaulting --conversion --programmatic-validation --force
make generate
make manifests
```
### (可选)生成webhook配置
```
controller-gen webhook paths=tutorial/api/project/v1
# controller-gen webhook paths=tutorial/api/project/v1 output:stdout
cat config/webhook/manifests.yaml
```
> - kubebuilder的default生成的代码中可直接调用defaulter-gen生成的代码。
> - kubebuilder的conversion生成的代码中可直接调用conversion-gen生成的代码。

（可选）在api/project/v1/UserGroup中增加如下comments，可通过上面的命令生成config/webhook/manifests.yaml中的配置
```
//+kubebuilder:webhook:webhookVersions=v1,verbs=create;update,path=/validate-project-wilds-com-v1-usergroup,mutating=false,failurePolicy=fail,matchPolicy=Equivalent,groups=project.wilds.com,resources=usergroups,versions=v1,name=validation.usergroup.project.wilds.com,sideEffects=None,timeoutSeconds=10,admissionReviewVersions=v1;v1beta1
//+kubebuilder:webhook:webhookVersions=v1,verbs=create;update,path=/mutate-project-wilds-com-v1-project,mutating=true,failurePolicy=fail,matchPolicy=Equivalent,groups=project.wilds.com,resources=usergroups,versions=v1,name=default.usergroup.project.wilds.com,sideEffects=None,timeoutSeconds=10,admissionReviewVersions=v1;v1beta1,reinvocationPolicy=IfNeeded
```