企业项目实战k8s篇（十六）Helm部署使用及chart仓库搭建

发布时间：2023-03-26 15:00

Helm部署使用及chart仓库搭建

一.helm概述
二.helm安装部署
- 1.helm安装
- 2.helm部署应用
- 补充：删除之前部署pvc
三.chart仓库及应用发布
- 1.应用创建
- 2.本地chart仓库及push插件
- 3.应用上传
- 4.应用安装
- 5.应用升级
- 6.应用卸载

一.helm概述

Helm是Kubernetes 应用的包管理工具，主要用来管理 Charts，类似Linux系统的yum。

Helm Chart 是用来封装 Kubernetes 原生应用程序的一系列 YAML 文件。可以在你部署应用的时候自定义应用程序的一些 Metadata，以便于应用程序的分发。

对于应用发布者而言，可以通过 Helm 打包应用、管理应用依赖关系、管理应用版本并发布应用到软件仓库。

对于使用者而言，使用 Helm 后不用需要编写复杂的应用部署文件，可以以简单的方式在 Kubernetes 上查找、安装、升级、回滚、卸载应用程序。

Helm V3 与 V2 最大的区别在于去掉了tiller

二.helm安装部署

1.helm安装

Helm 官网：https://helm.sh/docs/intro/

下载helm helm-v3.4.1-linux-amd64.tar.gz

创建helm目录，用于进行helm相关实验，解压压缩包到目录

[root@server1 ~]# mkdir helm
[root@server1 ~]# cd helm/
[root@server1 helm]# ls
[root@server1 helm]# mv ../helm
helm/                           helm-v3.4.1-linux-amd64.tar.gz  
[root@server1 helm]# mv ../helm-v3.4.1-linux-amd64.tar.gz  .
[root@server1 helm]# ls
helm-v3.4.1-linux-amd64.tar.gz
[root@server1 helm]# tar zxf helm-v3.4.1-linux-amd64.tar.gz 
[root@server1 helm]# ls
helm-v3.4.1-linux-amd64.tar.gz  linux-amd64

将二进制可执行文件 helm 复制到 /usr/local/bin/，测试helm命令 helm help

[root@server1 helm]# cd linux-amd64/
[root@server1 linux-amd64]# ls
helm  LICENSE  README.md
[root@server1 linux-amd64]# cp helm  /usr/local/bin/
[root@server1 linux-amd64]# ls
helm  LICENSE  README.md
[root@server1 linux-amd64]# helm help
The Kubernetes package manager

Common actions for Helm:

- helm search:    search for charts
- helm pull:      download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

设置helm命令补齐

[root@server1 linux-amd64]# cd
[root@server1 ~]# echo "source <(helm completion bash)" >> ~/.bashrc
[root@server1 ~]# source  .bashrc 
[root@server1 ~]# helm 
completion  dependency  get         install     list        plugin      repo        search      status      test        upgrade     version     
create      env         history     lint        package     pull        rollback    show        template    uninstall   verify

Helm 添加第三方 Chart 库

[root@server1 ~]# helm repo  add  bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
[root@server1 ~]# helm repo list
NAME   	URL                               
bitnami	https://charts.bitnami.com/bitnami

2.helm部署应用

查询redis应用

[root@server1 ~]# helm search repo redis
NAME                 	CHART VERSION	APP VERSION	DESCRIPTION                                       
bitnami/redis        	14.8.7       	6.2.5      	Open source, advanced key-value store. It is of...
bitnami/redis-cluster	6.3.2        	6.2.5      	Open source, advanced key-value store. It is of...

拉取应用

[root@server1 ~]# helm pull bitnami/redis-cluster

解压压缩包redis-cluster-6.3.2.tgz，进入目录，values.yaml内为所有变量 Chart.yaml 描述Chart的基本信息,包括名称版本等

[root@server1 ~]# mv redis-cluster-6.3.2.tgz  helm/
[root@server1 ~]# cd helm/
[root@server1 helm]# tar zxf redis-cluster-6.3.2.tgz 
[root@server1 helm]# cd redis-cluster/
[root@server1 redis-cluster]# ls
Chart.lock  charts  Chart.yaml  img  README.md  templates  values.yaml
[root@server1 redis-cluster]# ll values.yaml 
-rw-r--r-- 1 root root 34207 Jul 30 08:12 values.yaml
[root@server1 redis-cluster]# vim values.yaml

拉取部署应用所需镜像redis-cluster:6.2.5-debian-10-r0，更改tag并上传至集群harbor仓库

[root@server3 ~]# docker pull bitnami/redis-cluster:6.2.5-debian-10-r0
6.2.5-debian-10-r0: Pulling from bitnami/redis-cluster
86a19151e740: Already exists 
57a1d2d33896: Pull complete 
205fbdb2c4df: Pull complete 
b50f4b53d89e: Pull complete 
daec4ad90bad: Pull complete 
3a2283fca554: Pull complete 
795c1eb71802: Pull complete 
4084d80f7f4f: Pull complete 
Digest: sha256:0adf6a9c86f820646455894283af74205209ceaa1dd1c3751e811b4c8ef02652
Status: Downloaded newer image for bitnami/redis-cluster:6.2.5-debian-10-r0
docker.io/bitnami/redis-cluster:6.2.5-debian-10-r0
[root@server3 ~]# docker tag  docker.io/bitnami/redis-cluster:6.2.5-debian-10-r0 reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0
[root@server3 ~]# docker push reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0
The push refers to repository [reg.westos.org/bitnami/redis-cluster]
265c6517575c: Pushed 
27927d559e88: Pushed 
f96e0118c822: Pushed 
1615551dea1b: Pushed 
c65559991f66: Pushed 
8a9b93540aad: Pushed 
566ccae6ccab: Pushed 
3fa01eaf81a5: Mounted from library/metrics-server 
6.2.5-debian-10-r0: digest: sha256:0adf6a9c86f820646455894283af74205209ceaa1dd1c3751e811b4c8ef02652 size: 1996

执行安装命令

[root@server1 redis-cluster]# helm install redis-cluster .
NAME: redis-cluster
LAST DEPLOYED: Wed Aug  4 02:57:59 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **


To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "default" redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)

You have deployed a Redis&trade; Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis&trade; cluster:

1. Run a Redis&trade; pod that you can use as a client:
   kubectl run --namespace default redis-cluster-client --rm --tty -i --restart='Never' \
    --env REDIS_PASSWORD=$REDIS_PASSWORD \
   --image reg.westos.org/bitnami/redis-cluster:6.2.5-debian-10-r0 -- bash

2. Connect using the Redis&trade; CLI:

redis-cli -c -h redis-cluster -a $REDIS_PASSWORD

查看已安装应用

[root@server1 redis-cluster]# helm list
NAME         	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART              	APP VERSION
redis-cluster	default  	1       	2021-08-04 02:57:59.283039665 -0400 EDT	deployed	redis-cluster-6.3.2	6.2.5      
[root@server1 redis-cluster]# kubectl  get pod
NAME                          READY   STATUS    RESTARTS   AGE
php-apache-6cc67f7957-crtvk   1/1     Running   0          3h16m
redis-cluster-0               1/1     Running   1          100s
redis-cluster-1               1/1     Running   1          100s
redis-cluster-2               1/1     Running   1          100s
redis-cluster-3               1/1     Running   2          99s
redis-cluster-4               1/1     Running   1          99s
redis-cluster-5               1/1     Running   0          99s

补充：删除之前部署pvc

[root@server1 redis-cluster]# kubectl  get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS          REASON   AGE
pvc-037cf46b-d7f5-41f6-b156-17799899daaa   1Gi        RWO            Delete           Bound    default/www-web-2   managed-nfs-storage            4d2h
pvc-22524877-c04c-498e-955a-5f121a1e40cc   100Mi      RWX            Delete           Bound    default/nfs-pv1     managed-nfs-storage            4d2h
pvc-5a835670-8962-4714-8311-c8a441bf98eb   1Gi        RWO            Delete           Bound    default/www-web-0   managed-nfs-storage            4d2h
pvc-b227ba79-3f52-4d34-a1fd-655ff1516592   1Gi        RWO            Delete           Bound    default/www-web-1   managed-nfs-storage            4d2h
[root@server1 redis-cluster]# kubectl  get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
nfs-pv1     Bound    pvc-22524877-c04c-498e-955a-5f121a1e40cc   100Mi      RWX            managed-nfs-storage   4d2h
www-web-0   Bound    pvc-5a835670-8962-4714-8311-c8a441bf98eb   1Gi        RWO            managed-nfs-storage   4d2h
www-web-1   Bound    pvc-b227ba79-3f52-4d34-a1fd-655ff1516592   1Gi        RWO            managed-nfs-storage   4d2h
www-web-2   Bound    pvc-037cf46b-d7f5-41f6-b156-17799899daaa   1Gi        RWO            managed-nfs-storage   4d2h
[root@server1 redis-cluster]# kubectl  delete  pvc --all
persistentvolumeclaim "nfs-pv1" deleted
persistentvolumeclaim "www-web-0" deleted
persistentvolumeclaim "www-web-1" deleted
persistentvolumeclaim "www-web-2" deleted
[root@server1 redis-cluster]# kubectl  get pvc
No resources found in default namespace.

三.chart仓库及应用发布

1.应用创建

构建一个 Helm Chart，查看mychart 结构

[root@server1 helm]# helm create  mychart
Creating mychart
[root@server1 helm]# tree mychart/
mychart/
├── charts
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

3 directories, 10 files

编写mychart的应用描述信息编写应用部署信息：

[root@server1 helm]# cd mychart/
[root@server1 mychart]# ls
charts  Chart.yaml  templates  values.yaml
[root@server1 mychart]# vim values.yaml 
[root@server1 mychart]# vim Chart.yaml 
[root@server1 mychart]# vim values.yaml 
[root@server1 mychart]# vim Chart.yaml 
[root@server1 mychart]# cd ..

检查依赖和模板配置是否正确:

$ helm  lint mychart/
==> Linting mychart/
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed

将应用打包

[root@server1 helm]# helm package  mychart/
Successfully packaged chart and saved it to: /root/helm/mychart-0.1.0.tgz
[root@server1 helm]# ls
mychart  mychart-0.1.0.tgz  redis-cluster  redis-cluster-6.3.2.tgz

2.本地chart仓库及push插件

建立本地chart仓库，本地仓库以harbor仓库为基础，在安装harbor时指定chart模块，建立的harbor也可提供chart仓库服务

harbor搭建详见之前博客企业项目实战docker篇（四）企业级仓库Harbor搭建

harbor内创建charts项目

添加本地chart库westos（自指定仓库名，但路径一定要是真实有效路径），添加失败，显示未通过认证

[root@server1 helm]# helm repo  add  westos https://reg.westos.org/chartrepo/charts
Error: looks like "https://reg.westos.org/chartrepo/charts" is not a valid chart repository or cannot be reached: Get "https://reg.westos.org/chartrepo/charts/index.yaml": x509: certificate signed by unknown authority

将认证文件 ca.crt复制到 /etc/pki/ca-trust/source/anchors/，执行更新命令 update-ca-trust

[root@server1 helm]# cd /etc/pk
pkcs11/ pki/    
[root@server1 helm]# cd /etc/pki/ca-trust/source/anchors/
[root@server1 anchors]# ls
[root@server1 anchors]# pwd
/etc/pki/ca-trust/source/anchors
[root@server1 anchors]# cp /etc/docker/certs.d/reg.westos.org/ca.crt  .
[root@server1 anchors]# ls
ca.crt
[root@server1 anchors]# update-ca-trust

再次添加本地chart库 westos

[root@server1 anchors]# helm repo  add  westos https://reg.westos.org/chartrepo/charts
"westos" has been added to your repositories
[root@server1 anchors]# helm repo list
NAME   	URL                                    
bitnami	https://charts.bitnami.com/bitnami     
westos 	https://reg.westos.org/chartrepo/charts
[root@server1 anchors]# cd

仓库搭建完毕，安装push插件 helm-push_0.9.0_linux_amd64.tar.gz

插件压缩包

[root@server1 ~]# mv helm-push_0.9.0_linux_amd64.tar.gz  helm
[root@server1 ~]# cd helm/
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz

查看环境变量

[root@server1 helm]# helm env
HELM_BIN="helm"
HELM_CACHE_HOME="/root/.cache/helm"
HELM_CONFIG_HOME="/root/.config/helm"
HELM_DATA_HOME="/root/.local/share/helm"
HELM_DEBUG="false"
HELM_KUBEAPISERVER=""
HELM_KUBEASGROUPS=""
HELM_KUBEASUSER=""
HELM_KUBECONTEXT=""
HELM_KUBETOKEN=""
HELM_MAX_HISTORY="10"
HELM_NAMESPACE="default"
HELM_PLUGINS="/root/.local/share/helm/plugins"
HELM_REGISTRY_CONFIG="/root/.config/helm/registry.json"
HELM_REPOSITORY_CACHE="/root/.cache/helm/repository"
HELM_REPOSITORY_CONFIG="/root/.config/helm/repositories.yaml"

/root/.local/share/helm/plugins不存在，手动创建

[root@server1 helm]# cd /root/.local/share/helm/plugins
-bash: cd: /root/.local/share/helm/plugins: No such file or directory

进入创建好的目录，创建push目录，将push插件解压至该目录下/root/.local/share/helm/plugins/push/

[root@server1 helm]# cd /root/.local/share/helm/plugins
[root@server1 plugins]# ls
[root@server1 plugins]# mkdir push
[root@server1 plugins]# cd
[root@server1 ~]# cd helm/
[root@server1 helm]# tar zxf helm-push_0.9.0_linux_amd64.tar.gz  -C /root/.local/share/helm/plugins/push/

可以看到push命令可以使用

[root@server1 helm]# helm push 
completion  dependency  get         install     list        plugin      repo        search      status      test        upgrade     version     
create      env         history     lint        package     pull        rollback    show        template    uninstall   verify

3.应用上传

上传应用，应用使用myapp：v1镜像，tag：v1 版本号：0.1.0

[root@server1 helm]# helm push  mychart-0.1.0.tgz  westos --insecure -u admin -p westos
Pushing mychart-0.1.0.tgz to westos...
Done.

上传应用后要更新helm repo，可以查找到mychart

[root@server1 helm]# helm repo update 
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "westos" chart repository
[root@server1 helm]# helm search  repo  mychart
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                
westos/mychart	0.1.0        	v1         	A Helm chart for Kubernetes

查看mychart应用的values

[root@server1 helm]# helm show values  westos/mychart 
# Default values for mychart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: myapp
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "v1"

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podSecurityContext: {}
  # fsGroup: 2000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths: []
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  # targetMemoryUtilizationPercentage: 80

nodeSelector: {}

tolerations: []

affinity: {}

4.应用安装

应用安装helm install mychart westos/mychart

[root@server1 helm]# helm install mychart westos/mychart 
NAME: mychart
LAST DEPLOYED: Wed Aug  4 03:55:34 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
    export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=mychart,app.kubernetes.io/instance=mychart" -o jsonpath="{.items[0].metadata.name}")
    export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
    echo "Visit http://127.0.0.1:8080 to use your application"
    kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

查看部署，可以看到一切都按照values内配置自动部署完成

[root@server1 helm]# kubectl  get all
NAME                           READY   STATUS    RESTARTS   AGE
pod/mychart-6675bd6ffd-qxgj8   1/1     Running   0          93s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1               443/TCP   10d
service/mychart      ClusterIP   10.109.119.97           80/TCP    93s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mychart   1/1     1            1           93s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mychart-6675bd6ffd   1         1         1       93s
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz  mychart  mychart-0.1.0.tgz  redis-cluster  redis-cluster-6.3.2.tgz

测试访问

[root@server1 helm]# curl 10.109.119.97 
Hello MyApp | Version: v1 | "hostname.html">Pod Name</a>

5.应用升级

进入mychart目录，更换镜像myapp：v2，tag：v1 版本号 0.2.0，检测无误打包为mychart-0.2.0.tgz

[root@server1 helm]# cd mychart/
[root@server1 mychart]# vim values.yaml 
[root@server1 mychart]# vim Chart.yaml 
[root@server1 mychart]# cd ..
[root@server1 helm]# helm lint mychart
==> Linting mychart
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
[root@server1 helm]# helm package  mychart
Successfully packaged chart and saved it to: /root/helm/mychart-0.2.0.tgz
[root@server1 helm]# ls
helm-push_0.9.0_linux_amd64.tar.gz  mychart  mychart-0.1.0.tgz  mychart-0.2.0.tgz  redis-cluster  redis-cluster-6.3.2.tgz

push mychart-0.2.0.tgz至本地chart仓库 westos

[root@server1 helm]# helm push  mychart-0.2.0.tgz  westos --insecure -u admin -p westos
Pushing mychart-0.2.0.tgz to westos...
Done.

更新repo，查询mychart，默认显示最新版本

[root@server1 helm]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "westos" chart repository
^C
[root@server1 helm]# helm search repo mychart
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                
westos/mychart	0.2.0        	v2         	A Helm chart for Kubernetes

查看mychart所有版本

[root@server1 helm]# helm search repo mychart -l
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                
westos/mychart	0.2.0        	v2         	A Helm chart for Kubernetes
westos/mychart	0.1.0        	v1         	A Helm chart for Kubernetes

执行更新，当前部署mychart自动升级至0.2.0 v2版本

[root@server1 helm]# helm upgrade  mychart  westos/mychart
Release "mychart" has been upgraded. Happy Helming!
NAME: mychart
LAST DEPLOYED: Wed Aug  4 04:00:52 2021
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
    export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=mychart,app.kubernetes.io/instance=mychart" -o jsonpath="{.items[0].metadata.name}")
    export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
    echo "Visit http://127.0.0.1:8080 to use your application"
    kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT

获取访问ip，测试访问，证实版本为v2，升级成功

    [root@server1 helm]# kubectl  get pod
    NAME                       READY   STATUS    RESTARTS   AGE
    mychart-57fb89f5cc-h69xr   1/1     Running   0          27s
    [root@server1 helm]# kubectl  get svc
    NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   10d
    mychart      ClusterIP   10.109.119.97   <none>        80/TCP    5m52s
    [root@server1 helm]# curl 10.109.119.97
    Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
    [root@server1 helm]#

6.应用卸载

[root@server1 nfs-client]# helm list --all-namespaces 
NAME   	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART        	APP VERSION
mychart	default  	2       	2021-08-04 04:00:52.542465557 -0400 EDT	deployed	mychart-0.2.0	v2         
[root@server1 nfs-client]# helm uninstall  mychart 
release "mychart" uninstalled
[root@server1 nfs-client]# helm list --all-namespaces 
NAME	NAMESPACE	REVISION	UPDATED	STATUS	CHART	APP VERSION