k8s概念

動機

可以先去看k8s筆記

k8s cluster & namespace

namespace == virtual cluster
- cluster = master + workers

k8s Volume

StorageClass
- a set of disk partitions (a physical disk)
  - provisioner (What is this disk?)
    - parameters
- => PersistentVolume
  - a disk partition
  - => PersistentVolumeClaim
    - a folder with some limitation
      - size
      - What should k8s do when this pod has been deleted?
common arguments
- reclaimPolicy
- accessModes

k8s components

ConfigMap & Secret
- dict (key-value)
- dict[key] = value
  - Serect will encrypt!!
  - ConfigMap will not
- dict[key]
  - valueFrom
    - configMapKeyRef or secretKeyRef
      - name
      - key
DaemonSet
- run pods on each nodes
  - pods are daemons for a node
- 每個Pods都會共用同一個掛載硬碟
- no rollback
- 更新方面，DaemonSet更新方式為一個Pod先關閉，而後才起新的Pod
StatefulSets
- pod有唯一編號
- 使用PVC時，會有各自獨立的Storage
- 佈署與擴展時，每個 pod 的產生都是有其順序且逐一完成
- no rollback
Job
- pods will be cleaned up after pods’ tasks have finished
Ingress & Service
- Ingress: reverse proxy (url to service)
- Service: port forward & DNS (url)
  - LoadBalancer
    - routing (like reverse proxy, ip&port to pods)
  - NodePort
    - port forward
Deployment
- -> ReplicaSet -> Pod
- Health Check
- Rollback
- 每個Pods都會共用同一個掛載硬碟

k8s components management

how to find a component fulfilling requests?
- kind
  - other arguments
- label
  - key:value
- namespace
selector
- key: value

chain

service -> deployment(s) (with configmap) -> service -> …

k8s & Istio (define how to connect to pods!!)

k8s:
- proxy on Node
  - dispatch(by iptables) to pods in this Node
istio
- proxy on Service (in each pod)
  - like Ingress, but
    - proxy control pods directly
    - 可以實現內對外的gateway ()

這裡可以順便嘴一下，k8s的pod創完後就不能加pod，所以如果conntainer出事沒辦法插入probe，這樣與kprobe有什麼不一樣(所以才有像是istio之類的東西出現) issue

helm

yaml compiler & dict manager (control content by some conditions)
- with
  - some artifacts (reuse others’ yaml)
  - DSL (no!!)

k8s’s device plugin

就是driver in traditional OS
用法就是把device mount到對應的path (/dev/xxx)

Interface

Container Runtime Interface (CRI) (k8s&istio , docker&containerd)
- Pod Life Cycle (Add/Delete)
- Pod Status
- Image management
- Container Operations (attatch/exec)
Container Network Interface (CNI)
- 上網能力，通常都會希望能夠有連接外網的能力
  - mutilcast
  - 硬體加速 (DPDK,…)
    - 但k8s的proxy與相關功能(service discovery)都是走linux network stack!!
    - CNI要重新實現k8s的proxy與相關功能!!
- 分配 IP 地址，幫每個 Pod 找一個獨立不重複的 IP
- Network Policy， kubernetes 內部有 Network Policy 去限制 Pod 與 Pod 之間的網路傳輸
Container Storage Interface(CSI)
- 支援動態配置或是靜態配置
  - 其實對應到 k8s 就是 storageclass 以及 PersistentVolume 的概念
- 支援Attach，該 Node 有能力可以跟該儲存方案連接起來
- 支援Mount，該儲存空間給實體化後可以透過 Filesystem 去存取。
- 支援 Block Device (EBS) 或是可直接 Mountable Volumes (大家應該大部分都用這個)
- 支援本地儲存設備解決方案，譬如 (device mapper, lvm)，這些又是以前的儲存議題了
- 支援創建/刪除快照
- 可提供從先前創造的快照復原出任何空間

   CreateVolume +------------+ DeleteVolume
 +------------->|  CREATED   +--------------+
 |              +---+----+---+              |
 |       Controller |    | Controller       v
+++         Publish |    | Unpublish       +++
|X|          Volume |    | Volume          | |
+-+             +---v----+---+             +-+
                | NODE_READY |
                +---+----^---+
               Node |    | Node
              Stage |    | Unstage
             Volume |    | Volume
                +---v----+---+
                |  VOL_READY |
                +------------+
               Node |    | Node
            Publish |    | Unpublish
             Volume |    | Volume
                +---v----+---+
                | PUBLISHED  |
                +------------+

rdac

Role, ClusterRole 就是能使用什麼資源 RoleBinding就是把Role連到User或是ServiceAccount

這樣pod就能根據rbac的規定去操作k8s的資源，是根據餵到master的token去查權限

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep ACCOUNT | awk '{print $1}')

下面是針對gitlab ci runner的rbac

kind: ServiceAccount
apiVersion: v1
metadata:
  name: ci
  namespace: ci
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ci
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch","delete"]
- apiGroups: [""]
  resources: ["pods/binding"]
  verbs: ["get", "list", "watch", "create"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "watch", "create"]
- apiGroups: [""]
  resources: ["pods/status"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["apps"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["list", "get", "create", "delete", "update"]      
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["list", "get", "create", "delete", "update"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["list", "get", "create", "delete", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ci
subjects:
- kind: ServiceAccount
  name: ci
  namespace: ci
roleRef:
  kind: ClusterRole
  name: ci
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ci-default
subjects:
- kind: ServiceAccount
  name: default
  namespace: ci
roleRef:
  kind: ClusterRole
  name: ci
  apiGroup: rbac.authorization.k8s.io

多nodes

創master: kubeadm init
拿master的token: kubeadm token list
join: kubeadm join --discovery-token-unsafe-skip-ca-verification --token=TOKEN MASTER_IP:6443

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

範例

下面是用Secret, ConfigMap, Service, Deployment 簡單創一個DB

apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
type: Opaque
data:
  POSTGRES_PASSWORD: MTIzNDU2Nzg5Cg==
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_COMMON: pg2
  POSTGRES_PASSWORD: "123456789"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:12.9-alpine
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_COMMON
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: POSTGRES_COMMON
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: POSTGRES_PASSWORD
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  ports:
   - port: 5432
  selector:
   app: postgres

secret as files

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
data:
  username: YWRtaW4=
  password: YTYyZmpiZDM3OTQyZGNz

spec:
  volumes:
  - name: secret-volume
    secret:
      secretName: test-secret
  containers:
    - name: test-container
      image: alpine:latest
      command: ["sleep", "9999"]
      volumeMounts:
          - name: secret-volume
            mountPath: /etc/secret-volume

這樣在/etc/secret-volume/username就能直接cat出username

kubectl exec shell-demo -- cat /etc/secret-volume/username kubectl exec -it my-pod main-app -- /bin/bash

動機#

k8s cluster & namespace#

k8s Volume#

k8s components#

k8s components management#

chain#

k8s & Istio (define how to connect to pods!!)#

helm#

k8s’s device plugin#

Interface#

rdac#

多nodes#

範例#

secret as files#

Ref#

動機