Большинство читателей уже так или иначе пробовали устанавливать Kubernetes с помощью kubespray или других средств автоматизации, доступных у большинства поставщиков облачных решений. Можно также всё делать с нуля с использованием kubectl, ведь сам процесс в принципе достаточно неплохо обкатан. Но что если нужна тестовая лаборатория, или даже не сильно крупная серверная ферма, построенная на собственных серверах с ограниченными ресурсами, либо просто на устаревшем оборудовании?
Можно воспользоваться k3s, минималистичной сборкой Kubernetes, особенностью которой является возможность работать как на мелких компьютерах типа Raspberry Pi и небольших виртуальных машинах, так и на обычных железных серверах. Данной статьей начинается небольшой цикл статей о построении домашнего или небольшого производственного кластера Kubernetes, в частности в этой статье будет рассмотрена установка полноценного кластера Kubernetes on-premise на основе k3s, в кластере будет установлен ingress на базе проекта Contour, балансировщик Metallb, панель управления Headlamp, система управления томами Longhorn вместе с панелью управления.
Такой выбор составных частей обусловлен следующими соображениями:
Операционная система Debian 10 с последними обновлениями, отключен swap. Имена серверов k3s1, k3s2 и k3s3, запись в днс *.example.com указывает на свободный публичный ip от metallb. Установлен исполняемый файл arkade, с помощью которого можно автоматизировать установку других программ, в частности k3sup, а также установить приложения в кластер Kubernetes. Дополнительно настроен доступ к серверам по ssh по ключу для пользователя root.
#!/bin/bash
#set -e
export NODE_1="k3s1"
export NODE_2="k3s2"
export NODE_3="k3s3"
export USER=root
# The first server starts the cluster
k3sup install \
--cluster \
--user $USER \
--host $NODE_1 \
--k3s-extra-args '--disable servicelb --disable traefik '
# The second node joins
k3sup join \
--server \
--host $NODE_2 \
--user $USER \
--server-user $USER \
--server-host $NODE_1
# The third node joins
k3sup join \
--server \
--host $NODE_3 \
--user $USER \
--server-user $USER \
--server-host $NODE_1
Процесс установки:
$ ./bootstrap-k8s.sh
Running: k3sup install
2021/02/27 15:28:38 k3s1
Public IP: k3s1
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Result: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Saving file to: /home/user/kubeconfig
# Test your cluster with:
export KUBECONFIG=/home/user/kubeconfig
kubectl config set-context default
kubectl get node -o wide
Running: k3sup join
Server IP: k3s1
K101de86a6148782e7d364df3657cbea52faf809c688be697c5d5d37ccb76c4fdae::server:021aeff4097220f3dc94145842b7bf40
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Output: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
Running: k3sup join
Server IP: k3s1
K101de86a6148782e7d364df3657cbea52faf809c688be697c5d5d37ccb76c4fdae::server:021aeff4097220f3dc94145842b7bf40
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Output: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
$ export KUBECONFIG=kubeconfig
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k3s1 Ready etcd,master 92s v1.19.8+k3s1
k3s2 Ready etcd,master 60s v1.19.8+k3s1
k3s3 Ready etcd,master 30s v1.19.8+k3s1
После установки меняем адрес сервера в конфигурационном файле и подчищаем кластер:
$ sed s/127.0.0.1/k3s1/ -i kubeconfig
$ export KUBECONFIG=kubeconfig
$ kubectl delete svc/traefik-prometheus -n kube-system
service "traefik-prometheus" deleted
$ kubectl delete svc/traefik -n kube-system
service "traefik" deleted
$ kubectl delete deployment.apps/traefik -n kube-system
deployment.apps "traefik" deleted
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- <тут надо вписать ip, используемый для metallb>/32
Установка:
$ kubectl apply -f metallb.yaml
namespace/metallb-system created
configmap/config created
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
$ kubectl apply -f https://projectcontour.io/quickstart/contour.yaml
namespace/projectcontour created
serviceaccount/contour created
serviceaccount/envoy created
configmap/contour created
customresourcedefinition.apiextensions.k8s.io/extensionservices.projectcontour.io created
customresourcedefinition.apiextensions.k8s.io/httpproxies.projectcontour.io created
customresourcedefinition.apiextensions.k8s.io/tlscertificatedelegations.projectcontour.io created
serviceaccount/contour-certgen created
rolebinding.rbac.authorization.k8s.io/contour created
role.rbac.authorization.k8s.io/contour-certgen created
job.batch/contour-certgen-v1.13.0 created
clusterrolebinding.rbac.authorization.k8s.io/contour created
clusterrole.rbac.authorization.k8s.io/contour created
service/contour created
service/envoy created
deployment.apps/contour created
daemonset.apps/envoy created
Дополнительной настройки в данном случае не требуется.
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Для его работы необходима учетная запись с доступом к кластеру, создадим её и выдадим доступ:
$ kubectl -n kube-system create serviceaccount headlamp-admin
serviceaccount/headlamp-admin created
$ kubectl create clusterrolebinding headlamp-admin --serviceaccount=kube-system:headlamp-admin --clusterrole=cluster-admin
clusterrolebinding.rbac.authorization.k8s.io/headlamp-admin created
Для доступа к интерфейсу можно его «прокинуть» наружу с помощью ingress, для этого создадим файл headlamp-ingress.yaml с следующим содержимым (настройка шифрования ssl опущена для простоты):
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: headlamp
namespace: kube-system
annotations:
kubernetes.io/tls-acme: "false"
spec:
rules:
-
host: headlamp.example.com
http:
paths:
-
path: /
backend:
serviceName: headlamp
servicePort: 80
Добавление ingress:
$ kubectl apply -f headlamp-ingress.yaml
ingress.extensions/headlamp created
Для того, чтобы получить доступ к интерфейсу, нужно знать токен. Чтобы его получить, можно использовать следующую команду:
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secrets | grep -oP headlamp-admin-token-[a-z0-9]+) | awk '/^token/ {print $2}'
Вводим полученный токен в форму headlamp и можем управлять кластером из браузера:
k3s1 # df -h | grep longhorn
/dev/mapper/vg-longhorn 37G 74M 37G 1% /var/lib/longhorn
Также для корректной работы надо наличие iscsiadm, в случае Debian надо установить пакет open-iscsi на каждом сервере.
Установка Longhorn после подготовительных работ может быть произведена так:
$ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.0/deploy/longhorn.yaml
namespace/longhorn-system created
serviceaccount/longhorn-service-account created
clusterrole.rbac.authorization.k8s.io/longhorn-role created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-bind created
customresourcedefinition.apiextensions.k8s.io/engines.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/replicas.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/settings.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/volumes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/engineimages.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/nodes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/instancemanagers.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/sharemanagers.longhorn.io created
configmap/longhorn-default-setting created
podsecuritypolicy.policy/longhorn-psp created
role.rbac.authorization.k8s.io/longhorn-psp-role created
rolebinding.rbac.authorization.k8s.io/longhorn-psp-binding created
configmap/longhorn-storageclass created
daemonset.apps/longhorn-manager created
service/longhorn-backend created
deployment.apps/longhorn-ui created
service/longhorn-frontend created
deployment.apps/longhorn-driver-deployer created
Во время установки создается сервис с веб-интерфейсом, с помощью которого можно управлять томами. Сделаем его доступным извне, содержимое файла с описанием ingress такое:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: longhorn
namespace: longhorn-system
annotations:
kubernetes.io/tls-acme: "false"
spec:
rules:
-
host: longhorn.example.com
http:
paths:
-
path: /
backend:
serviceName: longhorn-frontend
servicePort: 80
Добавление ingress:
$ kubectl apply -f longhorn-ingress.yaml
ingress.extensions/longhorn created
Источник статьи: https://habr.com/ru/company/southbridge/blog/551214/
Можно воспользоваться k3s, минималистичной сборкой Kubernetes, особенностью которой является возможность работать как на мелких компьютерах типа Raspberry Pi и небольших виртуальных машинах, так и на обычных железных серверах. Данной статьей начинается небольшой цикл статей о построении домашнего или небольшого производственного кластера Kubernetes, в частности в этой статье будет рассмотрена установка полноценного кластера Kubernetes on-premise на основе k3s, в кластере будет установлен ingress на базе проекта Contour, балансировщик Metallb, панель управления Headlamp, система управления томами Longhorn вместе с панелью управления.
Такой выбор составных частей обусловлен следующими соображениями:
- использование новейших технологий с целью их изучения и обкатки для будущего внедрения на «больших» кластерах;
- ПО с наличием бесплатной версии, но лучше вообще с открытым исходным кодом;
- простота и легкость в установке, обслуживании и работе;
- отсутствие привязок к железу, в идеальном случае — полностью программная реализация.
Начальные условия
Нужно три сервера (можно виртуальные, но лучше железные), на каждом из них есть один публичный адрес, еще один публичный адрес будет использоваться для Metallb, для сервисов, которые будут развернуты в кластере.Операционная система Debian 10 с последними обновлениями, отключен swap. Имена серверов k3s1, k3s2 и k3s3, запись в днс *.example.com указывает на свободный публичный ip от metallb. Установлен исполняемый файл arkade, с помощью которого можно автоматизировать установку других программ, в частности k3sup, а также установить приложения в кластер Kubernetes. Дополнительно настроен доступ к серверам по ssh по ключу для пользователя root.
Установка k3s
Установка производится с помощью скрипта, использующего программу k3sup. Содержимое скрипта:#!/bin/bash
#set -e
export NODE_1="k3s1"
export NODE_2="k3s2"
export NODE_3="k3s3"
export USER=root
# The first server starts the cluster
k3sup install \
--cluster \
--user $USER \
--host $NODE_1 \
--k3s-extra-args '--disable servicelb --disable traefik '
# The second node joins
k3sup join \
--server \
--host $NODE_2 \
--user $USER \
--server-user $USER \
--server-host $NODE_1
# The third node joins
k3sup join \
--server \
--host $NODE_3 \
--user $USER \
--server-user $USER \
--server-host $NODE_1
Процесс установки:
$ ./bootstrap-k8s.sh
Running: k3sup install
2021/02/27 15:28:38 k3s1
Public IP: k3s1
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Result: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Saving file to: /home/user/kubeconfig
# Test your cluster with:
export KUBECONFIG=/home/user/kubeconfig
kubectl config set-context default
kubectl get node -o wide
Running: k3sup join
Server IP: k3s1
K101de86a6148782e7d364df3657cbea52faf809c688be697c5d5d37ccb76c4fdae::server:021aeff4097220f3dc94145842b7bf40
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Output: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
Running: k3sup join
Server IP: k3s1
K101de86a6148782e7d364df3657cbea52faf809c688be697c5d5d37ccb76c4fdae::server:021aeff4097220f3dc94145842b7bf40
[INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Logs: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
Output: [INFO] Finding release for channel v1.19
[INFO] Using v1.19.8+k3s1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v1.19.8+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
$ export KUBECONFIG=kubeconfig
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k3s1 Ready etcd,master 92s v1.19.8+k3s1
k3s2 Ready etcd,master 60s v1.19.8+k3s1
k3s3 Ready etcd,master 30s v1.19.8+k3s1
После установки меняем адрес сервера в конфигурационном файле и подчищаем кластер:
$ sed s/127.0.0.1/k3s1/ -i kubeconfig
$ export KUBECONFIG=kubeconfig
$ kubectl delete svc/traefik-prometheus -n kube-system
service "traefik-prometheus" deleted
$ kubectl delete svc/traefik -n kube-system
service "traefik" deleted
$ kubectl delete deployment.apps/traefik -n kube-system
deployment.apps "traefik" deleted
Установка и настройка Metallb
Для начала создаём namespace и configmap, содержимое файла metallb.yaml:apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- <тут надо вписать ip, используемый для metallb>/32
Установка:
$ kubectl apply -f metallb.yaml
namespace/metallb-system created
configmap/config created
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
secret/memberlist created
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
Установка и настройка Contour
Установка производится так:$ kubectl apply -f https://projectcontour.io/quickstart/contour.yaml
namespace/projectcontour created
serviceaccount/contour created
serviceaccount/envoy created
configmap/contour created
customresourcedefinition.apiextensions.k8s.io/extensionservices.projectcontour.io created
customresourcedefinition.apiextensions.k8s.io/httpproxies.projectcontour.io created
customresourcedefinition.apiextensions.k8s.io/tlscertificatedelegations.projectcontour.io created
serviceaccount/contour-certgen created
rolebinding.rbac.authorization.k8s.io/contour created
role.rbac.authorization.k8s.io/contour-certgen created
job.batch/contour-certgen-v1.13.0 created
clusterrolebinding.rbac.authorization.k8s.io/contour created
clusterrole.rbac.authorization.k8s.io/contour created
service/contour created
service/envoy created
deployment.apps/contour created
daemonset.apps/envoy created
Дополнительной настройки в данном случае не требуется.
Установка и настройка cert-manager
Для установки достаточно запустить следующую команду:$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.2.0/cert-manager.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
Установка и настройка Headlamp
Headlamp — это современный расширяемый веб-интерфейс для управления кластером k8s. Установим его.Для его работы необходима учетная запись с доступом к кластеру, создадим её и выдадим доступ:
$ kubectl -n kube-system create serviceaccount headlamp-admin
serviceaccount/headlamp-admin created
$ kubectl create clusterrolebinding headlamp-admin --serviceaccount=kube-system:headlamp-admin --clusterrole=cluster-admin
clusterrolebinding.rbac.authorization.k8s.io/headlamp-admin created
Для доступа к интерфейсу можно его «прокинуть» наружу с помощью ingress, для этого создадим файл headlamp-ingress.yaml с следующим содержимым (настройка шифрования ssl опущена для простоты):
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: headlamp
namespace: kube-system
annotations:
kubernetes.io/tls-acme: "false"
spec:
rules:
-
host: headlamp.example.com
http:
paths:
-
path: /
backend:
serviceName: headlamp
servicePort: 80
Добавление ingress:
$ kubectl apply -f headlamp-ingress.yaml
ingress.extensions/headlamp created
Для того, чтобы получить доступ к интерфейсу, нужно знать токен. Чтобы его получить, можно использовать следующую команду:
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secrets | grep -oP headlamp-admin-token-[a-z0-9]+) | awk '/^token/ {print $2}'
Вводим полученный токен в форму headlamp и можем управлять кластером из браузера:
Установка и настройка Longhorn
На каждом сервере надо создать и смонтировать каталог /var/lib/longhorn (используется по умолчанию), где будут храниться данные, у меня для таких целей смонтирован том LVM2 с файловой системой ext4:k3s1 # df -h | grep longhorn
/dev/mapper/vg-longhorn 37G 74M 37G 1% /var/lib/longhorn
Также для корректной работы надо наличие iscsiadm, в случае Debian надо установить пакет open-iscsi на каждом сервере.
Установка Longhorn после подготовительных работ может быть произведена так:
$ kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.0/deploy/longhorn.yaml
namespace/longhorn-system created
serviceaccount/longhorn-service-account created
clusterrole.rbac.authorization.k8s.io/longhorn-role created
clusterrolebinding.rbac.authorization.k8s.io/longhorn-bind created
customresourcedefinition.apiextensions.k8s.io/engines.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/replicas.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/settings.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/volumes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/engineimages.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/nodes.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/instancemanagers.longhorn.io created
customresourcedefinition.apiextensions.k8s.io/sharemanagers.longhorn.io created
configmap/longhorn-default-setting created
podsecuritypolicy.policy/longhorn-psp created
role.rbac.authorization.k8s.io/longhorn-psp-role created
rolebinding.rbac.authorization.k8s.io/longhorn-psp-binding created
configmap/longhorn-storageclass created
daemonset.apps/longhorn-manager created
service/longhorn-backend created
deployment.apps/longhorn-ui created
service/longhorn-frontend created
deployment.apps/longhorn-driver-deployer created
Во время установки создается сервис с веб-интерфейсом, с помощью которого можно управлять томами. Сделаем его доступным извне, содержимое файла с описанием ingress такое:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: longhorn
namespace: longhorn-system
annotations:
kubernetes.io/tls-acme: "false"
spec:
rules:
-
host: longhorn.example.com
http:
paths:
-
path: /
backend:
serviceName: longhorn-frontend
servicePort: 80
Добавление ingress:
$ kubectl apply -f longhorn-ingress.yaml
ingress.extensions/longhorn created
Заключение
В результате получилась заготовка кластера Kubernetes на основе k3s. В следующих статьях я буду опираться на полученный кластер, добавляя к нему новые возможности. В частности стоит настроить шифрование ssl для Headlamp и Longhorn, а также добавить basic auth к веб-интерфейсу Longhorn.Источник статьи: https://habr.com/ru/company/southbridge/blog/551214/