Ubuntu 22.04部署Prometheus Operator监控K8s集群-育师

1. 环境准备与基础配置

在Ubuntu 22.04 LTS上部署Prometheus Operator前，需要确保基础环境符合要求。我推荐使用全新安装的Ubuntu 22.04系统，避免已有服务造成端口冲突。

1.1 系统级优化

首先执行系统更新和基础工具安装：

sudo apt update && sudo apt upgrade -y sudo apt install -y curl wget git gnupg2 software-properties-common apt-transport-https ca-certificates

调整内核参数以优化Kubernetes网络性能：

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sudo sysctl --system

提示：对于生产环境，建议单独准备监控节点，避免与其他服务竞争资源。Prometheus Operator对磁盘IO要求较高，使用SSD存储可获得更好性能。

1.2 Kubernetes集群部署

使用kubeadm部署Kubernetes集群（单节点测试环境配置）：

sudo apt install -y docker.io sudo systemctl enable --now docker curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt update sudo apt install -y kubelet=1.28.0-00 kubeadm=1.28.0-00 kubectl=1.28.0-00 sudo apt-mark hold kubelet kubeadm kubectl sudo kubeadm init --pod-network-cidr=10.244.0.0/16 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config

安装网络插件（这里选择Flannel）：

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

2. Prometheus Operator核心组件部署

2.1 使用Helm安装Operator

添加Prometheus社区Helm仓库：

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update

创建监控专用命名空间：

kubectl create namespace monitoring

安装Prometheus Operator（使用自定义values优化配置）：

cat <<EOF > custom-values.yaml prometheus: prometheusSpec: scrapeInterval: 30s evaluationInterval: 30s retention: 15d resources: requests: memory: 2Gi cpu: 1 limits: memory: 4Gi cpu: 2 storageSpec: volumeClaimTemplate: spec: storageClassName: "standard" accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi alertmanager: alertmanagerSpec: replicas: 2 resources: requests: memory: 512Mi cpu: 500m limits: memory: 1Gi cpu: 1 EOF helm install prometheus-operator prometheus-community/kube-prometheus-stack \ -n monitoring \ -f custom-values.yaml

2.2 验证安装状态

检查Pod运行状态：

kubectl get pods -n monitoring -w

预期输出应包含以下关键组件：

prometheus-prometheus-operator-prometheus-0 3/3 Running prometheus-operator-operator-xxx 1/1 Running alertmanager-prometheus-operator-alertmanager-0 2/2 Running grafana-xxx 1/1 Running

获取Prometheus Web UI访问地址：

kubectl port-forward -n monitoring svc/prometheus-operator-prometheus 9090:9090

3. 高级配置与优化策略

3.1 自定义监控目标配置

创建ServiceMonitor监控Kubernetes API Server：

cat <<EOF | kubectl apply -f - apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kube-apiserver namespace: monitoring labels: release: prometheus-operator spec: jobLabel: component endpoints: - port: https scheme: https bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token tlsConfig: insecureSkipVerify: true namespaceSelector: matchNames: - default selector: matchLabels: component: apiserver provider: kubernetes EOF

3.2 资源配额与自动扩缩

配置Prometheus垂直自动扩缩（VPA）：

kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/vertical-pod-autoscaler/hack/vpa-beta1-crd.yaml kubectl apply -n monitoring -f - <<EOF apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: prometheus-vpa spec: targetRef: apiVersion: "apps/v1" kind: StatefulSet name: prometheus-prometheus-operator-prometheus updatePolicy: updateMode: "Auto" EOF

3.3 长期存储方案

集成Thanos实现长期存储（可选）：

helm repo add bitnami https://charts.bitnami.com/bitnami helm install thanos bitnami/thanos \ --set objstoreConfig=thanos-config \ --set existingObjstoreSecret=thanos-objstore-secret \ -n monitoring

4. 告警与可视化配置

4.1 自定义告警规则

创建关键告警规则示例：

cat <<EOF | kubectl apply -f - apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kubernetes-alerts namespace: monitoring labels: release: prometheus-operator spec: groups: - name: node.rules rules: - alert: NodeCPUHigh expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 10m labels: severity: warning annotations: summary: "High CPU usage on {{ $labels.instance }}" description: "{{ $labels.instance }} CPU usage is {{ $value }}%" EOF

4.2 Grafana仪表板管理

导入常用Kubernetes仪表板：

DASHBOARD_ID=315 kubectl apply -n monitoring -f - <<EOF apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: kubernetes-cluster labels: app: grafana spec: json: | $(curl -sSL https://grafana.com/api/dashboards/${DASHBOARD_ID}/revisions/latest/download | jq -c) EOF

配置Grafana数据源自动发现：

kubectl patch -n monitoring prometheus prometheus-operator-prometheus --type merge -p '{ "spec": { "grafana": { "additionalDataSources": [{ "name": "Loki", "type": "loki", "url": "http://loki.monitoring.svc.cluster.local:3100", "access": "proxy" }] } } }'

5. 生产环境最佳实践

5.1 高可用配置

配置Alertmanager集群与持久化存储：

kubectl patch -n monitoring alertmanager prometheus-operator-alertmanager --type merge -p '{ "spec": { "replicas": 3, "storage": { "volumeClaimTemplate": { "spec": { "storageClassName": "standard", "resources": { "requests": { "storage": "10Gi" } } } } } } }'

5.2 安全加固

启用RBAC和网络策略：

kubectl apply -n monitoring -f - <<EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: prometheus-allow-only-grafana spec: podSelector: matchLabels: app.kubernetes.io/name: prometheus ingress: - from: - podSelector: matchLabels: app.kubernetes.io/name: grafana EOF

5.3 性能调优

优化Prometheus抓取配置：

kubectl patch -n monitoring prometheus prometheus-operator-prometheus --type merge -p '{ "spec": { "scrapeInterval": "1m", "scrapeTimeout": "10s", "evaluationInterval": "1m", "retention": "30d", "walCompression": true, "query": { "lookbackDelta": "5m", "maxConcurrency": 20, "timeout": "2m" } } }'

我在实际生产环境中发现，当监控目标超过5000个时，需要特别注意以下配置：