操作系统信息
例如：虚拟机，Centos7.9，4C/8G

Kubernetes版本信息
v1.20.4

容器运行时
docker 19.03.9

KubeSphere版本信息
在已有k8s集群上在线安装kubesphere v3.1.1

longhorn作为默认存储，使用kubesphere自带的自定义监控功能对longhorn中的相关指标进行监控

使用kubesphere的监控功能，需要提前开启监控的可插拔组件，可参考https://v3-1.docs.kubesphere.io/zh/docs/project-user-guide/custom-application-monitoring/introduction/

自定义longhorn监控面板

安装kubesphere时会默认安装其自带的Prometheus，但是这个Prometheus没有监控longhorn相关指标，需要安装一个longhorn serviceMonitor

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

namespace: monitoring #### 注意修改ns

labels:

spec:

selector:

matchLabels:

app: longhorn-manager

namespaceSelector:

matchNames:

- longhorn-system

endpoints:

- port: manager

到kubesphere的自定义监控页面创建新的自定义监控面板，其中选用longhorn的相关指标即可

根据需求，可以自定义监控指标的展示方式

也可以保存模板之后，直接编辑其yaml文件，可以使用如下代码：

apiVersion: monitoring.kubesphere.io/v1alpha1

kind: ClusterDashboard

metadata:

annotations:

kubesphere.io/creator: admin

kubesphere.io/description: longhorn相关指标

spec:

datasource: prometheus

panels:

- format: none

id: 2

targets:

- expr: |

count(longhorn_volume_capacity_bytes) OR on() vector(0)

title: Total Number Of Volumes

type: singlestat

- format: none

id: 4

targets:

- expr: count(longhorn_volume_robustness==1) OR on() vector(0)

title: Number Of Healthy Volumes

type: singlestat

- format: none

id: 5

targets:

- expr: count(longhorn_volume_robustness==2) OR on() vector(0)

title: Number Of Degraded Volumes

type: singlestat

- format: none

id: 6

targets:

- expr: count(longhorn_volume_robustness==3) OR on() vector(0)

title: Number Of Fault Volumes

type: singlestat

- format: none

id: 7

targets:

- expr: count(longhorn_volume_state==2) OR on() vector(0)

title: Number Of Attached Volumes

type: singlestat

- format: none

id: 8

targets:

- expr: count(longhorn_volume_state==3) OR on() vector(0)

title: Number Of Detached Volumes

type: singlestat

- format: none

id: 12

targets:

- expr: avg(longhorn_node_count_total) OR on() vector(0)

title: Total Number Of Nodes

type: singlestat

- format: none

id: 13

targets:

- expr: >-

count(longhorn_node_status{condition=“allowScheduling”}==0) OR on()

vector(0)

title: Number Of Disabled Nodes

type: singlestat

- format: none

id: 14

targets:

- expr: >-

(count(longhorn_node_status{condition=“schedulable”}==1) OR on()

vector(0)) -

(count(longhorn_node_status{condition=“allowScheduling”}==0) OR on()

vector(0))

title: Number Of Schedulable Nodes

type: singlestat

- format: none

id: 15

targets:

- expr: >-

(avg(longhorn_node_count_total) OR on() vector(0)) -

(count(longhorn_node_status{condition=“ready”}==1) OR on()

vector(0))

title: Number Of Failed Nodes

type: singlestat

- id: 23

title: CPU & MEMORY

type: row

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 24

lines: true

targets:

- expr: >-

(longhorn_node_cpu_usage_millicpu /

longhorn_node_cpu_capacity_millicpu) * 100

legendFormat: ‘{{node}}’

refId: 1

step: 1m

title: Node CPU Usage/Capacity

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 25

lines: true

targets:

- expr: >-

( longhorn_node_memory_usage_bytes /

longhorn_node_memory_capacity_bytes ) * 100

legendFormat: ‘{{node}}’

refId: 1

step: 1m

title: Node Memory Usage/Capacity

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Instance managers are pods that contains the engine and replica

processes of Longhorn volumes. See more at

https://longhorn.io/docs/1.0.2/concepts/#11-the-longhorn-manager-and-the-longhorn-engine

id: 26

lines: true

targets:

- expr: longhorn_instance_manager_cpu_usage_millicpu

legendFormat: ‘{{instance_manager}}’

refId: 1

step: 1m

title: Instance Manager CPU Usage

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Instance managers are pods that contains the engine and replica

processes of Longhorn volumes. See more at

https://longhorn.io/docs/1.0.2/concepts/#11-the-longhorn-manager-and-the-longhorn-engine

id: 27

lines: true

targets:

- expr: >-

(longhorn_instance_manager_cpu_usage_millicpu/longhorn_instance_manager_cpu_requests_millicpu)*100

legendFormat: ‘{{instance_manager}}’

refId: 1

step: 1m

title: Instance Manager CPU Usage/Request

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Instance managers are pods that contains the engine and replica

processes of Longhorn volumes. See more at

https://longhorn.io/docs/1.0.2/concepts/#11-the-longhorn-manager-and-the-longhorn-engine

id: 28

lines: true

targets:

- expr: longhorn_instance_manager_memory_usage_bytes

legendFormat: ‘{{instance_manager}}’

refId: 1

step: 1m

title: Instance Manager Memory Usage

type: graph

yaxes:

- format: Byte

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Longhorn manager pods manage the control plane of the Longhorn system.

e.g. Volume scheduling, attaching, detaching, backup, etc,..

id: 29

lines: true

targets:

- expr: longhorn_manager_cpu_usage_millicpu

legendFormat: ‘{{manager}}’

refId: 1

step: 1m

title: Longhorn Manager CPU Usage

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Longhorn manager pods manage the control plane of the Longhorn system.

e.g. Volume scheduling, attaching, detaching, backup, etc,..

id: 30

lines: true

targets:

- expr: longhorn_manager_memory_usage_bytes

legendFormat: ‘{{manager}}’

refId: 1

step: 1m

title: Longhorn Manager Memory Usage

type: graph

yaxes:

- format: Byte

- id: 9

title: VOLUMES

type: row

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: >-

Note that Longhorn volume actual size is not the size of the filesystem

inside a Longhorn volume. See more at :

https://longhorn.io/docs/1.0.2/volumes-and-nodes/volume-size/#volume-actual-size

id: 1

lines: true

targets:

- expr: >-

( (avg by (volume) (longhorn_volume_actual_size_bytes))/ (avg by

(volume) (longhorn_volume_capacity_bytes)) ) *100

legendFormat: ‘{{volume}}’

refId: 1

step: 1m

title: Volume Actual Size/Capacity

type: graph

yaxes:

- format: percent (0-100)

- bars: true

colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: The capacity of each Longhorn volume

id: 3

stack: true

targets:

- expr: avg by (volume) (longhorn_volume_capacity_bytes)

legendFormat: ‘{{volume}}’

refId: 1

step: 1m

title: Volume Capacity

type: graph

yaxes:

- format: Byte

- id: 10

title: NODES

type: row

- bars: true

colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 18

targets:

- expr: longhorn_node_storage_capacity_bytes

legendFormat: ‘{{node}}’

refId: 1

step: 1m

title: Node Capacity

type: graph

yaxes:

- format: Byte

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 16

lines: true

stack: true

targets:

- expr: >-

(longhorn_node_storage_usage_bytes/longhorn_node_storage_capacity_bytes)

* 100

legendFormat: ‘{{node}}’

refId: 1

step: 1m

title: Node Storage Usage/Capacity

type: graph

yaxes:

- format: percent (0-100)

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 19

lines: true

targets:

- expr: count by (node) (longhorn_volume_state==2)

legendFormat: ‘{{node}}’

refId: 1

step: 1m

title: Number of Volumes Per Node

type: graph

yaxes:

- format: none

- id: 20

title: DISKS

type: row

- colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

description: The capacity of each Longhorn volume

id: 21

lines: true

targets:

- expr: (longhorn_disk_usage_bytes/longhorn_disk_capacity_bytes)*100

legendFormat: ‘{{disk}}’

refId: 1

step: 1m

title: Disk Space Usage

type: graph

yaxes:

- format: percent (0-100)

- bars: true

colors:

- ‘#60acfc’

- ‘#23c2db’

- ‘#64d5b2’

- ‘#d5ec5a’

- ‘#ffb64e’

- ‘#fb816d’

- ‘#d15c7f’

id: 22

stack: true

targets:

- expr: longhorn_disk_capacity_bytes

legendFormat: ‘{{disk}}’

refId: 1

step: 1m

title: Disk Capacity

type: graph

yaxes:

- format: Byte

time:

from: now-30m

to: now

title: Custom Monitoring