之前日志收集都很正常,从11点开始日志查询发现一直没有日志进来,但是终端看是有日志的

这是fluentbit-operator容器日志
2021-05-27T12:26:20.836Z INFO controller-runtime.metrics metrics server is starting to listen {“addr”: “:8080”}

2021-05-27T12:26:20.836Z INFO setup starting manager

2021-05-27T12:26:20.836Z INFO controller-runtime.manager starting metrics server {“path”: “/metrics”}

2021-05-27T12:26:22.337Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.337Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbit”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.437Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.438Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbit”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.438Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbit”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.438Z INFO controller-runtime.controller Starting Controller {“controller”: “fluentbit”}

2021-05-27T12:26:22.438Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.438Z INFO controller-runtime.controller Starting workers {“controller”: “fluentbit”, “worker count”: 1}

2021-05-27T12:26:22.538Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.639Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.736Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbit”, “request”: “kubesphere-logging-system/fluent-bit”}

2021-05-27T12:26:22.739Z INFO controller-runtime.controller Starting EventSource {“controller”: “fluentbitconfig”, “source”: “kind source: /, Kind=”}

2021-05-27T12:26:22.839Z INFO controller-runtime.controller Starting Controller {“controller”: “fluentbitconfig”}

2021-05-27T12:26:22.839Z INFO controller-runtime.controller Starting workers {“controller”: “fluentbitconfig”, “worker count”: 1}

2021-05-27T12:26:23.146Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:23.150Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/tail-auditing”}

2021-05-27T12:26:23.154Z ERROR controller-runtime.controller Reconciler error {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/tail-events”, “error”: "Operation cannot be fulfilled on secrets \“fluent-bit-config\”: the object has been modified; please apply your changes to the latest version and try again"}

github.com/go-logr/zapr.(*zapLogger).Error

/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211

k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152

k8s.io/apimachinery/pkg/util/wait.JitterUntil

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153

k8s.io/apimachinery/pkg/util/wait.Until

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88

2021-05-27T12:26:24.160Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/docker”}

2021-05-27T12:26:24.163Z ERROR controller-runtime.controller Reconciler error {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/kubelet”, “error”: "Operation cannot be fulfilled on secrets \“fluent-bit-config\”: the object has been modified; please apply your changes to the latest version and try again"}

github.com/go-logr/zapr.(*zapLogger).Error

/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211

k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152

k8s.io/apimachinery/pkg/util/wait.JitterUntil

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153

k8s.io/apimachinery/pkg/util/wait.Until

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88

2021-05-27T12:26:25.200Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/tail”}

2021-05-27T12:26:25.239Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/filter-events”}

2021-05-27T12:26:25.244Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/kubernetes”}

2021-05-27T12:26:25.339Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/systemd”}

2021-05-27T12:26:25.343Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/filter-auditing”}

2021-05-27T12:26:25.347Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/es-auditing”}

2021-05-27T12:26:25.351Z ERROR controller-runtime.controller Reconciler error {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/es-events”, “error”: "Operation cannot be fulfilled on secrets \“fluent-bit-config\”: the object has been modified; please apply your changes to the latest version and try again"}

github.com/go-logr/zapr.(*zapLogger).Error

/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211

k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152

k8s.io/apimachinery/pkg/util/wait.JitterUntil

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153

k8s.io/apimachinery/pkg/util/wait.Until

/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88

2021-05-27T12:26:26.365Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/es”}

2021-05-27T12:26:26.371Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.376Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/tail-events”}

2021-05-27T12:26:26.380Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/kubelet”}

2021-05-27T12:26:26.385Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/es-events”}

2021-05-27T12:26:26.441Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.446Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.451Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.456Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.460Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.464Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.483Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.492Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.533Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.539Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.543Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.548Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.553Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.558Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.633Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.639Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.644Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.649Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.654Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

2021-05-27T12:26:26.735Z DEBUG controller-runtime.controller Successfully Reconciled {“controller”: “fluentbitconfig”, “request”: “kubesphere-logging-system/fluent-bit-config”}

我删除fluentbit-operator容器也是不行

    crazylsq 你用内置还是外置es,es集群正常吗?内置可以用看看集群状态

     kubectl -n kubesphere-logging-system exec -it elasticsearch-logging-data-0 -- curl localhost:9200/_cluster/health?pretty

      另外我早上看 又有日志进来了,但是查看最近1天的,可以看到 5-27 11:00 - 5-28 08 这个时间段是空白的

        crazylsq 估计这个时间段日志不可写或者其他什么原因,查看下这段时间内es的日志。默认日志组件是会有定期清理的,如果磁盘容量满了,就可能发生不可写状态,看是不是这个原因。

          3 个月 后

          xiaolidw23456 可以edit cc ks-installer里面修改,也可以修改pvc size大小、增加节点数量等

          ps:文档比较全面,多看文档。

            xiaolidw23456 当然可以。不过,为了避免ks-installer同步后,会修改你手动修改pvc的问题,建议从edit cc ks-installer来增加容量。

            5 天 后

            xiaolidw23456 删除了,restart ks-installer了么?如果是bug,方便再github上记录下么?
            可以先手动修改一下。