Production Recommendations

Best practices and recommended settings when going production.

PV settings

Below settings are recommended for a production environment.

• Configure more readable names for PV directory
• Enable Automatic Mount Point Recovery
• The --writeback option is strongly advised against, as it can easily cause data loss especially when used inside containers, if not properly managed. See "Write Cache in Client (Community Edition)" and "Write Cache in Client (Cloud Service)".
• When cluster is low on resources, refer to optimization techniques in Resource Optimization.

Collect mount pod logs using EFK

Troubleshooting CSI Driver usually involves reading mount pod logs, if checking mount pod logs in real time isn't enough, consider deploying an EFK (Elasticsearch + Fluentd + Kibana) stack (or other suitable systems) in Kubernetes Cluster to collect pod logs for query. Taking EFK for example:

• Elasticsearch: index logs and provide a complete full-text search engine, which can facilitate users to retrieve the required data from the log. For installation, please refer to the official documentation.
• Fluentd: fetch container log files, filter and transform log data, and then deliver the data to the Elasticsearch cluster. For installation, please refer to the official documentation.
• Kibana: visual analysis of logs, including log search, processing, and gorgeous dashboard display, etc. For installation, please refer to the official documentation.

Mount pod is labeled app.kubernetes.io/name: juicefs-mount. Add below config to the Fluentd configuration:

<filter kubernetes.**>
  @id filter_log
  @type grep
  <regexp>
    key $.kubernetes.labels.app_kubernetes_io/name
    pattern ^juicefs-mount$
  </regexp>
</filter>

And add the following parser plugin to the Fluentd configuration file:

<filter kubernetes.**>
  @id filter_parser
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>