帝王谷资源网

nginx-ingress-controller日志持久化方案的解决
服务器 2024/12/27 佚名
3 2
帝王谷资源网 Design By www.wdxyy.com

最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。

nginx-ingress-controller的日志

nginx-ingress-controller的日志包括三个部分：

  • controller日志： 输出到stdout，通过启动参数中的–log_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
  • accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
  • errorlog：输出到stderr，配置方式与accesslog类似。

给controller日志落盘

  • 给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
  • 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。

controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。

给nginx日志落盘

修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。

accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：

$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
  su root list
  rotate 7
  daily
  maxsize 50M
  copytruncate
  missingok
  create 0644 www-data root
}

官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.

自动化ops

nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？

问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？

可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image，里面只执行如下脚本：

#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown -R $userID:$userID $logdir

脚本读取一些环境变量， 确认需要修改哪个目录，改成怎样的user group。

将脚本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.

再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。

一个deploy yaml如下：

---
apiVersion: v1
kind: Service
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 type: ClusterIP
 ports:
 - name: http
  port: 80
  targetPort: 80
  protocol: TCP
 - name: https
  port: 443
  targetPort: 443
  protocol: TCP
 selector:
  app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 ports:
 - port: 80
  targetPort: 8080
 selector:
  app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: default
 namespace: kube-system
spec:
 backend:
  serviceName: default-http-backend
  servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
 name: nginx-configuration
 namespace: kube-system
 labels:
  app: ingress-nginx
data:
 use-forwarded-headers: "true"
 # 此处配置nginx日志的重定向目标
 access-log-path: /var/log/nginx_ingress_controller/access.log
 error-log-path: /var/log/nginx_ingress_controller/error.log

---

# 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件
apiVersion: v1
data:
 nginx.log: |
  {{ user_nginx_log.host_path }}/access.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10M
    copytruncate
    missingok
    create 0644 root root
  }
  {{ user_nginx_log.host_path }}/error.log {
    rotate {{ user_nginx_log.rotate_count }}
    daily
    maxsize {{ user_nginx_log.rotate_size }}
    minsize 10M
    copytruncate
    missingok
    create 0644 root root
  }
kind: ConfigMap
metadata:
 name: nginx-ingress-logrotate
 namespace: kube-system
---

kind: ConfigMap
apiVersion: v1
metadata:
 name: tcp-services
 namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
 name: udp-services
 namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: nginx-ingress-serviceaccount
 namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: nginx-ingress-clusterrole
rules:
 - apiGroups:
   - ""
  resources:
   - configmaps
   - endpoints
   - nodes
   - pods
   - secrets
  verbs:
   - list
   - watch
 - apiGroups:
   - ""
  resources:
   - nodes
  verbs:
   - get
 - apiGroups:
   - ""
  resources:
   - services
  verbs:
   - get
   - list
   - watch
 - apiGroups:
   - "extensions"
  resources:
   - ingresses
  verbs:
   - get
   - list
   - watch
 - apiGroups:
   - ""
  resources:
    - events
  verbs:
    - create
    - patch
 - apiGroups:
   - "extensions"
  resources:
   - ingresses/status
  verbs:
   - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
 name: nginx-ingress-role
 namespace: kube-system
rules:
 - apiGroups:
   - ""
  resources:
   - configmaps
   - pods
   - secrets
   - namespaces
  verbs:
   - get
 - apiGroups:
   - ""
  resources:
   - configmaps
  resourceNames:
   # Defaults to "<election-id>-<ingress-class>"
   # Here: "<ingress-controller-leader>-<nginx>"
   # This has to be adapted if you change either parameter
   # when launching the nginx-ingress-controller.
   - "ingress-controller-leader-nginx"
  verbs:
   - get
   - update
 - apiGroups:
   - ""
  resources:
   - configmaps
  verbs:
   - create
 - apiGroups:
   - ""
  resources:
   - endpoints
  verbs:
   - get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
 name: nginx-ingress-role-nisa-binding
 namespace: kube-system
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: nginx-ingress-role
subjects:
 - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: nginx-ingress-clusterrole-nisa-binding
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: nginx-ingress-clusterrole
subjects:
 - kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: ingress-nginx
 namespace: kube-system
spec:
 selector:
  matchLabels:
   app: ingress-nginx
 template:
  metadata:
   labels:
    app: ingress-nginx
   annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
  spec:
   serviceAccountName: nginx-ingress-serviceaccount
   tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: NoSchedule
   affinity:
    nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
       - key: "system/ingress"
        operator: In
        values:
        - "true"
   dnsPolicy: ClusterFirstWithHostNet
   hostNetwork: true
   # 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好
   initContainers:
   - name: adddirperm
    image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
    env:
    - name: LOG_DIR
     value: /var/log/nginx_ingress_controller
    - name: USER_ID
      value: "33"
    volumeMounts:
    - name: logdir
     mountPath: /var/log/nginx_ingress_controller
   containers:
   - name: nginx-ingress-controller
    image: "{{ image_registry.addr }}/{{ image.ingress }}"
    imagePullPolicy: IfNotPresent
    args:
    - /nginx-ingress-controller
    - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
    - --configmap=$(POD_NAMESPACE)/nginx-configuration
    - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
    - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx
    - --annotations-prefix=nginx.ingress.kubernetes.io
    
    # 设置controller日志的输出路径和方式
    - --log_dir=/var/log/nginx_ingress_controller
    - --logtostderr=false
    securityContext:
     capabilities:
       drop:
       - ALL
       add:
       - NET_BIND_SERVICE
     # www-data -> 33
     runAsUser: 33
    env:
     - name: POD_NAME
      valueFrom:
       fieldRef:
        fieldPath: metadata.name
     - name: POD_NAMESPACE
      valueFrom:
       fieldRef:
        fieldPath: metadata.namespace
    ports:
    - name: http
     containerPort: 80
    - name: https
     containerPort: 443
    resources:
     requests:
      cpu: 100m
      memory: 256Mi
    livenessProbe:
     failureThreshold: 3
     httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
     initialDelaySeconds: 10
     periodSeconds: 10
     successThreshold: 1
     timeoutSeconds: 1
    readinessProbe:
     failureThreshold: 3
     httpGet:
      path: /healthz
      port: 10254
      scheme: HTTP
     periodSeconds: 10
     successThreshold: 1
     timeoutSeconds: 1
    volumeMounts:
    # 配置挂载容器中控制器组件和nginx的日志输出路径
    - name: logdir
     mountPath: /var/log/nginx_ingress_controller
    # 配置nginx日志的logrotate配置挂载路径
    - name: logrotateconf
     mountPath: /etc/logrotate.d/nginx.log
     subPath: nginx.log
   volumes:
   # 控制器组件和nginx的日志输出路径为宿主机的hostpath
   - name: logdir
    hostPath:
     path: {{ user_nginx_log.host_path }}
     type: ""
   # nginx日志的轮转配置文件来自于configmap
   - name: logrotateconf
    configMap:
     name: nginx-ingress-logrotate
     items:
     - key: nginx.log
      path: nginx.log
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: default-http-backend
 namespace: kube-system
 labels:
  app: default-http-backend
spec:
 selector:
  matchLabels:
   app: default-http-backend
 template:
  metadata:
   labels:
    app: default-http-backend
  spec:
   terminationGracePeriodSeconds: 60
   tolerations:
   - key: dedicated
    value: ingress-nginx
    effect: NoSchedule
   affinity:
    nodeAffinity:
     requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
       - key: "system/ingress"
        operator: In
        values:
        - "true"
   containers:
   - name: default-http-backend
    # Any image is permissible as long as:
    # 1. It serves a 404 page at /
    # 2. It serves 200 on a /healthz endpoint
    image: "{{ image_registry.addr }}/{{ image.http_backend }}"
    imagePullPolicy: IfNotPresent
    livenessProbe:
     httpGet:
      path: /healthz
      port: 8080
      scheme: HTTP
     initialDelaySeconds: 30
     timeoutSeconds: 5
    ports:
    - containerPort: 8080
    resources:
     limits:
      cpu: 10m
      memory: 20Mi
     requests:
      cpu: 10m
      memory: 20Mi
---

最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumeMount之类的配置）。不过还是看个人使用感受吧~

标签：
nginx,ingress,controller日志持久化,nginx,日志持久化
帝王谷资源网 Design By www.wdxyy.com
广告合作：本站广告合作请联系QQ：858582 申请时备注：广告合作（否则不回）
免责声明：本站文章均来自网站采集或用户投稿，网站不提供任何软件下载或自行开发的软件！ 如有用户或公司发现本站内容信息存在侵权行为，请邮件告知！ 858582#qq.com
帝王谷资源网 Design By www.wdxyy.com
www.wdxyy.com 帝王谷资源网
139,976互联网资源
144,792高清电影
21,817无损音乐
631,128技术资源

更新日志

友情链接

杰晶网络 DDR爱好者之家 桃源资源网 杰网资源 富贵资源网 南强小屋 铁雪资源网 幽灵资源网 万梅资源网 狼山资源网 白云岛资源网 昆仑资源网 相思资源网 明霞山资源网 内蒙古资源网 黑松山资源网 茶园资源网 饿虎岗资源网 大旗谷资源网 常春岛资源网 岱庙资源网 兴国资源网 快活林资源网 蝙蝠岛资源网 帝王谷资源网 白云城资源网 伏龙阁资源网 清风细雨楼 天枫庄资源网 圆月山庄资源网 无争山庄资源网 神水资源网 移花宫资源网 神剑山庄资源网 无为清净楼资源网 金钱帮资源网 丐帮资源网 华山资源网 极乐门资源网 小李飞刀资源网 凤求凰客栈 风云阁资源网 金狮镖局 鸳鸯亭资源网 千金楼资源网 更多链接