按当前主机配置整理 Traefik v3、CrowdSec AppSec、Cloudflare Authenticated Origin Pull、DOCKER-USER Cloudflare allowlist、自动证书和业务容器 labels。

2025年7月16日

6 min read

这套入口现在主要做这几件事:

  1. Traefik 负责 80/443 入口、Docker 服务发现、自动申请和续期 Let’s Encrypt 证书。
  2. Cloudflare Authenticated Origin Pull 负责 mTLS,只有 Cloudflare 带客户端证书回源时业务路由才放行。
  3. CrowdSec bouncer 挂在 Traefik entrypoint 上,负责 IP 决策拦截,并把请求转给 CrowdSec AppSec 做 inline WAF 判断。
  4. UFW 的 DOCKER-USER 链只允许 Cloudflare IP 段打进 Docker 暴露的 80/443,作为 mTLS 外面的第二层保护。
Warning (IMPORTANT)

下面的配置按当前主机整理,但我把真实邮箱、CrowdsecLapiKey、Basic Auth hash 等敏感值换成了占位符。不要把服务器上的真实 key 写进公开博客。

旧配置的问题

原来那版配置能表达大方向,但有几处实际问题:

  1. docker-compose.ymltraefik: 下面的缩进不对,直接复制不能跑。
  2. cloudflare-mtls.ymlclientAuth 放错层级,caFiles: - ... 也是错误 YAML。
  3. 只给 web 配了 forwardedHeaders.trustedIPs,当前主机在 webwebsecure 都配置了 Cloudflare IPv4 段。
  4. CrowdSec 插件版本旧了,当前主机是 crowdsec-bouncer-traefik-plugin v1.4.6,并且已经接了 AppSec。
  5. ban.html 文件名和当前主机不一致,当前用的是 banbanban.html
  6. “扫 443 和 80 端口直接丢弃”这个说法不准确。当前主机真正的外层限制在 DOCKER-USER,Traefik 里的 catchall 只是兜底路由;TLS 黑洞连接还会在日志里产生 dial tcp 192.0.2.1:1: i/o timeout

当前主机状态

当前主机上跑的是:

traefik traefik:v3.6.12

crowdsec crowdsecurity/crowdsec:v1.7.8-debian

真实流量里能看到 Cloudflare Origin Pull 证书:

TLSClientSubject="CN=origin-pull.cloudflare.net,O=Cloudflare Inc.,L=San Francisco,ST=CA,C=US"

TLSVersion="1.3"

RouterName="newsjoin@docker"

业务 router 也必须挂上 mTLS 选项。只定义 cloudflare-mtls.yml 不够,关键是业务容器 labels 里要有这一条:

- traefik.http.routers.<router-name>.tls.options=cloudflare-mtls@file

当前 newsjoin 的公开 router 都已经挂了这个选项,包括首页、/news/api/home/api/home/card-items 和 preview API。

目录结构

traefik/

├── acme.json

├── docker-compose.yml

├── cf-cert/

│ └── cloudflare-ca.pem

├── dynamic_conf/

│ ├── auth.yml

│ ├── banbanban.html

│ ├── buffer-middleware.yml

│ ├── cloudflare-mtls.yml

│ ├── compressor.yml

│ ├── crowdsec-middleware.yml

│ └── drop-ip-access.yml

└── logs/

└── traefik.log

crowdsec/

├── docker-compose.yml

└── config/

└── acquis.d/

└── appsec.yaml

准备工作:

docker network create traefik-net

touch acme.json

chmod 600 acme.json

cloudflare-ca.pem 用 Cloudflare Authenticated Origin Pull 的 CA 证书。Cloudflare 会在回源 TLS 握手时出示客户端证书,Traefik 用这个 CA 验证它;没有客户端证书的直连请求会在 TLS 层失败。

Traefik 配置

services:

traefik:

image: traefik:v3.6.12

container_name: traefik

restart: unless-stopped

command:

# 全局中间件:HTTP/HTTPS 入口都先过 CrowdSec,再做压缩。

- '--entrypoints.web.http.middlewares=crowdsec-bouncer@file,global-compressor@file'

- '--entrypoints.websecure.http.middlewares=crowdsec-bouncer@file,global-compressor@file'

# CrowdSec Traefik bouncer plugin。

- '--experimental.plugins.crowdsec-bouncer-traefik-plugin.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin'

- '--experimental.plugins.crowdsec-bouncer-traefik-plugin.version=v1.4.6'

# Dashboard 不要开 api.insecure。要暴露就走 websecure + auth + mTLS。

- '--api.dashboard=true'

- '--api.basePath=/traefik-dash'

- '--entrypoints.web.address=:80'

- '--entrypoints.websecure.address=:443'

# 信任 Cloudflare 回源 IP,Traefik 才会接受 CF-Connecting-IP 这类 forwarded headers。

- '--entrypoints.web.forwardedheaders.trustedips=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22'

- '--entrypoints.websecure.forwardedheaders.trustedips=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22'

- '--providers.docker=true'

- '--providers.docker.exposedbydefault=false'

- '--providers.docker.network=traefik-net'

- '--log.level=INFO'

- '--log.filePath=/var/log/traefik.log'

- '--log.format=json'

- '--providers.file=true'

- '--providers.file.directory=/etc/traefik/dynamic_conf'

- '--providers.file.watch=true'

- '--accesslog=true'

- '--accesslog.format=json'

- '--certificatesresolvers.myresolver.acme.httpchallenge=true'

- '--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web'

- '--certificatesresolvers.myresolver.acme.email=<YOUR_EMAIL>'

- '--certificatesresolvers.myresolver.acme.storage=/etc/traefik/acme.json'

ports:

- '80:80'

- '443:443'

volumes:

- '/var/run/docker.sock:/var/run/docker.sock:ro'

- './acme.json:/etc/traefik/acme.json'

- './dynamic_conf:/etc/traefik/dynamic_conf:ro'

- './logs:/var/log:rw'

- './cf-cert:/etc/traefik/cf-cert:ro'

networks:

- traefik-net

labels:

- traefik.enable=true

- traefik.http.routers.dashboard.rule=Host(`traefik.example.com`) && PathPrefix(`/traefik-dash`)

- traefik.http.routers.dashboard.entrypoints=websecure

- traefik.http.routers.dashboard.service=api@internal

- traefik.http.routers.dashboard.middlewares=auth@file

- traefik.http.routers.dashboard.tls=true

- traefik.http.routers.dashboard.tls.certresolver=myresolver

- traefik.http.routers.dashboard.tls.options=cloudflare-mtls@file

networks:

traefik-net:

name: traefik-net

external: true

当前主机的 dashboard label 里最重要的是 auth@filecloudflare-mtls@file;示例里额外把 rule 写出来,避免依赖 Docker provider 的默认 host rule。

mTLS 配置

tls:

options:

cloudflare-mtls:

minVersion: VersionTLS12

clientAuth:

caFiles:

- /etc/traefik/cf-cert/cloudflare-ca.pem

clientAuthType: 'RequireAndVerifyClientCert'

Warning (IMPORTANT)

这个文件只是在 file provider 里定义了一个 TLS option。每个需要保护的 HTTPS router 都要显式引用 tls.options=cloudflare-mtls@file

动态中间件

http:

middlewares:

global-compressor:

compress:

excludedContentTypes:

- 'image/png'

- 'image/jpeg'

- 'image/gif'

- 'application/pdf'

minResponseBodyBytes: 1024

http:

middlewares:

buffer:

buffering:

maxRequestBodyBytes: 20000000

memRequestBodyBytes: 20000000

maxResponseBodyBytes: 20000000

memResponseBodyBytes: 20000000

retryExpression: 'IsNetworkError() && Attempts() < 2'

http:

middlewares:

auth:

basicAuth:

users:

# docker run --rm httpd:2.4 htpasswd -nb admin '<PASSWORD>'

- 'admin:<HTPASSWD_HASH>'

CrowdSec bouncer

http:

middlewares:

crowdsec-bouncer:

plugin:

crowdsec-bouncer-traefik-plugin:

CrowdsecLapiKey: '<CS_BOUNCER_KEY>'

Enabled: 'true'

crowdsecMode: 'stream'

crowdsecLapiScheme: 'http'

crowdsecLapiHost: 'crowdsec:8080'

# AppSec inline WAF。Traefik 把请求发给 CrowdSec AppSec 引擎判断。

crowdsecAppsecEnabled: true

crowdsecAppsecHost: 'crowdsec:7422'

crowdsecAppsecPath: '/'

crowdsecAppsecFailureBlock: true

crowdsecAppsecUnreachableBlock: true

banHTMLFilePath: '/etc/traefik/dynamic_conf/banbanban.html'

forwardedHeadersCustomName: 'CF-Connecting-IP'

forwardedHeadersTrustedIPs:

- '173.245.48.0/20'

- '103.21.244.0/22'

- '103.22.200.0/22'

- '103.31.4.0/22'

- '141.101.64.0/18'

- '108.162.192.0/18'

- '190.93.240.0/20'

- '188.114.96.0/20'

- '197.234.240.0/22'

- '198.41.128.0/17'

- '162.158.0.0/15'

- '104.16.0.0/13'

- '104.24.0.0/14'

- '172.64.0.0/13'

- '131.0.72.0/22'

# 这些来源会绕过 bouncer 检查,只放自己的内网或运维出口。

clientTrustedIPs:

- '192.168.0.0/16'

- '10.0.0.0/8'

- '172.16.0.0/12'

如果你的源站会被 Cloudflare 用 IPv6 回源,entrypoints.*.forwardedheaders.trustedipsforwardedHeadersTrustedIPs 也要补 Cloudflare IPv6 段。当前主机的 UFW 已经有 IPv6 allowlist,但最近 Traefik 回源日志里看到的是 IPv4。

兜底路由

http:

routers:

catchall-http:

rule: 'HostRegexp(`{host:.+}`)'

entryPoints:

- web

service: noop@internal

priority: 1

tcp:

routers:

catchall-tls:

rule: 'HostSNI(`*`)'

entryPoints:

- websecure

service: 'blackhole-tcp-svc'

priority: 1

tls:

passthrough: false

services:

blackhole-tcp-svc:

loadBalancer:

servers:

- address: '192.0.2.1:1'

这段只是低优先级兜底。真实业务 router 的 priority 要比 1 高。当前主机日志里会看到 TLS catchall 连接打到 192.0.2.1:1 后超时,这是黑洞写法的副作用。

CrowdSec 配置

services:

crowdsec:

image: crowdsecurity/crowdsec:v1.7.8-debian

container_name: crowdsec

restart: unless-stopped

environment:

- GID=999

- BOUNCER_KEY_TRAEFIK=<CS_BOUNCER_KEY>

- COLLECTIONS=crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules

volumes:

- /var/run/docker.sock:/var/run/docker.sock:ro

- ./data:/var/lib/crowdsec/data

- ./config:/etc/crowdsec

networks:

- traefik-net

networks:

traefik-net:

external: true

name: traefik-net

appsec_configs:

- crowdsecurity/appsec-default

labels:

type: appsec

listen_addr: 0.0.0.0:7422

path: /

source: appsec

当前主机上 cscli collections list 能看到这些已启用:

crowdsecurity/traefik

crowdsecurity/appsec-virtual-patching

crowdsecurity/appsec-generic-rules

crowdsecurity/http-cve

AppSec 的作用不是事后看日志封 IP,而是让 Traefik bouncer 在请求进入业务前把请求内容交给 CrowdSec AppSec 判断。上面的 crowdsecAppsecEnabled: truecrowdsecAppsecHost: "crowdsec:7422"appsec.yaml 是一套。

主机防火墙

只靠 mTLS 不够。当前主机还在 UFW 的 DOCKER-USER 链限制 Docker 暴露的 80/443:外部打进来的 Web 流量必须来自 Cloudflare IP 段;Docker 容器自己的出站 HTTP/HTTPS 要提前 RETURN,否则会误伤构建、抓取和依赖下载。

*filter

:DOCKER-USER - [0:0]

:ufw-docker-cloudflare-web - [0:0]

-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 80 -j RETURN

-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 443 -j RETURN

-A DOCKER-USER ! -i br+ -p tcp -m multiport --dports 80,443 -j ufw-docker-cloudflare-web

-A ufw-docker-cloudflare-web -s 173.245.48.0/20 -j RETURN

-A ufw-docker-cloudflare-web -s 103.21.244.0/22 -j RETURN

-A ufw-docker-cloudflare-web -s 103.22.200.0/22 -j RETURN

-A ufw-docker-cloudflare-web -s 103.31.4.0/22 -j RETURN

-A ufw-docker-cloudflare-web -s 141.101.64.0/18 -j RETURN

-A ufw-docker-cloudflare-web -s 108.162.192.0/18 -j RETURN

-A ufw-docker-cloudflare-web -s 190.93.240.0/20 -j RETURN

-A ufw-docker-cloudflare-web -s 188.114.96.0/20 -j RETURN

-A ufw-docker-cloudflare-web -s 197.234.240.0/22 -j RETURN

-A ufw-docker-cloudflare-web -s 198.41.128.0/17 -j RETURN

-A ufw-docker-cloudflare-web -s 162.158.0.0/15 -j RETURN

-A ufw-docker-cloudflare-web -s 104.16.0.0/13 -j RETURN

-A ufw-docker-cloudflare-web -s 104.24.0.0/14 -j RETURN

-A ufw-docker-cloudflare-web -s 172.64.0.0/13 -j RETURN

-A ufw-docker-cloudflare-web -s 131.0.72.0/22 -j RETURN

-A ufw-docker-cloudflare-web -j DROP

COMMIT

IPv6 同理:

*filter

:DOCKER-USER - [0:0]

:ufw6-docker-cloudflare-web - [0:0]

-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 80 -j RETURN

-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 443 -j RETURN

-A DOCKER-USER ! -i br+ -p tcp -m multiport --dports 80,443 -j ufw6-docker-cloudflare-web

-A ufw6-docker-cloudflare-web -s 2400:cb00::/32 -j RETURN

-A ufw6-docker-cloudflare-web -s 2606:4700::/32 -j RETURN

-A ufw6-docker-cloudflare-web -s 2803:f800::/32 -j RETURN

-A ufw6-docker-cloudflare-web -s 2405:b500::/32 -j RETURN

-A ufw6-docker-cloudflare-web -s 2405:8100::/32 -j RETURN

-A ufw6-docker-cloudflare-web -s 2a06:98c0::/29 -j RETURN

-A ufw6-docker-cloudflare-web -s 2c0f:f248::/32 -j RETURN

-A ufw6-docker-cloudflare-web -j DROP

COMMIT

改 UFW 前先测语法:

iptables-restore --test < /etc/ufw/after.rules

ip6tables-restore --test < /etc/ufw/after6.rules

systemctl restart ufw

ufw status 不是 Docker 入口保护的最终证据。检查这类规则时,优先看 iptables -S DOCKER-USERip6tables -S DOCKER-USER,再做真实直连测试。

业务容器 labels

每个业务容器至少要有这些 labels:

services:

app:

image: ghcr.io/example/app:latest

networks:

- traefik-net

labels:

- traefik.enable=true

- traefik.http.routers.app.rule=Host(`app.example.com`)

- traefik.http.routers.app.entrypoints=websecure

- traefik.http.routers.app.tls=true

- traefik.http.routers.app.tls.certresolver=myresolver

- traefik.http.routers.app.tls.options=cloudflare-mtls@file

- traefik.http.services.app.loadbalancer.server.port=3000

networks:

traefik-net:

name: traefik-net

external: true

如果要按真实访客 IP 限速,不要用 Cloudflare edge IP 当 key。当前主机的限速 middleware 是按 CF-Connecting-IP 取源:

- traefik.http.middlewares.app-ratelimit.ratelimit.average=30

- traefik.http.middlewares.app-ratelimit.ratelimit.burst=10

- traefik.http.middlewares.app-ratelimit.ratelimit.period=1m

- traefik.http.middlewares.app-ratelimit.ratelimit.sourcecriterion.requestheadername=CF-Connecting-IP

- traefik.http.routers.app.middlewares=app-ratelimit

当前 newsjoin 按路径拆了几个 router:/news30 rpm/api/home/card-items20 rpm、其他公开 home/preview API 是 120 rpm。拆 router 的原因是不同接口的正常访问频率差异很大,不能全塞进一个桶。

验证

部署后至少做这几项:

docker compose -f /home/ubuntu/docker-data/traefik/docker-compose.yml config

docker exec traefik traefik version

docker exec crowdsec cscli lapi status

docker exec crowdsec cscli capi status

docker exec crowdsec cscli collections list

docker exec crowdsec cscli appsec-configs list

iptables -S DOCKER-USER

ip6tables -S DOCKER-USER

业务侧要确认:

  1. 正常走 Cloudflare 的域名请求返回 200
  2. Traefik access log 里有 TLSClientSubject="CN=origin-pull.cloudflare.net,..."
  3. 业务 router 的 labels 里有 tls.options=cloudflare-mtls@file
  4. 直连源站 IP 不带 Cloudflare 客户端证书时失败,常见表现是 TLS certificate required 或被主机防火墙超时。

这四项同时成立,才算 Cloudflare mTLS、Traefik router 和主机防火墙这三层都接上了。