这套入口现在主要做这几件事:
- Traefik 负责 80/443 入口、Docker 服务发现、自动申请和续期 Let’s Encrypt 证书。
- Cloudflare Authenticated Origin Pull 负责 mTLS,只有 Cloudflare 带客户端证书回源时业务路由才放行。
- CrowdSec bouncer 挂在 Traefik entrypoint 上,负责 IP 决策拦截,并把请求转给 CrowdSec AppSec 做 inline WAF 判断。
- UFW 的
DOCKER-USER链只允许 Cloudflare IP 段打进 Docker 暴露的 80/443,作为 mTLS 外面的第二层保护。
Warning (IMPORTANT)
下面的配置按当前主机整理,但我把真实邮箱、CrowdsecLapiKey、Basic Auth hash
等敏感值换成了占位符。不要把服务器上的真实 key 写进公开博客。
旧配置的问题
原来那版配置能表达大方向,但有几处实际问题:
docker-compose.yml里traefik:下面的缩进不对,直接复制不能跑。cloudflare-mtls.yml的clientAuth放错层级,caFiles: - ...也是错误 YAML。- 只给
web配了forwardedHeaders.trustedIPs,当前主机在web和websecure都配置了 Cloudflare IPv4 段。 - CrowdSec 插件版本旧了,当前主机是
crowdsec-bouncer-traefik-plugin v1.4.6,并且已经接了 AppSec。 ban.html文件名和当前主机不一致,当前用的是banbanban.html。- “扫 443 和 80 端口直接丢弃”这个说法不准确。当前主机真正的外层限制在
DOCKER-USER,Traefik 里的 catchall 只是兜底路由;TLS 黑洞连接还会在日志里产生dial tcp 192.0.2.1:1: i/o timeout。
当前主机状态
当前主机上跑的是:
traefik traefik:v3.6.12
crowdsec crowdsecurity/crowdsec:v1.7.8-debian
真实流量里能看到 Cloudflare Origin Pull 证书:
TLSClientSubject="CN=origin-pull.cloudflare.net,O=Cloudflare Inc.,L=San Francisco,ST=CA,C=US"
TLSVersion="1.3"
RouterName="newsjoin@docker"
业务 router 也必须挂上 mTLS 选项。只定义 cloudflare-mtls.yml 不够,关键是业务容器 labels 里要有这一条:
- traefik.http.routers.<router-name>.tls.options=cloudflare-mtls@file
当前 newsjoin 的公开 router 都已经挂了这个选项,包括首页、/news、/api/home、/api/home/card-items 和 preview API。
目录结构
traefik/
├── acme.json
├── docker-compose.yml
├── cf-cert/
│ └── cloudflare-ca.pem
├── dynamic_conf/
│ ├── auth.yml
│ ├── banbanban.html
│ ├── buffer-middleware.yml
│ ├── cloudflare-mtls.yml
│ ├── compressor.yml
│ ├── crowdsec-middleware.yml
│ └── drop-ip-access.yml
└── logs/
└── traefik.log
crowdsec/
├── docker-compose.yml
└── config/
└── acquis.d/
└── appsec.yaml
准备工作:
docker network create traefik-net
touch acme.json
chmod 600 acme.json
cloudflare-ca.pem 用 Cloudflare Authenticated Origin Pull 的 CA 证书。Cloudflare 会在回源 TLS 握手时出示客户端证书,Traefik 用这个 CA 验证它;没有客户端证书的直连请求会在 TLS 层失败。
Traefik 配置
services:
traefik:
image: traefik:v3.6.12
container_name: traefik
restart: unless-stopped
command:
# 全局中间件:HTTP/HTTPS 入口都先过 CrowdSec,再做压缩。
- '--entrypoints.web.http.middlewares=crowdsec-bouncer@file,global-compressor@file'
- '--entrypoints.websecure.http.middlewares=crowdsec-bouncer@file,global-compressor@file'
# CrowdSec Traefik bouncer plugin。
- '--experimental.plugins.crowdsec-bouncer-traefik-plugin.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin'
- '--experimental.plugins.crowdsec-bouncer-traefik-plugin.version=v1.4.6'
# Dashboard 不要开 api.insecure。要暴露就走 websecure + auth + mTLS。
- '--api.dashboard=true'
- '--api.basePath=/traefik-dash'
- '--entrypoints.web.address=:80'
- '--entrypoints.websecure.address=:443'
# 信任 Cloudflare 回源 IP,Traefik 才会接受 CF-Connecting-IP 这类 forwarded headers。
- '--entrypoints.web.forwardedheaders.trustedips=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22'
- '--entrypoints.websecure.forwardedheaders.trustedips=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22'
- '--providers.docker=true'
- '--providers.docker.exposedbydefault=false'
- '--providers.docker.network=traefik-net'
- '--log.level=INFO'
- '--log.filePath=/var/log/traefik.log'
- '--log.format=json'
- '--providers.file=true'
- '--providers.file.directory=/etc/traefik/dynamic_conf'
- '--providers.file.watch=true'
- '--accesslog=true'
- '--accesslog.format=json'
- '--certificatesresolvers.myresolver.acme.httpchallenge=true'
- '--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web'
- '--certificatesresolvers.myresolver.acme.email=<YOUR_EMAIL>'
- '--certificatesresolvers.myresolver.acme.storage=/etc/traefik/acme.json'
ports:
- '80:80'
- '443:443'
volumes:
- '/var/run/docker.sock:/var/run/docker.sock:ro'
- './acme.json:/etc/traefik/acme.json'
- './dynamic_conf:/etc/traefik/dynamic_conf:ro'
- './logs:/var/log:rw'
- './cf-cert:/etc/traefik/cf-cert:ro'
networks:
- traefik-net
labels:
- traefik.enable=true
- traefik.http.routers.dashboard.rule=Host(`traefik.example.com`) && PathPrefix(`/traefik-dash`)
- traefik.http.routers.dashboard.entrypoints=websecure
- traefik.http.routers.dashboard.service=api@internal
- traefik.http.routers.dashboard.middlewares=auth@file
- traefik.http.routers.dashboard.tls=true
- traefik.http.routers.dashboard.tls.certresolver=myresolver
- traefik.http.routers.dashboard.tls.options=cloudflare-mtls@file
networks:
traefik-net:
name: traefik-net
external: true
当前主机的 dashboard label 里最重要的是 auth@file 和 cloudflare-mtls@file;示例里额外把 rule 写出来,避免依赖 Docker provider 的默认 host rule。
mTLS 配置
tls:
options:
cloudflare-mtls:
minVersion: VersionTLS12
clientAuth:
caFiles:
- /etc/traefik/cf-cert/cloudflare-ca.pem
clientAuthType: 'RequireAndVerifyClientCert'
Warning (IMPORTANT)
这个文件只是在 file provider 里定义了一个 TLS option。每个需要保护的 HTTPS
router 都要显式引用 tls.options=cloudflare-mtls@file。
动态中间件
http:
middlewares:
global-compressor:
compress:
excludedContentTypes:
- 'image/png'
- 'image/jpeg'
- 'image/gif'
- 'application/pdf'
minResponseBodyBytes: 1024
http:
middlewares:
buffer:
buffering:
maxRequestBodyBytes: 20000000
memRequestBodyBytes: 20000000
maxResponseBodyBytes: 20000000
memResponseBodyBytes: 20000000
retryExpression: 'IsNetworkError() && Attempts() < 2'
http:
middlewares:
auth:
basicAuth:
users:
# docker run --rm httpd:2.4 htpasswd -nb admin '<PASSWORD>'
- 'admin:<HTPASSWD_HASH>'
CrowdSec bouncer
http:
middlewares:
crowdsec-bouncer:
plugin:
crowdsec-bouncer-traefik-plugin:
CrowdsecLapiKey: '<CS_BOUNCER_KEY>'
Enabled: 'true'
crowdsecMode: 'stream'
crowdsecLapiScheme: 'http'
crowdsecLapiHost: 'crowdsec:8080'
# AppSec inline WAF。Traefik 把请求发给 CrowdSec AppSec 引擎判断。
crowdsecAppsecEnabled: true
crowdsecAppsecHost: 'crowdsec:7422'
crowdsecAppsecPath: '/'
crowdsecAppsecFailureBlock: true
crowdsecAppsecUnreachableBlock: true
banHTMLFilePath: '/etc/traefik/dynamic_conf/banbanban.html'
forwardedHeadersCustomName: 'CF-Connecting-IP'
forwardedHeadersTrustedIPs:
- '173.245.48.0/20'
- '103.21.244.0/22'
- '103.22.200.0/22'
- '103.31.4.0/22'
- '141.101.64.0/18'
- '108.162.192.0/18'
- '190.93.240.0/20'
- '188.114.96.0/20'
- '197.234.240.0/22'
- '198.41.128.0/17'
- '162.158.0.0/15'
- '104.16.0.0/13'
- '104.24.0.0/14'
- '172.64.0.0/13'
- '131.0.72.0/22'
# 这些来源会绕过 bouncer 检查,只放自己的内网或运维出口。
clientTrustedIPs:
- '192.168.0.0/16'
- '10.0.0.0/8'
- '172.16.0.0/12'
如果你的源站会被 Cloudflare 用 IPv6 回源,entrypoints.*.forwardedheaders.trustedips 和 forwardedHeadersTrustedIPs 也要补 Cloudflare IPv6 段。当前主机的 UFW 已经有 IPv6 allowlist,但最近 Traefik 回源日志里看到的是 IPv4。
兜底路由
http:
routers:
catchall-http:
rule: 'HostRegexp(`{host:.+}`)'
entryPoints:
- web
service: noop@internal
priority: 1
tcp:
routers:
catchall-tls:
rule: 'HostSNI(`*`)'
entryPoints:
- websecure
service: 'blackhole-tcp-svc'
priority: 1
tls:
passthrough: false
services:
blackhole-tcp-svc:
loadBalancer:
servers:
- address: '192.0.2.1:1'
这段只是低优先级兜底。真实业务 router 的 priority 要比 1 高。当前主机日志里会看到 TLS catchall 连接打到 192.0.2.1:1 后超时,这是黑洞写法的副作用。
CrowdSec 配置
services:
crowdsec:
image: crowdsecurity/crowdsec:v1.7.8-debian
container_name: crowdsec
restart: unless-stopped
environment:
- GID=999
- BOUNCER_KEY_TRAEFIK=<CS_BOUNCER_KEY>
- COLLECTIONS=crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./data:/var/lib/crowdsec/data
- ./config:/etc/crowdsec
networks:
- traefik-net
networks:
traefik-net:
external: true
name: traefik-net
appsec_configs:
- crowdsecurity/appsec-default
labels:
type: appsec
listen_addr: 0.0.0.0:7422
path: /
source: appsec
当前主机上 cscli collections list 能看到这些已启用:
crowdsecurity/traefik
crowdsecurity/appsec-virtual-patching
crowdsecurity/appsec-generic-rules
crowdsecurity/http-cve
AppSec 的作用不是事后看日志封 IP,而是让 Traefik bouncer 在请求进入业务前把请求内容交给 CrowdSec AppSec 判断。上面的 crowdsecAppsecEnabled: true、crowdsecAppsecHost: "crowdsec:7422" 和 appsec.yaml 是一套。
主机防火墙
只靠 mTLS 不够。当前主机还在 UFW 的 DOCKER-USER 链限制 Docker 暴露的 80/443:外部打进来的 Web 流量必须来自 Cloudflare IP 段;Docker 容器自己的出站 HTTP/HTTPS 要提前 RETURN,否则会误伤构建、抓取和依赖下载。
*filter
:DOCKER-USER - [0:0]
:ufw-docker-cloudflare-web - [0:0]
-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 80 -j RETURN
-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 443 -j RETURN
-A DOCKER-USER ! -i br+ -p tcp -m multiport --dports 80,443 -j ufw-docker-cloudflare-web
-A ufw-docker-cloudflare-web -s 173.245.48.0/20 -j RETURN
-A ufw-docker-cloudflare-web -s 103.21.244.0/22 -j RETURN
-A ufw-docker-cloudflare-web -s 103.22.200.0/22 -j RETURN
-A ufw-docker-cloudflare-web -s 103.31.4.0/22 -j RETURN
-A ufw-docker-cloudflare-web -s 141.101.64.0/18 -j RETURN
-A ufw-docker-cloudflare-web -s 108.162.192.0/18 -j RETURN
-A ufw-docker-cloudflare-web -s 190.93.240.0/20 -j RETURN
-A ufw-docker-cloudflare-web -s 188.114.96.0/20 -j RETURN
-A ufw-docker-cloudflare-web -s 197.234.240.0/22 -j RETURN
-A ufw-docker-cloudflare-web -s 198.41.128.0/17 -j RETURN
-A ufw-docker-cloudflare-web -s 162.158.0.0/15 -j RETURN
-A ufw-docker-cloudflare-web -s 104.16.0.0/13 -j RETURN
-A ufw-docker-cloudflare-web -s 104.24.0.0/14 -j RETURN
-A ufw-docker-cloudflare-web -s 172.64.0.0/13 -j RETURN
-A ufw-docker-cloudflare-web -s 131.0.72.0/22 -j RETURN
-A ufw-docker-cloudflare-web -j DROP
COMMIT
IPv6 同理:
*filter
:DOCKER-USER - [0:0]
:ufw6-docker-cloudflare-web - [0:0]
-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 80 -j RETURN
-A DOCKER-USER -i docker0 -p tcp -m tcp --dport 443 -j RETURN
-A DOCKER-USER ! -i br+ -p tcp -m multiport --dports 80,443 -j ufw6-docker-cloudflare-web
-A ufw6-docker-cloudflare-web -s 2400:cb00::/32 -j RETURN
-A ufw6-docker-cloudflare-web -s 2606:4700::/32 -j RETURN
-A ufw6-docker-cloudflare-web -s 2803:f800::/32 -j RETURN
-A ufw6-docker-cloudflare-web -s 2405:b500::/32 -j RETURN
-A ufw6-docker-cloudflare-web -s 2405:8100::/32 -j RETURN
-A ufw6-docker-cloudflare-web -s 2a06:98c0::/29 -j RETURN
-A ufw6-docker-cloudflare-web -s 2c0f:f248::/32 -j RETURN
-A ufw6-docker-cloudflare-web -j DROP
COMMIT
改 UFW 前先测语法:
iptables-restore --test < /etc/ufw/after.rules
ip6tables-restore --test < /etc/ufw/after6.rules
systemctl restart ufw
ufw status 不是 Docker 入口保护的最终证据。检查这类规则时,优先看 iptables -S DOCKER-USER、ip6tables -S DOCKER-USER,再做真实直连测试。
业务容器 labels
每个业务容器至少要有这些 labels:
services:
app:
image: ghcr.io/example/app:latest
networks:
- traefik-net
labels:
- traefik.enable=true
- traefik.http.routers.app.rule=Host(`app.example.com`)
- traefik.http.routers.app.entrypoints=websecure
- traefik.http.routers.app.tls=true
- traefik.http.routers.app.tls.certresolver=myresolver
- traefik.http.routers.app.tls.options=cloudflare-mtls@file
- traefik.http.services.app.loadbalancer.server.port=3000
networks:
traefik-net:
name: traefik-net
external: true
如果要按真实访客 IP 限速,不要用 Cloudflare edge IP 当 key。当前主机的限速 middleware 是按 CF-Connecting-IP 取源:
- traefik.http.middlewares.app-ratelimit.ratelimit.average=30
- traefik.http.middlewares.app-ratelimit.ratelimit.burst=10
- traefik.http.middlewares.app-ratelimit.ratelimit.period=1m
- traefik.http.middlewares.app-ratelimit.ratelimit.sourcecriterion.requestheadername=CF-Connecting-IP
- traefik.http.routers.app.middlewares=app-ratelimit
当前 newsjoin 按路径拆了几个 router:/news 是 30 rpm、/api/home/card-items 是 20 rpm、其他公开 home/preview API 是 120 rpm。拆 router 的原因是不同接口的正常访问频率差异很大,不能全塞进一个桶。
验证
部署后至少做这几项:
docker compose -f /home/ubuntu/docker-data/traefik/docker-compose.yml config
docker exec traefik traefik version
docker exec crowdsec cscli lapi status
docker exec crowdsec cscli capi status
docker exec crowdsec cscli collections list
docker exec crowdsec cscli appsec-configs list
iptables -S DOCKER-USER
ip6tables -S DOCKER-USER
业务侧要确认:
- 正常走 Cloudflare 的域名请求返回
200。 - Traefik access log 里有
TLSClientSubject="CN=origin-pull.cloudflare.net,..."。 - 业务 router 的 labels 里有
tls.options=cloudflare-mtls@file。 - 直连源站 IP 不带 Cloudflare 客户端证书时失败,常见表现是 TLS certificate required 或被主机防火墙超时。
这四项同时成立,才算 Cloudflare mTLS、Traefik router 和主机防火墙这三层都接上了。