在 Linux Router 上配置 Source-Based Routing 策略(源进源出)
<p>近期从朋友手里嫖了一点带宽,但是上游的网络配置明显禁止了非对称路由,所以按照我们 AS47778 的骨干设计来讲,内网的路由不对称,会导致有些并非来自该上游的来向包被选路到该上游出口。</p> <p>所以,我们接下来基于 nftables 和 iproute2 来配置了 Source-Based Routing,来保证仅有来自该上游的包可以选择该上游进行转发。</p> <h2 id="netfilter-和-nftables">Netfilter 和 nftables </h2><p>Netfilter 是 Linux 内核中的一个用于管理数据包的框架。一切数据包都从 Netfilter 经过,经历如下图所示的周期。在 prerouting、input、output、forward 和 postrouting 阶段,Netfilter 允许 iptables 和 nftables 这样的应用程序对数据包进行控制,这也正是防火墙的基础。</p> <p><img src="/post/bgp-router-with-sbr/Netfilter-packet-flow.png" loading="lazy" alt="Netfilter-packet-flow" ></p> <p>看起来很麻烦吧,所以我们来简化一下。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-plain" data-lang="plain"><span class="line"><span class="cl">(NETIN) -> PREROUTING -> <ROUTING> -> FOWARD -> POSTROUTING -> (NETOUT) </span></span><span class="line"><span class="cl"> | ↑ </span></span><span class="line"><span class="cl"> | <ROUTING> </span></span><span class="line"><span class="cl"> ↓ ↑ </span></span><span class="line"><span class="cl"> INPUT -> [LOCAL] -> OUTPUT </span></span></code></pre></td></tr></table> </div> </div><p>对于到本地的包 A,从 NETIN 进入之后,会通过 <code>PREROUTING</code> 链,到达 ROUTING 进行路由决策,检查到是本机包后进入 <code>INPUT</code> 链,然后被实际的发送给本地的程序。<br> 本地的程序处理后,发送向外部的包 B,从 LOCAL 进入后到达 <code>OUTPUT</code> 链,再次到达 ROUTING 进行路由决策,通过 <code>POSTROUTING</code> 链后最终从 NETOUT 离开本机。</p> <p>对于需要转发的包 C,从 NETIN 进入之后,通过 <code>PREROUINTG</code> 链,到达 ROUTING 进行路由决策,检查到需要转发,进入 <code>FORWARD</code> 链,通过 <code>POSTROUTING</code> 链后最终从 NETOUT 离开本机。</p> <p>对于我们目前的情况而言,我们来分析以下不同包的情况:</p> <ol> <li>去包从该上游进入网络内部,在本节点需要转发到网内的去包:正常转发。</li> <li>去包从该上游进入网络内部,在本节点需要响应回包:回包正常按照选路离开节点或收到回包后源进源出即可。</li> <li>去包从其他节点进入网络内部,在本节点需要继续转发在网内的去包:正常按照选路离开节点即可。</li> <li>去包从其他节点进入网络内部,在本节点需要响应回包:回包不可以从该上游离开网络,需要源进源出。</li> </ol> <p>在 Linux 的网络栈中,我们可以通过向包添加 fwmark (通常是一个 32bit 的无符号证书) 的方式来为数据包添加 tag,通过这个 tag 来在路由决策里将路由导向不同的 NETOUT。</p> <p>同时也存在着一个 conntrack 机制和对应的 conntrack 表,用来追踪整个连接,通常被用于 NAT 环境中方便维护转发关系,我们在这里也会使用用来解决 2 的回包进行源进源出问题。</p> <p>对于我们的场景而言,我们主要需要以上的 2 和 3 两个情况进行源进源出策略。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span><span class="lnt">14 </span><span class="lnt">15 </span><span class="lnt">16 </span><span class="lnt">17 </span><span class="lnt">18 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">#!/usr/sbin/nft -f </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl">table inet source_routing { </span></span><span class="line"><span class="cl"> chain prerouting { </span></span><span class="line"><span class="cl"> type filter hook prerouting priority mangle; policy accept; </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> iifname "up1" ct mark set 0x00000001 </span></span><span class="line"><span class="cl"> iifname "node1" ct mark set 0x00000002 </span></span><span class="line"><span class="cl"> iifname "node2" ct mark set 0x00000003 </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> ct state {established, related} meta mark set ct mark </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> chain output { </span></span><span class="line"><span class="cl"> type route hook output priority mangle; policy accept; </span></span><span class="line"><span class="cl"> meta mark set ct mark </span></span><span class="line"><span class="cl"> } </span></span><span class="line"><span class="cl">} </span></span></code></pre></td></tr></table> </div> </div><p>我们在这里创建一个名为 source_routing 的表,同时为 IPv4 和 IPv6 应用 (inet)。</p> <p>在 <code>PREROUTING</code> 链上,为来自不同接口的 packet 的连接打上了 ct mark,用来追踪连接。<br> 对于已经被 ct mark 匹配并追踪的既有连接的包,我们直接将 ct mark 作为 fwmark 的值复制。<br> 通过<code>PREROUTING</code> 链和 conntrack,我们给符合 2 条件的包打上了 fwmark。</p> <p>在 <code>OUTPUT</code> 链上,是本地程序响应的包,符合 4 条件。我们直接 copy ct mark 到 fwmark。</p> <h2 id="iproute2路由分表和策略路由">iproute2、路由分表和策略路由 </h2><p>在 Linux 中,最多可以存在 255 (1-255) 个路由表,系统默认为我们配置 255 (local) / 254 (main) / 253 (default) 三个路由表。</p> <p>local 表中包含本机路由和广播信息,127.0.0.1/8 的路由就来自该路由表,该表的内容由内核主动维护,理论来讲不需要额外的修改。</p> <p>main 表中包含默认配置的路由信息,是默认使用工具添加的表格,正常的 <code>ip route add</code> 就会将路由添加到该表。</p> <p>default 表默认内容为空,可以手动指明向该表写路由。</p> <p>对于我们需求而言,我们需要创造 N 个表(N 即该节点的外联节点+上游个数),并给对应的表添加路由来。</p> <p>在 Linux with iproute2,我们需要在 <code>/etc/iproute2/rt_tables</code> 维护表格配置。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span><span class="lnt">6 </span><span class="lnt">7 </span><span class="lnt">8 </span><span class="lnt">9 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-fallback" data-lang="fallback"><span class="line"><span class="cl">255 local </span></span><span class="line"><span class="cl">254 main </span></span><span class="line"><span class="cl">253 default </span></span><span class="line"><span class="cl">0 unspec </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"># Following is what we modified. </span></span><span class="line"><span class="cl">100 up1_table </span></span><span class="line"><span class="cl">101 node1_table </span></span><span class="line"><span class="cl">102 node2_table </span></span></code></pre></td></tr></table> </div> </div><p>我们在这里添加了 <code>100</code> <code>101</code> <code>102</code> 三个表,分别 mapping 到 <code>up1_table</code> <code>node1_table</code> <code>node2_table</code> 上。</p> <p>对于我们的要求,可以直接添加 <code>default via $gateway dev $device</code> 到各个路由上。</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt"> 1 </span><span class="lnt"> 2 </span><span class="lnt"> 3 </span><span class="lnt"> 4 </span><span class="lnt"> 5 </span><span class="lnt"> 6 </span><span class="lnt"> 7 </span><span class="lnt"> 8 </span><span class="lnt"> 9 </span><span class="lnt">10 </span><span class="lnt">11 </span><span class="lnt">12 </span><span class="lnt">13 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="cp">#!/bin/bash </span></span></span><span class="line"><span class="cl"><span class="c1"># policy.sh</span> </span></span><span class="line"><span class="cl"><span class="nb">export</span> <span class="nv">src</span><span class="o">=</span><span class="s2">"fe80::2"</span> <span class="c1"># 通过该变量来保证不同 interface 上出包 src 的 IP 一致。</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"><span class="k">while</span> <span class="nv">IFS</span><span class="o">=</span><span class="s1">'|'</span> <span class="nb">read</span> -r fwmark device table gateway<span class="p">;</span> <span class="k">do</span> </span></span><span class="line"><span class="cl"> <span class="nv">fwmark</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$fwmark</span><span class="s2">"</span> <span class="p">|</span> tr -d <span class="s1">'[:space:]'</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> <span class="nv">device</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$device</span><span class="s2">"</span> <span class="p">|</span> tr -d <span class="s1">'[:space:]'</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> <span class="nv">table</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$table</span><span class="s2">"</span> <span class="p">|</span> tr -d <span class="s1">'[:space:]'</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> <span class="nv">gateway</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$gateway</span><span class="s2">"</span> <span class="p">|</span> tr -d <span class="s1">'[:space:]'</span><span class="k">)</span> </span></span><span class="line"><span class="cl"> </span></span><span class="line"><span class="cl"> sudo ip route add default via <span class="s2">"</span><span class="nv">$gateway</span><span class="s2">"</span> dev <span class="s2">"</span><span class="nv">$device</span><span class="s2">"</span> src <span class="s2">"</span><span class="nv">$src</span><span class="s2">"</span> table <span class="s2">"</span><span class="nv">$table</span><span class="s2">"</span> </span></span><span class="line"><span class="cl"> sudo ip rule add fwmark <span class="s2">"</span><span class="nv">$fwmark</span><span class="s2">"</span> lookup <span class="s2">"</span><span class="nv">$table</span><span class="s2">"</span> </span></span><span class="line"><span class="cl"><span class="k">done</span> </span></span></code></pre></td></tr></table> </div> </div><div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span><span class="lnt">2 </span><span class="lnt">3 </span><span class="lnt">4 </span><span class="lnt">5 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-csv" data-lang="csv"><span class="line"><span class="cl"><span class="s"># policy.conf</span><span class="p"> </span></span></span><span class="line"><span class="cl"><span class="s"># fwmark|device|table|gateway</span><span class="p"> </span></span></span><span class="line"><span class="cl"><span class="s">0x1|up1|up1_table|fe80::1</span><span class="p"> </span></span></span><span class="line"><span class="cl"><span class="s">0x2|node1|node1_table|fe80::1</span><span class="p"> </span></span></span><span class="line"><span class="cl"><span class="s">0x3|node2|node2_table|fe80::1</span><span class="p"> </span></span></span></code></pre></td></tr></table> </div> </div><div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">cat policy.conf <span class="p">|</span> sudo policy.sh </span></span></code></pre></td></tr></table> </div> </div><p>通过这个小脚本,我们分别给每个表都按照配置加入了对应的到 gateway 的默认路由,然后配置了包含 fwmark 的包都按照对应的 fwmark 命中到表上。</p> <p>此时对于来自该上游并需要在本地响应的回包:</p> <ol> <li>入包命中 <code>PREROUTING</code> 链,并被打上 ct mark</li> <li>回包进入 <code>OUTPUT</code> 链,将 ct mark 复制到 fwmark 上</li> <li>回包进入 ROUTING 路由决策,匹配 fwmark 到对应的仅默认路由路由表上</li> <li>回包离开本机</li> </ol> <p>以上,我们成功配置了源进源出的网络路由方案,顺带对 fwmark 和 nftables 等有了初步的了解。若要进一步了解 nftables 及其配置, 官方 wiki 是个不错的选择,无论是配置过程中遇到问题还是希望进行更复杂的配置,都能在<a class="link" href="https://wiki.nftables.org/wiki-nftables/index.php/Main_Page" target="_blank" rel="noopener" >官方 wiki</a> 找到详尽的说明。总之,配网愉快~</p>