www.463.com 永利官网误乐域 / Tue, 05 Nov 2019 21:32:09 +0800 Tue, 05 Nov 2019 21:32:09 +0800 Jekyll v4.0.0 如何编写一个 CoreDNS 插件 <blockquote> <p>目前测试环境中有很多个 DNS 服务器,不同项目组使用的 DNS 服务器不同,但是不可避免的他们会访问一些公共域名;老的 DNS 服务器都是 dnsmasq,改起来很麻烦,最近研究了一下 CoreDNS,通过编写插件的方式可以实现让多个 CoreDNS 实例实现分布式的统一控制,以下记录了插件编写过程</p> </blockquote> <h2 id="一coredns-简介">一、CoreDNS 简介</h2> <p>CoreDNS 目前是 CNCF 旗下的项目(已毕业),为 Kubernetes 等云原生环境提供可靠的 DNS 服务发现等功能;官网的描述只有一句话: <strong>CoreDNS: DNS and Service Discovery</strong>,而实际上分析源码以后发现 CoreDNS 实际上是基于 Caddy (一个现代化的负载均衡器)而开发的,通过插件式注入,并监听 TCP/UDP 端口提供 DNS 服务;<strong>得益于 Caddy 的插件机制,CoreDNS 支持自行编写插件,拦截 DNS 请求然后处理,</strong>通过这个插件机制你可以在 CoreDNS 上实现各种功能,比如构建分布式一致性的 DNS 集群、动态的 DNS 负载均衡等等</p> <h2 id="二coredns-插件规范">二、CoreDNS 插件规范</h2> <h3 id="21插件模式">2.1、插件模式</h3> <p>CoreDNS 插件编写目前有两种方式:</p> <ul> <li>深度耦合 CoreDNS,使用 Go 编写插件,直接编译进 CoreDNS 二进制文件</li> <li>通过 GRPC 解耦,任意语言编写 GRPC 接口实现,CoreDNS 通过 GRPC 与插件交互</li> </ul> <p>由于 GRPC 链接实际上借助于 CoreDNS 的 GRPC 插件,同时 GRPC 会有网络开销,TCP 链接不稳定可能造成 DNS 响应过慢等问题,所以本文只介绍如何使用 Go 编写 CoreDNS 的插件,这种插件将直接编译进 CoreDNS 二进制文件中</p> <h3 id="22插件注册">2.2、插件注册</h3> <p>在通常情况下,插件中应当包含一个 <code class="highlighter-rouge">setup.go</code> 文件,这个文件的 <code class="highlighter-rouge">init</code> 方法调用插件注册,类似这样</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">init</span><span class="p">()</span> <span class="p">{</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Register</span><span class="p">(</span><span class="s">"gdns"</span><span class="p">,</span> <span class="n">setup</span><span class="p">)</span> <span class="p">}</span> </code></pre></div></div> <p>注册方法的第一个参数是插件名称,第二个是一个 func,func 签名如下</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// SetupFunc is used to set up a plugin, or in other words,</span> <span class="c">// execute a directive. It will be called once per key for</span> <span class="c">// each server block it appears in.</span> <span class="k">type</span> <span class="n">SetupFunc</span> <span class="k">func</span><span class="p">(</span><span class="n">c</span> <span class="o">*</span><span class="n">Controller</span><span class="p">)</span> <span class="kt">error</span> </code></pre></div></div> <p><strong>在这个 SetupFunc 中,插件编写者应当通过 <code class="highlighter-rouge">*Controller</code> 拿到 CoreDNS 的配置并解析它,从而完成自己插件的初始化配置;</strong>比如你的插件需要连接 Etcd,那么在这个方法里你要通过 <code class="highlighter-rouge">*Controller</code> 遍历配置,拿到 Etcd 的地址、证书、用户名密码配置等信息;</p> <p>如果配置信息没有问题,该插件应当初始化完成;如果有问题就报错退出,然后整个 CoreDNS 启动失败;如果插件初始化完成,最后不要忘记将自己的插件加入到整个插件链路中(CoreDNS 根据情况逐个调用)</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">setup</span><span class="p">(</span><span class="n">c</span> <span class="o">*</span><span class="n">caddy</span><span class="o">.</span><span class="n">Controller</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span> <span class="n">e</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">etcdParse</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="k">return</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Error</span><span class="p">(</span><span class="s">"gdns"</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="n">dnsserver</span><span class="o">.</span><span class="n">GetConfig</span><span class="p">(</span><span class="n">c</span><span class="p">)</span><span class="o">.</span><span class="n">AddPlugin</span><span class="p">(</span><span class="k">func</span><span class="p">(</span><span class="n">next</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Handler</span><span class="p">)</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Handler</span> <span class="p">{</span> <span class="n">e</span><span class="o">.</span><span class="n">Next</span> <span class="o">=</span> <span class="n">next</span> <span class="k">return</span> <span class="n">e</span> <span class="p">})</span> <span class="k">return</span> <span class="no">nil</span> <span class="p">}</span> </code></pre></div></div> <h3 id="23插件结构体">2.3、插件结构体</h3> <p>一般来说,每一个插件都会定义一个结构体,<strong>结构体中包含必要的 CoreDNS 内置属性,以及当前插件特性的相关配置;</strong>一个样例的插件结构体如下所示</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">GDNS</span> <span class="k">struct</span> <span class="p">{</span> <span class="c">// Next 属性在 Setup 之后会被设置到下一个插件的引用,以便在本插件解析失败后可以交由下面的插件继续解析</span> <span class="n">Next</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Handler</span> <span class="c">// Fall 列表用来控制哪些域名的请求解析失败后可以继续穿透到下一个插件重新处理</span> <span class="n">Fall</span> <span class="n">fall</span><span class="o">.</span><span class="n">F</span> <span class="c">// Zones 表示当前插件应该 case 哪些域名的 DNS 请求</span> <span class="n">Zones</span> <span class="p">[]</span><span class="kt">string</span> <span class="c">// PathPrefix 和 Client 就是插件本身的业务属性了,由于插件要连 Etcd</span> <span class="c">// PathPrefix 就是 Etcd 目录前缀,Client 是一个 Etcd 的 client</span> <span class="c">// endpoints 是 Etcd api 端点的地址</span> <span class="n">PathPrefix</span> <span class="kt">string</span> <span class="n">Client</span> <span class="o">*</span><span class="n">etcdcv3</span><span class="o">.</span><span class="n">Client</span> <span class="n">endpoints</span> <span class="p">[]</span><span class="kt">string</span> <span class="c">// Stored here as well, to aid in testing.</span> <span class="p">}</span> </code></pre></div></div> <h3 id="24插件接口">2.4、插件接口</h3> <p>一个 Go 编写的 CoreDNS 插件实际上只需要实现一个 <code class="highlighter-rouge">Handler</code> 接口既可,接口定义如下</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Handler is like dns.Handler except ServeDNS may return an rcode</span> <span class="c">// and/or error.</span> <span class="c">//</span> <span class="c">// If ServeDNS writes to the response body, it should return a status</span> <span class="c">// code. CoreDNS assumes *no* reply has yet been written if the status</span> <span class="c">// code is one of the following:</span> <span class="c">//</span> <span class="c">// * SERVFAIL (dns.RcodeServerFailure)</span> <span class="c">//</span> <span class="c">// * REFUSED (dns.RecodeRefused)</span> <span class="c">//</span> <span class="c">// * FORMERR (dns.RcodeFormatError)</span> <span class="c">//</span> <span class="c">// * NOTIMP (dns.RcodeNotImplemented)</span> <span class="c">//</span> <span class="c">// All other response codes signal other handlers above it that the</span> <span class="c">// response message is already written, and that they should not write</span> <span class="c">// to it also.</span> <span class="c">//</span> <span class="c">// If ServeDNS encounters an error, it should return the error value</span> <span class="c">// so it can be logged by designated error-handling plugin.</span> <span class="c">//</span> <span class="c">// If writing a response after calling another ServeDNS method, the</span> <span class="c">// returned rcode SHOULD be used when writing the response.</span> <span class="c">//</span> <span class="c">// If handling errors after calling another ServeDNS method, the</span> <span class="c">// returned error value SHOULD be logged or handled accordingly.</span> <span class="c">//</span> <span class="c">// Otherwise, return values should be propagated down the plugin</span> <span class="c">// chain by returning them unchanged.</span> <span class="n">Handler</span> <span class="k">interface</span> <span class="p">{</span> <span class="n">ServeDNS</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span> <span class="n">dns</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="o">*</span><span class="n">dns</span><span class="o">.</span><span class="n">Msg</span><span class="p">)</span> <span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="n">Name</span><span class="p">()</span> <span class="kt">string</span> <span class="p">}</span> </code></pre></div></div> <ul> <li><code class="highlighter-rouge">ServeDNS</code> 方法是插件需要实现的主要逻辑方法,DNS 请求接受后会从这个方法传入,插件编写者需要实现查询并返回结果</li> <li><code class="highlighter-rouge">Name</code> 方法只返回一个插件名称标识,具体作用记不太清楚,好像是为了判断插件命名唯一性然后做链式顺序调用的,原则只要你不跟系统插件重名就行</li> </ul> <p><strong>基本逻辑就是在 setup 阶段通过配置文件创建你的插件结构体对象;然后插件结构体实现这个 <code class="highlighter-rouge">Handler</code> 接口,运行期 CoreDNS 会调用接口的 <code class="highlighter-rouge">ServeDNS</code> 方法来向插件查询 DNS 请求</strong></p> <h3 id="25servedns-方法">2.5、ServeDNS 方法</h3> <p>ServeDNS 方法入参有 3 个:</p> <ul> <li><code class="highlighter-rouge">context.Context</code> 用来控制超时等情况的 context</li> <li><code class="highlighter-rouge">dns.ResponseWriter</code> 插件通过这个对象写入对 Client DNS 请求的响应结果</li> <li><code class="highlighter-rouge">*dns.Msg</code> 这个是 Client 发起的 DNS 请求,插件负责处理它,比如当你发现请求类型是 <code class="highlighter-rouge">AAAA</code> 而你的插件又不想去支持时要如何返回结果</li> </ul> <p>对于返回结果,插件编写者应当通过 <code class="highlighter-rouge">dns.ResponseWriter.WriteMsg</code> 方法写入返回结果,基本代码如下</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// ServeDNS implements the plugin.Handler interface.</span> <span class="k">func</span> <span class="p">(</span><span class="n">gDNS</span> <span class="o">*</span><span class="n">GDNS</span><span class="p">)</span> <span class="n">ServeDNS</span><span class="p">(</span><span class="n">ctx</span> <span class="n">context</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span> <span class="n">w</span> <span class="n">dns</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">dns</span><span class="o">.</span><span class="n">Msg</span><span class="p">)</span> <span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> <span class="c">// ...... 这里应当实现你的业务逻辑,查找相应的 DNS 记录</span> <span class="c">// 最后通过 new 一个 dns.Msg 作为返回结果</span> <span class="n">resp</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="n">dns</span><span class="o">.</span><span class="n">Msg</span><span class="p">)</span> <span class="n">resp</span><span class="o">.</span><span class="n">SetReply</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="n">resp</span><span class="o">.</span><span class="n">Authoritative</span> <span class="o">=</span> <span class="no">true</span> <span class="c">// records 是真正的记录结果,应当在业务逻辑区准备好</span> <span class="n">resp</span><span class="o">.</span><span class="n">Answer</span> <span class="o">=</span> <span class="nb">append</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">Answer</span><span class="p">,</span> <span class="n">records</span><span class="o">...</span><span class="p">)</span> <span class="c">// 返回结果</span> <span class="n">err</span> <span class="o">=</span> <span class="n">w</span><span class="o">.</span><span class="n">WriteMsg</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Error</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="c">// 告诉 CoreDNS 是否处理成功</span> <span class="k">return</span> <span class="n">dns</span><span class="o">.</span><span class="n">RcodeSuccess</span><span class="p">,</span> <span class="no">nil</span> <span class="p">}</span> </code></pre></div></div> <p><strong>需要注意的是,无论根据业务逻辑是否查询到 DNS 记录,都要返回响应结果(没有就返回空),错误或者未返回将会导致 Client 端查询 DNS 超时,然后不断重试,最终可能导致 Client 端服务故障</strong></p> <h3 id="26name-方法">2.6、Name 方法</h3> <p><code class="highlighter-rouge">Name</code> 方法非常简单,只需要返回当前插件名称既可;该方法的作用是为了其他插件判断本插件是否加载等情况</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Name implements the Handler interface.</span> <span class="k">func</span> <span class="p">(</span><span class="n">gDNS</span> <span class="o">*</span><span class="n">GDNS</span><span class="p">)</span> <span class="n">Name</span><span class="p">()</span> <span class="kt">string</span> <span class="p">{</span> <span class="k">return</span> <span class="s">"gdns"</span> <span class="p">}</span> </code></pre></div></div> <h2 id="三coredns-插件处理">三、CoreDNS 插件处理</h2> <p>对于实际的业务处理,可以通过 <code class="highlighter-rouge">case</code> 请求 <code class="highlighter-rouge">QType</code> 来做具体的业务实现</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// ServeDNS implements the plugin.Handler interface.</span> <span class="k">func</span> <span class="p">(</span><span class="n">gDNS</span> <span class="o">*</span><span class="n">GDNS</span><span class="p">)</span> <span class="n">ServeDNS</span><span class="p">(</span><span class="n">ctx</span> <span class="n">context</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span> <span class="n">w</span> <span class="n">dns</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">dns</span><span class="o">.</span><span class="n">Msg</span><span class="p">)</span> <span class="p">(</span><span class="kt">int</span><span class="p">,</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span> <span class="n">state</span> <span class="o">:=</span> <span class="n">request</span><span class="o">.</span><span class="n">Request</span><span class="p">{</span><span class="n">W</span><span class="o">:</span> <span class="n">w</span><span class="p">,</span> <span class="n">Req</span><span class="o">:</span> <span class="n">r</span><span class="p">}</span> <span class="n">zone</span> <span class="o">:=</span> <span class="n">plugin</span><span class="o">.</span><span class="n">Zones</span><span class="p">(</span><span class="n">gDNS</span><span class="o">.</span><span class="n">Zones</span><span class="p">)</span><span class="o">.</span><span class="n">Matches</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">Name</span><span class="p">())</span> <span class="k">if</span> <span class="n">zone</span> <span class="o">==</span> <span class="s">""</span> <span class="p">{</span> <span class="k">return</span> <span class="n">plugin</span><span class="o">.</span><span class="n">NextOrFailure</span><span class="p">(</span><span class="n">gDNS</span><span class="o">.</span><span class="n">Name</span><span class="p">(),</span> <span class="n">gDNS</span><span class="o">.</span><span class="n">Next</span><span class="p">,</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">r</span><span class="p">)</span> <span class="p">}</span> <span class="c">// ...业务处理</span> <span class="k">switch</span> <span class="n">state</span><span class="o">.</span><span class="n">QType</span><span class="p">()</span> <span class="p">{</span> <span class="k">case</span> <span class="n">dns</span><span class="o">.</span><span class="n">TypeA</span><span class="o">:</span> <span class="c">// A 记录查询业务逻辑</span> <span class="k">case</span> <span class="n">dns</span><span class="o">.</span><span class="n">TypeAAAA</span><span class="o">:</span> <span class="c">// AAAA 记录查询业务逻辑</span> <span class="k">default</span><span class="o">:</span> <span class="k">return</span> <span class="no">false</span> <span class="n">resp</span> <span class="o">:=</span> <span class="nb">new</span><span class="p">(</span><span class="n">dns</span><span class="o">.</span><span class="n">Msg</span><span class="p">)</span> <span class="n">resp</span><span class="o">.</span><span class="n">SetReply</span><span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="n">resp</span><span class="o">.</span><span class="n">Authoritative</span> <span class="o">=</span> <span class="no">true</span> <span class="n">resp</span><span class="o">.</span><span class="n">Answer</span> <span class="o">=</span> <span class="nb">append</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">Answer</span><span class="p">,</span> <span class="n">records</span><span class="o">...</span><span class="p">)</span> <span class="n">err</span> <span class="o">=</span> <span class="n">w</span><span class="o">.</span><span class="n">WriteMsg</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Error</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">return</span> <span class="n">dns</span><span class="o">.</span><span class="n">RcodeSuccess</span><span class="p">,</span> <span class="no">nil</span> <span class="p">}</span> </code></pre></div></div> <h2 id="四插件编译及测试">四、插件编译及测试</h2> <h3 id="41官方标准操作">4.1、官方标准操作</h3> <p>根据官方文档的描述,当你编写好插件以后,<strong>你的插件应当提交到一个 Git 仓库中,可以使 Github 等(保证可以 <code class="highlighter-rouge">go get</code> 拉取就行),然后修改 <code class="highlighter-rouge">plugin.cfg</code>,最后执行 <code class="highlighter-rouge">make</code> 既可</strong>;具体修改如下所示</p> <p><img src="http://cdn.oss.link/markdown/vey4u.png" alt="plugin.cfg" /></p> <p><strong>值得注意的是: 插件配置在 <code class="highlighter-rouge">plugin.cfg</code> 内的顺序决定了插件的执行顺序;通俗的讲,如果 Client 的一个 DNS 请求进来,CoreDNS 根据你在 <code class="highlighter-rouge">plugin.cfg</code> 内书写的顺序依次调用,而并非 <code class="highlighter-rouge">Corefile</code> 内的配置顺序</strong></p> <p>配置好以后直接执行 <code class="highlighter-rouge">make</code> 既可编译成功一个包含自定义插件的 CoreDNS 二进制文件(编译过程的 <code class="highlighter-rouge">go mod</code> 下载加速问题不在本文讨论范围内);你可以直接通过这个二进制测试插件的处理情况,当然这种测试不够直观,而且频繁修改由于 <code class="highlighter-rouge">go mod</code> 缓存等原因并不一定能保证每次编译的都包含最新插件代码,所以另一种方式请看下一章节</p> <h3 id="42经验性的操作">4.2、经验性的操作</h3> <p>根据个人测试以及对源码的分析,在修改 <code class="highlighter-rouge">plugin.cfg</code> 然后执行 <code class="highlighter-rouge">make</code> 命令后,实际上是进行了代码生成;当你通过 git 命令查看相关修改文件时,整个插件加载体系便没什么秘密可言了;<strong>在整个插件体系中,插件加载是通过 <code class="highlighter-rouge">init</code> 方法注册的,那么既然用 go 写插件,那么应该清楚 <code class="highlighter-rouge">init</code> 方法只有在包引用之后才会执行,所以整个插件体系实际上是这样事儿的:</strong></p> <p>首先 <code class="highlighter-rouge">make</code> 以后会修改 <code class="highlighter-rouge">core/plugin/zplugin.go</code> 文件,这个文件啥也不干,就是 <code class="highlighter-rouge">import</code> 来实现调用对应包的 <code class="highlighter-rouge">init</code> 方法</p> <p><img src="http://cdn.oss.link/markdown/ny1rz.png" alt="zplugin.go" /></p> <p>当 <code class="highlighter-rouge">init</code> 执行后你去追源码,实际上就是 Caddy 维护了一个 <code class="highlighter-rouge">map[string]Plugin</code>,<code class="highlighter-rouge">init</code> 会把你的插件 func 塞进去然后后面再调用,实现一个懒加载或者说延迟初始化</p> <p><img src="http://cdn.oss.link/markdown/idno4.png" alt="caddy_plugin" /></p> <p>接着修改了一下 <code class="highlighter-rouge">core/dnsserver/zdirectives.go</code>,这个里面也没啥,就是一个 <code class="highlighter-rouge">[]string</code>,<strong>但是 <code class="highlighter-rouge">[]string</code> 这玩意有顺序啊,这就是为什么你在 <code class="highlighter-rouge">plugin.cfg</code> 里写的顺序决定了插件处理顺序的原因(因为生成的这个切片有顺序)</strong></p> <p><img src="http://cdn.oss.link/markdown/bixos.png" alt="zdirectives.go" /></p> <p>综上所述,实际上 <code class="highlighter-rouge">make</code> 命令一共修改了两个文件,如果想在 IDE 内直接 debug CoreDNS + Plugin 源码,那么只需要这样做:</p> <p>复制自己编写的插件目录到 <code class="highlighter-rouge">plugin</code> 目录,类似这样</p> <p><img src="http://cdn.oss.link/markdown/whwuy.png" alt="gdns" /></p> <p>手动修改 <code class="highlighter-rouge">core/plugin/zplugin.go</code>,加入自己插件的 <code class="highlighter-rouge">import</code>(此时你直接复制系统其他插件,改一下目录名既可)</p> <p><img src="http://cdn.oss.link/markdown/g7wp0.png" alt="update_zplugin" /></p> <p>手动修改 <code class="highlighter-rouge">core/dnsserver/zdirectives.go</code> 把自己插件名称写进去(自己控制顺序),然后 debug 启动 <code class="highlighter-rouge">coredns.go</code> 里面的 main 方法测试既可</p> <p><img src="http://cdn.oss.link/markdown/4ucqg.png" alt="coredns.go" /></p> <h2 id="五本文参考">五、本文参考</h2> <ul> <li>Writing Plugins for CoreDNS: http://coredns.io/2016/12/19/writing-plugins-for-coredns</li> <li>how-to-add-plugins.md: http://github.com/coredns/coredns.io/blob/master/content/blog/how-to-add-plugins.md</li> <li>example plugin: http://github.com/coredns/example</li> </ul> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 05 Nov 2019 20:55:53 +0800 /2019/11/05/writing-plugin-for-coredns/ /2019/11/05/writing-plugin-for-coredns/ Golang Golang Golang Etcd client example <blockquote> <p>准备开发点东西,需要用到 Etcd,由于生产 Etcd 全部开启了 TLS 加密,所以客户端需要相应修改,以下为 Golang 链接 Etcd 并且使用客户端证书验证的样例代码</p> </blockquote> <h2 id="api-v2">API V2</h2> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="k">import</span> <span class="p">(</span> <span class="s">"context"</span> <span class="s">"crypto/tls"</span> <span class="s">"crypto/x509"</span> <span class="s">"io/ioutil"</span> <span class="s">"log"</span> <span class="s">"net"</span> <span class="s">"net/http"</span> <span class="s">"time"</span> <span class="s">"go.etcd.io/etcd/client"</span> <span class="p">)</span> <span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="c">// 为了保证 http 链接可信,需要预先加载目标证书签发机构的 CA 根证书</span> <span class="n">etcdCA</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">ioutil</span><span class="o">.</span><span class="n">ReadFile</span><span class="p">(</span><span class="s">"/Users/mritd/tmp/etcd_ssl/etcd-root-ca.pem"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="c">// etcd 启用了双向 TLS 认证,所以客户端证书同样需要加载</span> <span class="n">etcdClientCert</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">tls</span><span class="o">.</span><span class="n">LoadX509KeyPair</span><span class="p">(</span><span class="s">"/Users/mritd/tmp/etcd_ssl/etcd.pem"</span><span class="p">,</span> <span class="s">"/Users/mritd/tmp/etcd_ssl/etcd-key.pem"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="c">// 创建一个空的 CA Pool</span> <span class="c">// 因为后续只会链接 Etcd 的 api 端点,所以此处选择使用空的 CA Pool,然后只加入 Etcd CA 既可</span> <span class="c">// 如果期望链接其他 TLS 端点,那么最好使用 x509.SystemCertPool() 方法先 copy 一份系统根 CA</span> <span class="c">// 然后再向这个 Pool 中添加自定义 CA</span> <span class="n">rootCertPool</span> <span class="o">:=</span> <span class="n">x509</span><span class="o">.</span><span class="n">NewCertPool</span><span class="p">()</span> <span class="n">rootCertPool</span><span class="o">.</span><span class="n">AppendCertsFromPEM</span><span class="p">(</span><span class="n">etcdCA</span><span class="p">)</span> <span class="n">cfg</span> <span class="o">:=</span> <span class="n">client</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span> <span class="c">// Etcd http api 端点</span> <span class="n">Endpoints</span><span class="o">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"http://172.16.14.114:2379"</span><span class="p">},</span> <span class="c">// 自定义 Transport 实现自签 CA 加载以及 Client Cert 加载</span> <span class="c">// 其他参数最好从 client.DefaultTranspor copy,以保证与默认 client 相同的行为</span> <span class="n">Transport</span><span class="o">:</span> <span class="o">&amp;</span><span class="n">http</span><span class="o">.</span><span class="n">Transport</span><span class="p">{</span> <span class="n">Proxy</span><span class="o">:</span> <span class="n">http</span><span class="o">.</span><span class="n">ProxyFromEnvironment</span><span class="p">,</span> <span class="c">// Dial 方法已被启用,采用新的 DialContext 设置超时</span> <span class="n">DialContext</span><span class="o">:</span> <span class="p">(</span><span class="o">&amp;</span><span class="n">net</span><span class="o">.</span><span class="n">Dialer</span><span class="p">{</span> <span class="n">KeepAlive</span><span class="o">:</span> <span class="m">30</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span> <span class="n">Timeout</span><span class="o">:</span> <span class="m">30</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">})</span><span class="o">.</span><span class="n">DialContext</span><span class="p">,</span> <span class="c">// 自定义 CA 及 Client Cert 配置</span> <span class="n">TLSClientConfig</span><span class="o">:</span> <span class="o">&amp;</span><span class="n">tls</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span> <span class="n">RootCAs</span><span class="o">:</span> <span class="n">rootCertPool</span><span class="p">,</span> <span class="n">Certificates</span><span class="o">:</span> <span class="p">[]</span><span class="n">tls</span><span class="o">.</span><span class="n">Certificate</span><span class="p">{</span><span class="n">etcdClientCert</span><span class="p">},</span> <span class="p">},</span> <span class="n">TLSHandshakeTimeout</span><span class="o">:</span> <span class="m">10</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">},</span> <span class="c">// set timeout per request to fail fast when the target endpoint is unavailable</span> <span class="n">HeaderTimeoutPerRequest</span><span class="o">:</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span> <span class="p">}</span> <span class="n">c</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">client</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="n">cfg</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="n">kapi</span> <span class="o">:=</span> <span class="n">client</span><span class="o">.</span><span class="n">NewKeysAPI</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> <span class="c">// set "/foo" key with "bar" value</span> <span class="n">log</span><span class="o">.</span><span class="n">Print</span><span class="p">(</span><span class="s">"Setting '/foo' key with 'bar' value"</span><span class="p">)</span> <span class="n">resp</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">kapi</span><span class="o">.</span><span class="n">Set</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span> <span class="s">"/foo"</span><span class="p">,</span> <span class="s">"bar"</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="c">// print common key info</span> <span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Set is done. Metadata is %q</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">resp</span><span class="p">)</span> <span class="p">}</span> <span class="c">// get "/foo" key's value</span> <span class="n">log</span><span class="o">.</span><span class="n">Print</span><span class="p">(</span><span class="s">"Getting '/foo' key value"</span><span class="p">)</span> <span class="n">resp</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">kapi</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span> <span class="s">"/foo"</span><span class="p">,</span> <span class="no">nil</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="c">// print common key info</span> <span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Get is done. Metadata is %q</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">resp</span><span class="p">)</span> <span class="c">// print value</span> <span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%q key has %q value</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">resp</span><span class="o">.</span><span class="n">Node</span><span class="o">.</span><span class="n">Key</span><span class="p">,</span> <span class="n">resp</span><span class="o">.</span><span class="n">Node</span><span class="o">.</span><span class="n">Value</span><span class="p">)</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <h2 id="api-v3">API V3</h2> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span> <span class="k">import</span> <span class="p">(</span> <span class="s">"context"</span> <span class="s">"crypto/tls"</span> <span class="s">"crypto/x509"</span> <span class="s">"io/ioutil"</span> <span class="s">"log"</span> <span class="s">"time"</span> <span class="s">"go.etcd.io/etcd/clientv3"</span> <span class="p">)</span> <span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span> <span class="c">// 为了保证 http 链接可信,需要预先加载目标证书签发机构的 CA 根证书</span> <span class="n">etcdCA</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">ioutil</span><span class="o">.</span><span class="n">ReadFile</span><span class="p">(</span><span class="s">"/Users/mritd/tmp/etcd_ssl/etcd-root-ca.pem"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="c">// etcd 启用了双向 TLS 认证,所以客户端证书同样需要加载</span> <span class="n">etcdClientCert</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">tls</span><span class="o">.</span><span class="n">LoadX509KeyPair</span><span class="p">(</span><span class="s">"/Users/mritd/tmp/etcd_ssl/etcd.pem"</span><span class="p">,</span> <span class="s">"/Users/mritd/tmp/etcd_ssl/etcd-key.pem"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="c">// 创建一个空的 CA Pool</span> <span class="c">// 因为后续只会链接 Etcd 的 api 端点,所以此处选择使用空的 CA Pool,然后只加入 Etcd CA 既可</span> <span class="c">// 如果期望链接其他 TLS 端点,那么最好使用 x509.SystemCertPool() 方法先 copy 一份系统根 CA</span> <span class="c">// 然后再向这个 Pool 中添加自定义 CA</span> <span class="n">rootCertPool</span> <span class="o">:=</span> <span class="n">x509</span><span class="o">.</span><span class="n">NewCertPool</span><span class="p">()</span> <span class="n">rootCertPool</span><span class="o">.</span><span class="n">AppendCertsFromPEM</span><span class="p">(</span><span class="n">etcdCA</span><span class="p">)</span> <span class="c">// 创建 api v3 的 client</span> <span class="n">cli</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">clientv3</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="n">clientv3</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span> <span class="c">// etcd http api 端点</span> <span class="n">Endpoints</span><span class="o">:</span> <span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"http://172.16.14.114:2379"</span><span class="p">},</span> <span class="n">DialTimeout</span><span class="o">:</span> <span class="m">5</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span> <span class="c">// 自定义 CA 及 Client Cert 配置</span> <span class="n">TLS</span><span class="o">:</span> <span class="o">&amp;</span><span class="n">tls</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span> <span class="n">RootCAs</span><span class="o">:</span> <span class="n">rootCertPool</span><span class="p">,</span> <span class="n">Certificates</span><span class="o">:</span> <span class="p">[]</span><span class="n">tls</span><span class="o">.</span><span class="n">Certificate</span><span class="p">{</span><span class="n">etcdClientCert</span><span class="p">},</span> <span class="p">},</span> <span class="p">})</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">defer</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span> <span class="n">_</span> <span class="o">=</span> <span class="n">cli</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span> <span class="p">}()</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">cancel</span> <span class="o">:=</span> <span class="n">context</span><span class="o">.</span><span class="n">WithTimeout</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span> <span class="m">3</span><span class="o">*</span><span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">)</span> <span class="n">putResp</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">cli</span><span class="o">.</span><span class="n">Put</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="s">"sample_key"</span><span class="p">,</span> <span class="s">"sample_value"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">putResp</span><span class="p">)</span> <span class="p">}</span> <span class="n">cancel</span><span class="p">()</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">cancel</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">WithTimeout</span><span class="p">(</span><span class="n">context</span><span class="o">.</span><span class="n">Background</span><span class="p">(),</span> <span class="m">3</span><span class="o">*</span><span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">)</span> <span class="n">delResp</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">cli</span><span class="o">.</span><span class="n">Delete</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="s">"sample_key"</span><span class="p">)</span> <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="n">log</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">delResp</span><span class="p">)</span> <span class="p">}</span> <span class="n">cancel</span><span class="p">()</span> <span class="p">}</span> </code></pre></div></div> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Wed, 16 Oct 2019 10:59:03 +0800 /2019/10/16/golang-etcd-client-example/ /2019/10/16/golang-etcd-client-example/ Golang Golang Podman 初试 - 容器发展史 <blockquote> <p>这是一篇纯介绍性文章,本文不包含任何技术层面的操作,本文仅作为后续 Podman 文章铺垫;本文细节部份并未阐述,很多地方并不详实(一家只谈,不可轻信)。</p> </blockquote> <h2 id="一缘起">一、缘起</h2> <h3 id="11鸿蒙">1.1、鸿蒙</h3> <p>在上古时期,天地初开,一群称之为 “运维” 的人们每天在一种叫作 “服务器” 的神秘盒子中创造属于他们的世界;他们在这个世界中每日劳作,一遍又一遍的写入他们的历史,比如搭建一个 nginx、布署一个 java web 应用…</p> <p>大多数人其实并没有那么聪明,他们所 “创造” 的事实上可能是有人已经创造过的东西,他们可能每天都在做着重复的劳动;久而久之,一些人厌倦了、疲惫了…又过了一段时间,一些功力深厚的老前辈创造了一些批量布署工具来帮助人们做一些重复性的劳动,这些工具被起名为 “Asible”、”Chef”、”Puppet” 等等…</p> <p>而随着时代的发展,”世界” 变得越来越复杂,运维们需要处理的事情越来越多,比如各种网络、磁盘环境的隔离,各种应用服务的高可用…在时代的洪流下,运维们急需要一种简单高效的布署工具,既能有一定的隔离性,又能方便使用,并且最大程度降低重复劳动来提升效率。</p> <h3 id="12创世">1.2、创世</h3> <p>在时代洪流的冲击下,一位名为 “Solomon Hykes” 的人异军突起,他创造了一个称之为 Docker 的工具,Docker 被创造以后就以灭世之威向运维们展示了它的强大;一个战斗力只有 5 的运维只需要学习 Docker 很短时间就可以完成资深运维们才能完成的事情,在某些情况下以前需要 1 天才能完成的工作使用 Docker 后几分钟就可以完成;此时运维们已经意识到 “新的时代” 开启了,接下来 Docker 开源并被整个运维界人们使用,Docker 也不断地完善增加各种各样的功能,此后世界正式进入 “容器纪元”。</p> <h2 id="二纷争">二、纷争</h2> <h3 id="21发展">2.1、发展</h3> <p>随着 Docker 的日益成熟,一些人开始在 Docker 之上创造更加强大的工具,一些人开始在 Docker 之下为其提供更稳定的运行环境…</p> <p>其中一个叫作 Google 的公司在 Docker 之上创建了名为 “Kuberentes” 的工具,Kubernetes 操纵 Docker 完成更加复杂的任务;Kubernetes 的出现更加印证了 Docker 的强大,以及 “容器纪元” 的发展正确性。</p> <h3 id="22野心">2.2、野心</h3> <p>当然这是一个充满利益的世界,Google 公司创造 Kubernetes 是可以为他们带来利益的,比如他们可以让 Kubernetes 深度适配他们的云平台,以此来增加云平台的销量等;此时 Docker 创始人也成立了一个公司,提供 Docker 的付费服务以及深度定制等;不过值得一提的是 Docker 公司提供的付费服务始终没有 Kubernetes 为 Google 公司带来的利益高,所以在利益的驱使下,Docker 公司开始动起了歪心思: <strong>创造一个 Kubernetes 的替代品,利用用户粘度复制 Kubernetes 的成功,从 Google 嘴里抢下这块蛋糕!</strong>此时 Docker 公司只想把蛋糕抢过来,但是他们根本没有在意到暗中一群人创造了一个叫 “rkt” 的东西也在妄图夺走他们嘴里的蛋糕。</p> <h3 id="23冲突">2.3、冲突</h3> <p>在一段时间的沉默后,Docker 公司又创造了 “Swarm” 这个工具,妄图夺走 Google 公司利用 Kubernetes 赢来的蛋糕;当然,Google 这个公司极其庞大,人数众多,而且在这个社会有很大的影响地位…</p> <p>终于,巨人苏醒了,Google 联合了 Redhat、Microsoft、IBM、Intel、Cisco 等公司决定对这个爱动歪脑筋的 Docker 公司进行制裁;当然制裁的手段不能过于暴力,那样会让别人落下把柄,成为别人的笑料,被人所不耻;<strong>最总他们决定制订规范,成立组织,明确规定 Docker 的角色,以及它应当拥有的能力,这些规范包括但不限于 <code class="highlighter-rouge">CRI</code>、<code class="highlighter-rouge">CNI</code> 等;自此之后各大公司宣布他们容器相关的工具只兼容 CRI 等相关标准,无论是 Docker 还是 rkt 等工具,只要实现了这些标准,就可以配合这些容器工具进行使用</strong>。</p> <h2 id="三成败">三、成败</h2> <p>自此之后,Docker 跌下神坛,各路大神纷纷创造满足 CRI 等规范的工具用来取代 Docker,Docker 丢失了往日一家独大的场面,最终为了顺应时代发展,拆分自己成为模块化组件;这些模块化组件被放置在 <a href="http://mobyproject.org/">mobyproject</a> 中方便其他人重复利用。</p> <p>时至今日,虽然 Docker 已经不负以前,但是仍然是容器化首选工具,因为 Docker 是一个完整的产品,它可以提供除了满足 CRI 等标准以外更加方便的功能;但是制裁并非没有结果,Google 公司借此创造了 cri-o 用来满足 CRI 标准,其他公司也相应创建了对应的 CRI 实现;<strong>为了进一步分化 Docker 势力,一个叫作 Podman 的工具被创建,它以 cri-o 为基础,兼容大部份 Docker 命令的方式开始抢夺 Dcoker 用户</strong>;到目前为止 Podman 已经可以在大部份功能上替代 Docker。</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Wed, 26 Jun 2019 23:22:49 +0800 /2019/06/26/podman-history-of-container/ /2019/06/26/podman-history-of-container/ Docker Podman Docker Podman Calico 3.6 转发外部流量到集群 Pod <blockquote> <p>由于开发有部份服务使用 GRPC 进行通讯,同时采用 Consul 进行服务发现;在微服务架构下可能会导致一些访问问题,目前解决方案就是打通开发环境网络与测试环境 Kubernetes 内部 Pod 网络;翻了好多资料发现都是 2.x 的,而目前测试集群 Calico 版本为 3.6.3,很多文档都不适用只能自己折腾,目前折腾完了这里记录一下</p> </blockquote> <p><strong>本文默认为读者已经存在一个运行正常的 Kubernetes 集群,并且采用 Calico 作为 CNI 组件,且 Calico 工作正常;同时应当在某个节点完成了 calicoctl 命令行工具的配置</strong></p> <h2 id="一问题描述">一、问题描述</h2> <p>在微服务架构下,由于服务组件很多,开发在本地机器想测试应用需要启动整套服务,这对开发机器的性能确实是个考验;但如果直接连接测试环境的服务,由于服务发现问题最终得到的具体服务 IP 是 Kubernetes Pod IP,此 IP 由集群内部 Calico 维护与分配,外部不可访问;最终目标为打通开发环境与集群内部网络,实现开发网络下直连 Pod IP,这或许在以后对生产服务暴露负载均衡有一定帮助意义;目前网络环境如下:</p> <p>开发网段: <code class="highlighter-rouge">10.10.0.0/24</code> 测试网段: <code class="highlighter-rouge">172.16.0.0/24</code> Kubernetes Pod 网段: <code class="highlighter-rouge">10.20.0.0/16</code></p> <h2 id="二打通网络">二、打通网络</h2> <p>首先面临的第一个问题是 Calico 处理,因为<strong>如果想要让数据包能从开发网络到达 Pod 网络,那么必然需要测试环境宿主机上的 Calico Node 帮忙转发</strong>;因为 Pod 网络由 Calico 维护,只要 Calico Node 帮忙转发那么数据一定可以到达 Pod IP 上;</p> <p>一开始我很天真的认为这就是个 <code class="highlighter-rouge">ip route add 10.20.0.0/16 via 172.16.0.13</code> 的问题… 后来发现</p> <p><img src="http://cdn.oss.link/markdown/hwp9s.jpg" alt="没那么简单" /></p> <p>经过翻文档、issue、blog 等最终发现需要进行以下步骤</p> <h3 id="21关闭全互联模式">2.1、关闭全互联模式</h3> <p><strong>注意: 关闭全互联时可能导致网络暂时中断,请在夜深人静时操作</strong></p> <p>首先执行以下命令查看是否存在默认的 BGP 配置</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl get bgpconfig default </code></pre></div></div> <p>如果存在则将其保存为配置文件</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl get bgpconfig default <span class="nt">-o</span> yaml <span class="o">&gt;</span> bgp.yaml </code></pre></div></div> <p>修改其中的 <code class="highlighter-rouge">spec.nodeToNodeMeshEnabled</code> 为 <code class="highlighter-rouge">false</code>,然后进行替换</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl apply <span class="nt">-f</span> bgp.yaml </code></pre></div></div> <p>如果不存在则手动创建一个配置,然后应用</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nb">cat</span> <span class="o">&lt;&lt;</span> <span class="no">EOF</span><span class="sh"> | calicoctl create -f - apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false asNumber: 63400 </span><span class="no">EOF </span></code></pre></div></div> <p>本部分参考:</p> <ul> <li><a href="http://docs.projectcalico.org/v3.6/networking/bgp">Disabling the full node-to-node BGP mesh</a></li> </ul> <h3 id="22开启集群内-rr-模式">2.2、开启集群内 RR 模式</h3> <p>在 Calico 3.3 后支持了集群内节点的 RR 模式,即将某个集群内的 Calico Node 转变为 RR 节点;将某个节点设置为 RR 节点只需要增加 <code class="highlighter-rouge">routeReflectorClusterID</code> 既可,为了后面方便配置同时增加了一个 lable 字段 <code class="highlighter-rouge">route-reflector: "true"</code></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl get node CALICO_NODE_NAME <span class="nt">-o</span> yaml <span class="o">&gt;</span> node.yaml </code></pre></div></div> <p>然后增加 <code class="highlighter-rouge">routeReflectorClusterID</code> 字段,样例如下</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">projectcalico.org/v3</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Node</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">annotations</span><span class="pi">:</span> <span class="s">projectcalico.org/kube-labels</span><span class="pi">:</span> <span class="s1">'</span><span class="s">{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"d13.node","node-role.kubernetes.io/k8s-master":"true"}'</span> <span class="na">creationTimestamp</span><span class="pi">:</span> <span class="s">2019-06-17T13:55:44Z</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">beta.kubernetes.io/arch</span><span class="pi">:</span> <span class="s">amd64</span> <span class="s">beta.kubernetes.io/os</span><span class="pi">:</span> <span class="s">linux</span> <span class="s">kubernetes.io/hostname</span><span class="pi">:</span> <span class="s">d13.node</span> <span class="s">node-role.kubernetes.io/k8s-master</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="na">route-reflector</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="c1"># 增加 lable</span> <span class="na">name</span><span class="pi">:</span> <span class="s">d13.node</span> <span class="na">resourceVersion</span><span class="pi">:</span> <span class="s2">"</span><span class="s">61822269"</span> <span class="na">uid</span><span class="pi">:</span> <span class="s">9a1897e0-9107-11e9-bc1c-90b11c53d1e3</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">bgp</span><span class="pi">:</span> <span class="na">ipv4Address</span><span class="pi">:</span> <span class="s">172.16.0.13/19</span> <span class="na">ipv4IPIPTunnelAddr</span><span class="pi">:</span> <span class="s">10.20.73.82</span> <span class="na">routeReflectorClusterID</span><span class="pi">:</span> <span class="s">172.16.20.1</span> <span class="c1"># 添加集群 ID</span> <span class="na">orchRefs</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">nodeName</span><span class="pi">:</span> <span class="s">d13.node</span> <span class="na">orchestrator</span><span class="pi">:</span> <span class="s">k8s</span> </code></pre></div></div> <p><strong>事实上我们应当导出多个 Calico Node 的配置,并将其配置为 RR 节点以进行冗余;对于 <code class="highlighter-rouge">routeReflectorClusterID</code> 目前测试只是作为一个 ID(至少在本文是这样的),所以理论上可以是任何 IP,个人猜测最好在同一集群网络下采用相同的 IP,由于这是真正的测试环境我没有对 ID 做过多的测试(怕玩挂)</strong></p> <p>修改完成后只需要应用一下就行</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl apply <span class="nt">-f</span> node.yaml </code></pre></div></div> <p>接下来需要创建对等规则,规则文件如下</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">kind</span><span class="pi">:</span> <span class="s">BGPPeer</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">projectcalico.org/v3</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">peer-to-rrs</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s2">"</span><span class="s">!has(route-reflector)"</span> <span class="na">peerSelector</span><span class="pi">:</span> <span class="s">has(route-reflector)</span> <span class="nn">---</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">BGPPeer</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">projectcalico.org/v3</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">rr-mesh</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s">has(route-reflector)</span> <span class="na">peerSelector</span><span class="pi">:</span> <span class="s">has(route-reflector)</span> </code></pre></div></div> <p>假定规则文件名称为 <code class="highlighter-rouge">rr.yaml</code>,则创建命令为 <code class="highlighter-rouge">calicoctl create -f rr.yaml</code>;此时在 RR 节点上使用 <code class="highlighter-rouge">calicoctl node status</code> 应该能看到类似如下输出</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.16.0.19 | node specific | up | 05:43:51 | Established | | 172.16.0.16 | node specific | up | 05:43:51 | Established | | 172.16.0.17 | node specific | up | 05:43:51 | Established | | 172.16.0.13 | node specific | up | 13:01:17 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. </code></pre></div></div> <p><strong><code class="highlighter-rouge">PEER ADDRESS</code> 应当包含所有非 RR 节点 IP(由于真实测试环境,以上输出已人为修改)</strong></p> <p>同时在非 RR 节点上使用 <code class="highlighter-rouge">calicoctl node status</code> 应该能看到以下输出</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 172.16.0.10 | node specific | up | 05:43:51 | Established | | 172.16.0.13 | node specific | up | 13:01:20 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. </code></pre></div></div> <p><strong><code class="highlighter-rouge">PEER ADDRESS</code> 应当包含所有 RR 节点 IP,此时原本的 Pod 网络连接应当已经恢复</strong></p> <p>本部分参考:</p> <ul> <li><a href="http://www.projectcalico.org/how-does-in-cluster-route-reflection-work/">In-cluster Route Reflection</a></li> <li><a href="http://docs.projectcalico.org/v3.6/networking/bgp">Configuring in-cluster route reflectors</a></li> </ul> <h3 id="23调整-ipip-规则">2.3、调整 IPIP 规则</h3> <p>先说一下 Calico IPIP 模式的三个可选项:</p> <ul> <li><code class="highlighter-rouge">Always</code>: 永远进行 IPIP 封装(默认)</li> <li><code class="highlighter-rouge">CrossSubnet</code>: 只在跨网段时才进行 IPIP 封装,适合有 Kubernetes 节点在其他网段的情况,属于中肯友好方案</li> <li><code class="highlighter-rouge">Never</code>: 从不进行 IPIP 封装,适合确认所有 Kubernetes 节点都在同一个网段下的情况</li> </ul> <p>在默认情况下,默认的 ipPool 启用了 IPIP 封装(至少通过官方安装文档安装的 Calico 是这样),并且封装模式为 <code class="highlighter-rouge">Always</code>;这也就意味着任何时候都会在原报文上封装新 IP 地址,<strong>在这种情况下将外部流量路由到 RR 节点,RR 节点再转发进行 IPIP 封装时,可能出现网络无法联通的情况(没仔细追查,网络渣,猜测是 Pod 那边得到的源 IP 不对导致的);</strong>此时我们应当调整 IPIP 封装策略为 <code class="highlighter-rouge">CrossSubnet</code></p> <p>导出 ipPool 配置</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>calicoctl get ippool default-ipv4-ippool <span class="nt">-o</span> yaml <span class="o">&gt;</span> ippool.yaml </code></pre></div></div> <p>修改 <code class="highlighter-rouge">ipipMode</code> 值为 <code class="highlighter-rouge">CrossSubnet</code></p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">projectcalico.org/v3</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">IPPool</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">creationTimestamp</span><span class="pi">:</span> <span class="s">2019-06-17T13:55:44Z</span> <span class="na">name</span><span class="pi">:</span> <span class="s">default-ipv4-ippool</span> <span class="na">resourceVersion</span><span class="pi">:</span> <span class="s2">"</span><span class="s">61858741"</span> <span class="na">uid</span><span class="pi">:</span> <span class="s">99a82055-9107-11e9-815b-b82a72dffa9f</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">blockSize</span><span class="pi">:</span> <span class="m">26</span> <span class="na">cidr</span><span class="pi">:</span> <span class="s">10.20.0.0/16</span> <span class="na">ipipMode</span><span class="pi">:</span> <span class="s">CrossSubnet</span> <span class="na">natOutgoing</span><span class="pi">:</span> <span class="no">true</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s">all()</span> </code></pre></div></div> <p>重新使用 <code class="highlighter-rouge">calicoctl apply -f ippool.yaml</code> 应用既可</p> <p>本部分参考:</p> <ul> <li><a href="http://docs.projectcalico.org/v3.6/networking/ip-in-ip">Configuring IP-in-IP</a></li> <li><a href="http://docs.projectcalico.org/v3.6/reference/calicoctl/resources/ippool">IP pool resource</a></li> </ul> <h3 id="24增加路由联通网络">2.4、增加路由联通网络</h3> <p>万事俱备只欠东风,最后只需要在开发机器添加路由既可</p> <p>将 Pod IP <code class="highlighter-rouge">10.20.0.0/16</code> 和 Service IP <code class="highlighter-rouge">10.254.0.0/16</code> 路由到 RR 节点 <code class="highlighter-rouge">172.16.0.13</code></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Pod IP</span> ip route add 10.20.0.0/16 via 172.16.0.13 <span class="c"># Service IP</span> ip route add 10.254.0.0/16 via 172.16.0.13 </code></pre></div></div> <p>当然最方便的肯定是将这一步在开发网络的路由上做,设置完成后开发网络就可以直连集群内的 Pod IP 和 Service IP 了;至于想直接访问 Service Name 只需要调整上游 DNS 解析既可</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 18 Jun 2019 22:20:54 +0800 /2019/06/18/calico-3.6-forward-network-traffic/ /2019/06/18/calico-3.6-forward-network-traffic/ Kubernetes Kuberentes Dockerfile 目前可扩展的语法 <blockquote> <p>最近在调整公司项目的 CI,目前主要使用 GitLab CI,在尝试多阶段构建中踩了点坑,然后发现了一些有意思的玩意</p> </blockquote> <p>本文参考:</p> <ul> <li><a href="http://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md">Dockerfile frontend experimental syntaxes</a></li> <li><a href="http://medium.com/@tonistiigi/advanced-multi-stage-build-patterns-6f741b852fae">Advanced multi-stage build patterns</a></li> <li><a href="http://docs.docker.com/engine/reference/commandline/build/">docker build Document</a></li> </ul> <h2 id="一起因">一、起因</h2> <p>公司目前主要使用 GitLab CI 作为主力 CI 构建工具,而且由于机器有限,我们对一些包管理器的本地 cache 直接持久化到了本机;比如 maven 的 <code class="highlighter-rouge">.m2</code> 目录,nodejs 的 <code class="highlighter-rouge">.npm</code> 目录等;虽然我们创建了对应的私服,但是在 build 时毕竟会下载,所以当时索性调整 GitLab Runner 在每个由 GitLab Runner 启动的容器中挂载这些缓存目录(GitLab CI 在 build 时会新启动容器运行 build 任务);今天调整 nodejs 项目浪了一下,直接采用 Dockerfile 的 multi-stage build 功能进行 “Build =&gt; Package(docker image)” 的实现,基本 Dockerfile 如下</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM gozap/build as builder COPY <span class="nb">.</span> /xxxx WORKDIR /xxxx RUN <span class="nb">source</span> ~/.bashrc <span class="se">\</span> <span class="o">&amp;&amp;</span> cnpm <span class="nb">install</span> <span class="se">\</span> <span class="o">&amp;&amp;</span> cnpm run build FROM gozap/nginx-react:v1.0.0 LABEL <span class="nv">maintainer</span><span class="o">=</span><span class="s2">"mritd &lt;mritd@linux.com&gt;"</span> COPY <span class="nt">--from</span><span class="o">=</span>builder /xxxx/public /usr/share/nginx/html EXPOSE 80 STOPSIGNAL SIGTERM CMD <span class="o">[</span><span class="s2">"nginx"</span>, <span class="s2">"-g"</span>, <span class="s2">"daemon off;"</span><span class="o">]</span> </code></pre></div></div> <p>本来这个 <code class="highlighter-rouge">cnpm</code> 命令是带有 cache 的(<a href="http://github.com/Gozap/dockerfile/blob/master/build/cnpm">见这里</a>),不过运行完 build 以后发现很慢,检查宿主机 cache 目录发现根本没有 cache…然后突然感觉</p> <p><img src="http://cdn.oss.link/markdown/6ieh4.jpg" alt="事情并没有这么简单" /></p> <p>仔细想想,情况应该是这样事儿的…</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+------------+ +-------------+ +----------------+ | | | | | | | | | build | | Multi-stage | | Runner +---------------&gt;+ conatiner +-----------&gt;+ Build | | | | | | | | | | | | | +------------+ +------+------+ +----------------+ ^ | | | | +------+------+ | | | Cache | | | +-------------+ </code></pre></div></div> <p><img src="http://cdn.oss.link/markdown/9ov8m.jpg" alt="挂载不管用" /></p> <p>后来经过查阅文档,发现 Dockerfile 是有扩展语法的(当然最终我还是没用),具体请见<del>下篇文章</del>(我怕被打死)下面,<strong>先说好,下面的内容无法完美的解决上面的问题,目前只是支持了一部分功能,当然未来很可能支持类似 <code class="highlighter-rouge">IF ELSE</code> 语法、直接挂载宿主机目录等功能</strong></p> <h2 id="二开启-dockerfile-扩展语法">二、开启 Dockerfile 扩展语法</h2> <h3 id="21开启实验性功能">2.1、开启实验性功能</h3> <p>目前这个扩展语法还处于实验性功能,所以需要配置 dockerd 守护进程,修改如下</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/dockerd <span class="nt">-H</span> unix:// <span class="se">\</span> <span class="nt">--init</span> <span class="se">\</span> <span class="nt">--live-restore</span> <span class="se">\</span> <span class="nt">--data-root</span><span class="o">=</span>/data/docker <span class="se">\</span> <span class="nt">--experimental</span> <span class="se">\</span> <span class="nt">--log-driver</span> json-file <span class="se">\</span> <span class="nt">--log-opt</span> max-size<span class="o">=</span>30m <span class="se">\</span> <span class="nt">--log-opt</span> max-file<span class="o">=</span>3 </code></pre></div></div> <p>主要是 <code class="highlighter-rouge">--experimental</code> 参数,参考<a href="http://docs.docker.com/engine/reference/commandline/dockerd/#description">官方文档</a>;<strong>同时在 build 前声明 <code class="highlighter-rouge">export DOCKER_BUILDKIT=1</code> 变量</strong></p> <h3 id="22修改-dockerfile">2.2、修改 Dockerfile</h3> <p>开启实验性功能后,只需要在 Dockerfile 头部增加 <code class="highlighter-rouge"># syntax=docker/dockerfile:experimental</code> 既可;为了保证稳定性,你也可以指定具体的版本号,类似这样</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># syntax=docker/dockerfile:1.1.1-experimental</span> FROM tomcat </code></pre></div></div> <h3 id="23可用的扩展语法">2.3、可用的扩展语法</h3> <ul> <li><code class="highlighter-rouge">RUN --mount=type=bind</code></li> </ul> <p>这个是默认的挂载模式,这个允许将上下文或者镜像以可都可写/只读模式挂载到 build 容器中,可选参数如下(不翻译了)</p> <table> <thead> <tr> <th>Option</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">target</code> (required)</td> <td>Mount path.</td> </tr> <tr> <td><code class="highlighter-rouge">source</code></td> <td>Source path in the <code class="highlighter-rouge">from</code>. Defaults to the root of the <code class="highlighter-rouge">from</code>.</td> </tr> <tr> <td><code class="highlighter-rouge">from</code></td> <td>Build stage or image name for the root of the source. Defaults to the build context.</td> </tr> <tr> <td><code class="highlighter-rouge">rw</code>,<code class="highlighter-rouge">readwrite</code></td> <td>Allow writes on the mount. Written data will be discarded.</td> </tr> </tbody> </table> <ul> <li><code class="highlighter-rouge">RUN --mount=type=cache</code></li> </ul> <p>专用于作为 cache 的挂载位置,一般用于 cache 包管理器的下载等</p> <table> <thead> <tr> <th>Option</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">id</code></td> <td>Optional ID to identify separate/different caches</td> </tr> <tr> <td><code class="highlighter-rouge">target</code> (required)</td> <td>Mount path.</td> </tr> <tr> <td><code class="highlighter-rouge">ro</code>,<code class="highlighter-rouge">readonly</code></td> <td>Read-only if set.</td> </tr> <tr> <td><code class="highlighter-rouge">sharing</code></td> <td>One of <code class="highlighter-rouge">shared</code>, <code class="highlighter-rouge">private</code>, or <code class="highlighter-rouge">locked</code>. Defaults to <code class="highlighter-rouge">shared</code>. A <code class="highlighter-rouge">shared</code> cache mount can be used concurrently by multiple writers. <code class="highlighter-rouge">private</code> creates a new mount if there are multiple writers. <code class="highlighter-rouge">locked</code> pauses the second writer until the first one releases the mount.</td> </tr> <tr> <td><code class="highlighter-rouge">from</code></td> <td>Build stage to use as a base of the cache mount. Defaults to empty directory.</td> </tr> <tr> <td><code class="highlighter-rouge">source</code></td> <td>Subpath in the <code class="highlighter-rouge">from</code> to mount. Defaults to the root of the <code class="highlighter-rouge">from</code>.</td> </tr> </tbody> </table> <p><strong>Example: cache Go packages</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># syntax = docker/dockerfile:experimental</span> FROM golang ... RUN <span class="nt">--mount</span><span class="o">=</span><span class="nb">type</span><span class="o">=</span>cache,target<span class="o">=</span>/root/.cache/go-build go build ... </code></pre></div></div> <p><strong>Example: cache apt packages</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># syntax = docker/dockerfile:experimental</span> FROM ubuntu RUN <span class="nb">rm</span> <span class="nt">-f</span> /etc/apt/apt.conf.d/docker-clean<span class="p">;</span> <span class="nb">echo</span> <span class="s1">'Binary::apt::APT::Keep-Downloaded-Packages "true";'</span> <span class="o">&gt;</span> /etc/apt/apt.conf.d/keep-cache RUN <span class="nt">--mount</span><span class="o">=</span><span class="nb">type</span><span class="o">=</span>cache,target<span class="o">=</span>/var/cache/apt <span class="nt">--mount</span><span class="o">=</span><span class="nb">type</span><span class="o">=</span>cache,target<span class="o">=</span>/var/lib/apt <span class="se">\</span> apt update <span class="o">&amp;&amp;</span> apt <span class="nb">install</span> <span class="nt">-y</span> gcc </code></pre></div></div> <ul> <li><code class="highlighter-rouge">RUN --mount=type=tmpfs</code></li> </ul> <p>专用于挂载 tmpfs 的选项</p> <table> <thead> <tr> <th>Option</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">target</code> (required)</td> <td>Mount path.</td> </tr> </tbody> </table> <ul> <li><code class="highlighter-rouge">RUN --mount=type=secret</code></li> </ul> <p>这个类似 k8s 的 secret,用来挂载一些不想打入镜像,但是构建时想使用的密钥等,例如 docker 的 <code class="highlighter-rouge">config.json</code>,S3 的 <code class="highlighter-rouge">credentials</code></p> <table> <thead> <tr> <th>Option</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">id</code></td> <td>ID of the secret. Defaults to basename of the target path.</td> </tr> <tr> <td><code class="highlighter-rouge">target</code></td> <td>Mount path. Defaults to <code class="highlighter-rouge">/run/secrets/</code> + <code class="highlighter-rouge">id</code>.</td> </tr> <tr> <td><code class="highlighter-rouge">required</code></td> <td>If set to <code class="highlighter-rouge">true</code>, the instruction errors out when the secret is unavailable. Defaults to <code class="highlighter-rouge">false</code>.</td> </tr> <tr> <td><code class="highlighter-rouge">mode</code></td> <td>File mode for secret file in octal. Default 0400.</td> </tr> <tr> <td><code class="highlighter-rouge">uid</code></td> <td>User ID for secret file. Default 0.</td> </tr> <tr> <td><code class="highlighter-rouge">gid</code></td> <td>Group ID for secret file. Default 0.</td> </tr> </tbody> </table> <p><strong>Example: access to S3</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># syntax = docker/dockerfile:experimental</span> FROM python:3 RUN pip <span class="nb">install </span>awscli RUN <span class="nt">--mount</span><span class="o">=</span><span class="nb">type</span><span class="o">=</span>secret,id<span class="o">=</span>aws,target<span class="o">=</span>/root/.aws/credentials aws s3 <span class="nb">cp </span>s3://... ... </code></pre></div></div> <p><strong>注意: <code class="highlighter-rouge">buildctl</code> 是 BuildKit 的命令,你要测试的话自己换成 <code class="highlighter-rouge">docker build</code> 相关参数</strong></p> <div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>buildctl build <span class="nt">--frontend</span><span class="o">=</span>dockerfile.v0 <span class="nt">--local</span> <span class="nv">context</span><span class="o">=</span><span class="nb">.</span> <span class="nt">--local</span> <span class="nv">dockerfile</span><span class="o">=</span><span class="nb">.</span> <span class="se">\</span> <span class="nt">--secret</span> <span class="nb">id</span><span class="o">=</span>aws,src<span class="o">=</span><span class="nv">$HOME</span>/.aws/credentials </code></pre></div></div> <ul> <li><code class="highlighter-rouge">RUN --mount=type=ssh</code></li> </ul> <p>允许 build 容器通过 SSH agent 访问 SSH key,并且支持 <code class="highlighter-rouge">passphrases</code></p> <table> <thead> <tr> <th>Option</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">id</code></td> <td>ID of SSH agent socket or key. Defaults to “default”.</td> </tr> <tr> <td><code class="highlighter-rouge">target</code></td> <td>SSH agent socket path. Defaults to <code class="highlighter-rouge">/run/buildkit/ssh_agent.${N}</code>.</td> </tr> <tr> <td><code class="highlighter-rouge">required</code></td> <td>If set to <code class="highlighter-rouge">true</code>, the instruction errors out when the key is unavailable. Defaults to <code class="highlighter-rouge">false</code>.</td> </tr> <tr> <td><code class="highlighter-rouge">mode</code></td> <td>File mode for socket in octal. Default 0600.</td> </tr> <tr> <td><code class="highlighter-rouge">uid</code></td> <td>User ID for socket. Default 0.</td> </tr> <tr> <td><code class="highlighter-rouge">gid</code></td> <td>Group ID for socket. Default 0.</td> </tr> </tbody> </table> <p><strong>Example: access to Gitlab</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># syntax = docker/dockerfile:experimental</span> FROM alpine RUN apk add <span class="nt">--no-cache</span> openssh-client RUN <span class="nb">mkdir</span> <span class="nt">-p</span> <span class="nt">-m</span> 0700 ~/.ssh <span class="o">&amp;&amp;</span> ssh-keyscan gitlab.com <span class="o">&gt;&gt;</span> ~/.ssh/known_hosts RUN <span class="nt">--mount</span><span class="o">=</span><span class="nb">type</span><span class="o">=</span>ssh ssh <span class="nt">-q</span> <span class="nt">-T</span> git@gitlab.com 2&gt;&amp;1 | <span class="nb">tee</span> /hello <span class="c"># "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here</span> <span class="c"># with the type of build progress is defined as `plain`.</span> </code></pre></div></div> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">eval</span> <span class="si">$(</span>ssh-agent<span class="si">)</span> <span class="nv">$ </span>ssh-add ~/.ssh/id_rsa <span class="o">(</span>Input your passphrase here<span class="o">)</span> <span class="nv">$ </span>buildctl build <span class="nt">--frontend</span><span class="o">=</span>dockerfile.v0 <span class="nt">--local</span> <span class="nv">context</span><span class="o">=</span><span class="nb">.</span> <span class="nt">--local</span> <span class="nv">dockerfile</span><span class="o">=</span><span class="nb">.</span> <span class="se">\</span> <span class="nt">--ssh</span> <span class="nv">default</span><span class="o">=</span><span class="nv">$SSH_AUTH_SOCK</span> </code></pre></div></div> <p>你也可以直接使用宿主机目录的 pem 文件,但是带有密码的 pem 目前不支持</p> <p><strong>目前根据文档测试,当前的挂载类型比如 <code class="highlighter-rouge">cache</code> 类型,仅用于 multi-stage 内的挂载,比如你有 2+ 个构建步骤,<code class="highlighter-rouge">cache</code> 挂载类型能帮你在各个阶段内共享文件;但是它目前无法解决直接将宿主机目录挂载到 multi-stage 的问题(可以采取些曲线救国方案,但是很不优雅);但是未来还是很有展望的,可以关注一下</strong></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Mon, 13 May 2019 22:57:07 +0800 /2019/05/13/dockerfile-extended-syntax/ /2019/05/13/dockerfile-extended-syntax/ Docker Docker Mac 下调校 Rime <blockquote> <p>由于对国内输入法隐私问题的担忧,决定放弃搜狗等输入法;为了更加 Geek 一些,最终决定了折腾 Rime(鼠须管) 输入法,以下为一些折腾的过程</p> </blockquote> <p><strong>国际惯例先放点图压压惊</strong></p> <p><img src="http://cdn.oss.link/markdown/t3otb.jpg" alt="example1" /> <img src="http://cdn.oss.link/markdown/ep8sl.jpg" alt="example2" /> <img src="http://cdn.oss.link/markdown/wth6n.jpg" alt="example3" /> <img src="http://cdn.oss.link/markdown/5b85o.jpg" alt="example4" /></p> <h2 id="一安装">一、安装</h2> <p>安装 Rime 没啥好说的,直接从<a href="http://rime.im">官网</a>下载最新版本的安装包既可;安装完成后配置文件位于 <code class="highlighter-rouge">~/Library/Rime</code> 位置;在进行后续折腾之前我建议还是先 <code class="highlighter-rouge">cp -r ~/Library/Rime ~/Library/Rime.bak</code> 备份一下配置文件,以防制后续折腾挂了还可以还原;安装完成以后按 <code class="highlighter-rouge">⌘ + 反引号(~)</code> 切换到 <code class="highlighter-rouge">朙月拼音-简化字</code> 既可开启简体中文输入</p> <h2 id="二乱码解决">二、乱码解决</h2> <p>安装完成后在打字时可能出现乱码情况(俗称豆腐块),这是由于 Rime 默认 utf-8 字符集比较大,预选词内会出现生僻字,而 mac 字体内又不包含这些字体,从而导致乱码;解决方案很简单,下载 <a href="http://github.com/mritd/rime/tree/master/fonts">花园明朝</a> A、B 两款字体安装既可,安装后重启一下就不会出现乱码了</p> <p><img src="http://cdn.oss.link/markdown/yfbis.png" alt="fonts" /></p> <h2 id="三配置文件">三、配置文件</h2> <p>官方并不建议直接修改原始的配置文件,因为输入法更新时会重新覆盖默认配置,可能导致某些自定义配置丢失;推荐作法是创建一系列的 patch 配置,通过类似打补丁替换这种方式来实现无感的增加自定义配置;</p> <p>由于使用的是 <code class="highlighter-rouge">朙月拼音-简化字</code> 输入方案,所以需要创建 <code class="highlighter-rouge">luna_pinyin_simp.custom.yaml</code> 等配置文件,后面就是查文档 + 各种 Google 一顿魔改了;目前我将我自己用的配置放在了 <a href="http://github.com/mritd/rime">Github</a> 上,有需要的可以直接 clone 下来,用里面的配置文件直接覆盖 <code class="highlighter-rouge">~/Library/Rime</code> 下的文件,然后重新部署既可,关于具体配置细节在下面写</p> <h2 id="四自定义配色">四、自定义配色</h2> <p>皮肤配色配置方案位于 <code class="highlighter-rouge">squirrel.custom.yaml</code> 配置文件中,我的配置目前是参考搜狗输入法皮肤自己调试的;官方也提供了一些皮肤外观配置,详见 <a href="http://gist.github.com/lotem/2290714">Gist</a>;想要切换皮肤配色只需要修改 <code class="highlighter-rouge">style/color_scheme</code> 为相应的皮肤配色名称既可</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">patch</span><span class="pi">:</span> <span class="na">show_notifications_when</span><span class="pi">:</span> <span class="s">appropriate</span> <span class="c1"># 状态通知,适当,也可设为全开(always)全关(never)</span> <span class="s">style/color_scheme</span><span class="pi">:</span> <span class="s">mritd_dark</span> <span class="c1"># 方案命名,不能有空格</span> <span class="na">preset_color_schemes</span><span class="pi">:</span> <span class="na">mritd_dark</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">www.463.com/mritd dark</span> <span class="na">author</span><span class="pi">:</span> <span class="s">mritd &lt;mritd1234@gmail.com&gt;</span> <span class="na">horizontal</span><span class="pi">:</span> <span class="no">true</span> <span class="c1"># 水平排列</span> <span class="na">inline_preedit</span><span class="pi">:</span> <span class="no">true</span> <span class="c1"># 单行显示,false双行显示</span> <span class="na">candidate_format</span><span class="pi">:</span> <span class="s2">"</span><span class="s">%c</span><span class="se">\u2005</span><span class="s">%@"</span> <span class="c1"># 用 1/6 em 空格 U+2005 来控制编号 %c 和候选词 %@ 前后的空间。</span> <span class="na">corner_radius</span><span class="pi">:</span> <span class="m">5</span> <span class="c1"># 候选条圆角</span> <span class="na">hilited_corner_radius</span><span class="pi">:</span> <span class="m">3</span> <span class="c1"># 高亮圆角</span> <span class="na">border_height</span><span class="pi">:</span> <span class="m">6</span> <span class="c1"># 窗口边界高度,大于圆角半径才生效</span> <span class="na">border_width</span><span class="pi">:</span> <span class="m">6</span> <span class="c1"># 窗口边界宽度,大于圆角半径才生效</span> <span class="na">border_color_width</span><span class="pi">:</span> <span class="m">0</span> <span class="c1">#font_face: "PingFangSC" # 候选词字体</span> <span class="na">font_point</span><span class="pi">:</span> <span class="m">16</span> <span class="c1"># 候选字词大小</span> <span class="na">label_font_point</span><span class="pi">:</span> <span class="m">14</span> <span class="c1"># 候选编号大小</span> <span class="na">text_color</span><span class="pi">:</span> <span class="s">0xdedddd</span> <span class="c1"># 拼音行文字颜色,24位色值,16进制,BGR顺序</span> <span class="na">back_color</span><span class="pi">:</span> <span class="s">0x4b4b4b</span> <span class="c1"># 候选条背景色</span> <span class="na">label_color</span><span class="pi">:</span> <span class="s">0x888785</span> <span class="c1"># 预选栏编号颜色</span> <span class="na">border_color</span><span class="pi">:</span> <span class="s">0x4b4b4b</span> <span class="c1"># 边框色</span> <span class="na">candidate_text_color</span><span class="pi">:</span> <span class="s">0xffffff</span> <span class="c1"># 预选项文字颜色</span> <span class="na">hilited_text_color</span><span class="pi">:</span> <span class="s">0xdedddd</span> <span class="c1"># 高亮拼音 (需要开启内嵌编码)</span> <span class="na">hilited_back_color</span><span class="pi">:</span> <span class="s">0x252320</span> <span class="c1"># 高亮拼音 (需要开启内嵌编码)</span> <span class="na">hilited_candidate_text_color</span><span class="pi">:</span> <span class="s">0xFFE696</span> <span class="c1"># 第一候选项文字颜色</span> <span class="na">hilited_candidate_back_color</span><span class="pi">:</span> <span class="s">0x4b4b4b</span> <span class="c1"># 第一候选项背景背景色</span> <span class="na">hilited_candidate_label_color</span><span class="pi">:</span> <span class="s">0xffffff</span> <span class="c1"># 第一候选项编号颜色</span> <span class="na">comment_text_color</span><span class="pi">:</span> <span class="s">0xdedddd</span> <span class="c1"># 拼音等提示文字颜色</span> </code></pre></div></div> <h2 id="五增加自定义快捷字符">五、增加自定义快捷字符</h2> <p>快捷字符例如在中文输入法状态下可以直接输入 <code class="highlighter-rouge">/dn</code> 来调出特殊符号输入;这些配置位于 <code class="highlighter-rouge">luna_pinyin_simp.custom.yaml</code> 的 <code class="highlighter-rouge">punctuator</code> 配置中,我目前自行定义了一些,有需要的可以依葫芦画瓢直接修改</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">punctuator</span><span class="pi">:</span> <span class="na">import_preset</span><span class="pi">:</span> <span class="s">symbols</span> <span class="na">symbols</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/fs"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">½</span><span class="pi">,</span><span class="nv">‰</span><span class="pi">,</span><span class="nv">¼</span><span class="pi">,</span><span class="nv">⅓</span><span class="pi">,</span><span class="nv">⅔</span><span class="pi">,</span><span class="nv">¾</span><span class="pi">,</span><span class="nv">⅒</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/dq"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">🌍</span><span class="pi">,</span><span class="nv">🌎</span><span class="pi">,</span><span class="nv">🌏</span><span class="pi">,</span><span class="nv">🌐</span><span class="pi">,</span><span class="nv">🌑</span><span class="pi">,</span><span class="nv">🌒</span><span class="pi">,</span><span class="nv">🌓</span><span class="pi">,</span><span class="nv">🌔</span><span class="pi">,</span><span class="nv">🌕</span><span class="pi">,</span><span class="nv">🌖</span><span class="pi">,</span><span class="nv">🌗</span><span class="pi">,</span><span class="nv">🌘</span><span class="pi">,</span><span class="nv">🌙</span><span class="pi">,</span><span class="nv">🌚</span><span class="pi">,</span><span class="nv">🌛</span><span class="pi">,</span><span class="nv">🌜</span><span class="pi">,</span><span class="nv">🌝</span><span class="pi">,</span><span class="nv">🌞</span><span class="pi">,</span><span class="nv">⭐</span><span class="pi">,</span><span class="nv">🌟</span><span class="pi">,</span><span class="nv">🌠</span><span class="pi">,</span><span class="nv">⛅</span><span class="pi">,</span><span class="nv">⚡</span><span class="pi">,</span><span class="nv">❄</span><span class="pi">,</span><span class="nv">🔥</span><span class="pi">,</span><span class="nv">💧</span><span class="pi">,</span><span class="nv">🌊</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/jt"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">⬆</span><span class="pi">,</span><span class="nv">↗</span><span class="pi">,</span><span class="nv">➡</span><span class="pi">,</span><span class="nv">↘</span><span class="pi">,</span><span class="nv">⬇</span><span class="pi">,</span><span class="nv">↙</span><span class="pi">,</span><span class="nv">⬅</span><span class="pi">,</span><span class="nv">↖</span><span class="pi">,</span><span class="nv">↕</span><span class="pi">,</span><span class="nv">↔</span><span class="pi">,</span><span class="nv">↩</span><span class="pi">,</span><span class="nv">↪</span><span class="pi">,</span><span class="nv">⤴</span><span class="pi">,</span><span class="nv">⤵</span><span class="pi">,</span><span class="nv">🔃</span><span class="pi">,</span><span class="nv">🔄</span><span class="pi">,</span><span class="nv">🔙</span><span class="pi">,</span><span class="nv">🔚</span><span class="pi">,</span><span class="nv">🔛</span><span class="pi">,</span><span class="nv">🔜</span><span class="pi">,</span><span class="nv">🔝</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/sg"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">🍇</span><span class="pi">,</span><span class="nv">🍈</span><span class="pi">,</span><span class="nv">🍉</span><span class="pi">,</span><span class="nv">🍊</span><span class="pi">,</span><span class="nv">🍋</span><span class="pi">,</span><span class="nv">🍌</span><span class="pi">,</span><span class="nv">🍍</span><span class="pi">,</span><span class="nv">🍎</span><span class="pi">,</span><span class="nv">🍏</span><span class="pi">,</span><span class="nv">🍐</span><span class="pi">,</span><span class="nv">🍑</span><span class="pi">,</span><span class="nv">🍒</span><span class="pi">,</span><span class="nv">🍓</span><span class="pi">,</span><span class="nv">🍅</span><span class="pi">,</span><span class="nv">🍆</span><span class="pi">,</span><span class="nv">🌽</span><span class="pi">,</span><span class="nv">🍄</span><span class="pi">,</span><span class="nv">🌰</span><span class="pi">,</span><span class="nv">🍞</span><span class="pi">,</span><span class="nv">🍖</span><span class="pi">,</span><span class="nv">🍗</span><span class="pi">,</span><span class="nv">🍔</span><span class="pi">,</span><span class="nv">🍟</span><span class="pi">,</span><span class="nv">🍕</span><span class="pi">,</span><span class="nv">🍳</span><span class="pi">,</span><span class="nv">🍲</span><span class="pi">,</span><span class="nv">🍱</span><span class="pi">,</span><span class="nv">🍘</span><span class="pi">,</span><span class="nv">🍙</span><span class="pi">,</span><span class="nv">🍚</span><span class="pi">,</span><span class="nv">🍛</span><span class="pi">,</span><span class="nv">🍜</span><span class="pi">,</span><span class="nv">🍝</span><span class="pi">,</span><span class="nv">🍠</span><span class="pi">,</span><span class="nv">🍢</span><span class="pi">,</span><span class="nv">🍣</span><span class="pi">,</span><span class="nv">🍤</span><span class="pi">,</span><span class="nv">🍥</span><span class="pi">,</span><span class="nv">🍡</span><span class="pi">,</span><span class="nv">🍦</span><span class="pi">,</span><span class="nv">🍧</span><span class="pi">,</span><span class="nv">🍨</span><span class="pi">,</span><span class="nv">🍩</span><span class="pi">,</span><span class="nv">🍪</span><span class="pi">,</span><span class="nv">🎂</span><span class="pi">,</span><span class="nv">🍰</span><span class="pi">,</span><span class="nv">🍫</span><span class="pi">,</span><span class="nv">🍬</span><span class="pi">,</span><span class="nv">🍭</span><span class="pi">,</span><span class="nv">🍮</span><span class="pi">,</span><span class="nv">🍯</span><span class="pi">,</span><span class="nv">🍼</span><span class="pi">,</span><span class="nv">🍵</span><span class="pi">,</span><span class="nv">🍶</span><span class="pi">,</span><span class="nv">🍷</span><span class="pi">,</span><span class="nv">🍸</span><span class="pi">,</span><span class="nv">🍹</span><span class="pi">,</span><span class="nv">🍺</span><span class="pi">,</span><span class="nv">🍻</span><span class="pi">,</span><span class="nv">🍴</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/dw"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">🙈</span><span class="pi">,</span><span class="nv">🙉</span><span class="pi">,</span><span class="nv">🙊</span><span class="pi">,</span><span class="nv">🐵</span><span class="pi">,</span><span class="nv">🐒</span><span class="pi">,</span><span class="nv">🐶</span><span class="pi">,</span><span class="nv">🐕</span><span class="pi">,</span><span class="nv">🐩</span><span class="pi">,</span><span class="nv">🐺</span><span class="pi">,</span><span class="nv">🐱</span><span class="pi">,</span><span class="nv">😺</span><span class="pi">,</span><span class="nv">😸</span><span class="pi">,</span><span class="nv">😹</span><span class="pi">,</span><span class="nv">😻</span><span class="pi">,</span><span class="nv">😼</span><span class="pi">,</span><span class="nv">😽</span><span class="pi">,</span><span class="nv">🙀</span><span class="pi">,</span><span class="nv">😿</span><span class="pi">,</span><span class="nv">😾</span><span class="pi">,</span><span class="nv">🐈</span><span class="pi">,</span><span class="nv">🐯</span><span class="pi">,</span><span class="nv">🐅</span><span class="pi">,</span><span class="nv">🐆</span><span class="pi">,</span><span class="nv">🐴</span><span class="pi">,</span><span class="nv">🐎</span><span class="pi">,</span><span class="nv">🐮</span><span class="pi">,</span><span class="nv">🐂</span><span class="pi">,</span><span class="nv">🐃</span><span class="pi">,</span><span class="nv">🐄</span><span class="pi">,</span><span class="nv">🐷</span><span class="pi">,</span><span class="nv">🐖</span><span class="pi">,</span><span class="nv">🐗</span><span class="pi">,</span><span class="nv">🐽</span><span class="pi">,</span><span class="nv">🐏</span><span class="pi">,</span><span class="nv">🐑</span><span class="pi">,</span><span class="nv">🐐</span><span class="pi">,</span><span class="nv">🐪</span><span class="pi">,</span><span class="nv">🐫</span><span class="pi">,</span><span class="nv">🐘</span><span class="pi">,</span><span class="nv">🐭</span><span class="pi">,</span><span class="nv">🐁</span><span class="pi">,</span><span class="nv">🐀</span><span class="pi">,</span><span class="nv">🐹</span><span class="pi">,</span><span class="nv">🐰</span><span class="pi">,</span><span class="nv">🐇</span><span class="pi">,</span><span class="nv">🐻</span><span class="pi">,</span><span class="nv">🐨</span><span class="pi">,</span><span class="nv">🐼</span><span class="pi">,</span><span class="nv">🐾</span><span class="pi">,</span><span class="nv">🐔</span><span class="pi">,</span><span class="nv">🐓</span><span class="pi">,</span><span class="nv">🐣</span><span class="pi">,</span><span class="nv">🐤</span><span class="pi">,</span><span class="nv">🐥</span><span class="pi">,</span><span class="nv">🐦</span><span class="pi">,</span><span class="nv">🐧</span><span class="pi">,</span><span class="nv">🐸</span><span class="pi">,</span><span class="nv">🐊</span><span class="pi">,</span><span class="nv">🐢</span><span class="pi">,</span><span class="nv">🐍</span><span class="pi">,</span><span class="nv">🐲</span><span class="pi">,</span><span class="nv">🐉</span><span class="pi">,</span><span class="nv">🐳</span><span class="pi">,</span><span class="nv">🐋</span><span class="pi">,</span><span class="nv">🐬</span><span class="pi">,</span><span class="nv">🐟</span><span class="pi">,</span><span class="nv">🐠</span><span class="pi">,</span><span class="nv">🐡</span><span class="pi">,</span><span class="nv">🐙</span><span class="pi">,</span><span class="nv">🐚</span><span class="pi">,</span><span class="nv">🐌</span><span class="pi">,</span><span class="nv">🐛</span><span class="pi">,</span><span class="nv">🐜</span><span class="pi">,</span><span class="nv">🐝</span><span class="pi">,</span><span class="nv">🐞</span><span class="pi">,</span><span class="nv">🦋</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/bq"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">😀</span><span class="pi">,</span><span class="nv">😁</span><span class="pi">,</span><span class="nv">😂</span><span class="pi">,</span><span class="nv">😃</span><span class="pi">,</span><span class="nv">😄</span><span class="pi">,</span><span class="nv">😅</span><span class="pi">,</span><span class="nv">😆</span><span class="pi">,</span><span class="nv">😉</span><span class="pi">,</span><span class="nv">😊</span><span class="pi">,</span><span class="nv">😋</span><span class="pi">,</span><span class="nv">😎</span><span class="pi">,</span><span class="nv">😍</span><span class="pi">,</span><span class="nv">😘</span><span class="pi">,</span><span class="nv">😗</span><span class="pi">,</span><span class="nv">😙</span><span class="pi">,</span><span class="nv">😚</span><span class="pi">,</span><span class="nv">😇</span><span class="pi">,</span><span class="nv">😐</span><span class="pi">,</span><span class="nv">😑</span><span class="pi">,</span><span class="nv">😶</span><span class="pi">,</span><span class="nv">😏</span><span class="pi">,</span><span class="nv">😣</span><span class="pi">,</span><span class="nv">😥</span><span class="pi">,</span><span class="nv">😮</span><span class="pi">,</span><span class="nv">😯</span><span class="pi">,</span><span class="nv">😪</span><span class="pi">,</span><span class="nv">😫</span><span class="pi">,</span><span class="nv">😴</span><span class="pi">,</span><span class="nv">😌</span><span class="pi">,</span><span class="nv">😛</span><span class="pi">,</span><span class="nv">😜</span><span class="pi">,</span><span class="nv">😝</span><span class="pi">,</span><span class="nv">😒</span><span class="pi">,</span><span class="nv">😓</span><span class="pi">,</span><span class="nv">😔</span><span class="pi">,</span><span class="nv">😕</span><span class="pi">,</span><span class="nv">😲</span><span class="pi">,</span><span class="nv">😷</span><span class="pi">,</span><span class="nv">😖</span><span class="pi">,</span><span class="nv">😞</span><span class="pi">,</span><span class="nv">😟</span><span class="pi">,</span><span class="nv">😤</span><span class="pi">,</span><span class="nv">😢</span><span class="pi">,</span><span class="nv">😭</span><span class="pi">,</span><span class="nv">😦</span><span class="pi">,</span><span class="nv">😧</span><span class="pi">,</span><span class="nv">😨</span><span class="pi">,</span><span class="nv">😬</span><span class="pi">,</span><span class="nv">😰</span><span class="pi">,</span><span class="nv">😱</span><span class="pi">,</span><span class="nv">😳</span><span class="pi">,</span><span class="nv">😵</span><span class="pi">,</span><span class="nv">😡</span><span class="pi">,</span><span class="nv">😠</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/ss"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">💪</span><span class="pi">,</span><span class="nv">👈</span><span class="pi">,</span><span class="nv">👉</span><span class="pi">,</span><span class="nv">👆</span><span class="pi">,</span><span class="nv">👇</span><span class="pi">,</span><span class="nv">✋</span><span class="pi">,</span><span class="nv">👌</span><span class="pi">,</span><span class="nv">👍</span><span class="pi">,</span><span class="nv">👎</span><span class="pi">,</span><span class="nv">✊</span><span class="pi">,</span><span class="nv">👊</span><span class="pi">,</span><span class="nv">👋</span><span class="pi">,</span><span class="nv">👏</span><span class="pi">,</span><span class="nv">👐</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/dn"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">⌘</span><span class="pi">,</span> <span class="nv">⌥</span><span class="pi">,</span> <span class="nv">⇧</span><span class="pi">,</span> <span class="nv">⌃</span><span class="pi">,</span> <span class="nv">⎋</span><span class="pi">,</span> <span class="nv">⇪</span><span class="pi">,</span> <span class="nv"></span><span class="pi">,</span> <span class="nv">⌫</span><span class="pi">,</span> <span class="nv">⌦</span><span class="pi">,</span> <span class="nv">↩︎</span><span class="pi">,</span> <span class="nv">⏎</span><span class="pi">,</span> <span class="nv">↑</span><span class="pi">,</span> <span class="nv">↓</span><span class="pi">,</span> <span class="nv">←</span><span class="pi">,</span> <span class="nv">→</span><span class="pi">,</span> <span class="nv">↖</span><span class="pi">,</span> <span class="nv">↘</span><span class="pi">,</span> <span class="nv">⇟</span><span class="pi">,</span> <span class="nv">⇞</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/fh"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">©</span><span class="pi">,</span><span class="nv">®</span><span class="pi">,</span><span class="nv">℗</span><span class="pi">,</span><span class="nv">ⓘ</span><span class="pi">,</span><span class="nv">℠</span><span class="pi">,</span><span class="nv">™</span><span class="pi">,</span><span class="nv">℡</span><span class="pi">,</span><span class="nv">␡</span><span class="pi">,</span><span class="nv">♂</span><span class="pi">,</span><span class="nv">♀</span><span class="pi">,</span><span class="nv">☉</span><span class="pi">,</span><span class="nv">☊</span><span class="pi">,</span><span class="nv">☋</span><span class="pi">,</span><span class="nv">☌</span><span class="pi">,</span><span class="nv">☍</span><span class="pi">,</span><span class="nv">☑︎</span><span class="pi">,</span><span class="nv">☒</span><span class="pi">,</span><span class="nv">☜</span><span class="pi">,</span><span class="nv">☝</span><span class="pi">,</span><span class="nv">☞</span><span class="pi">,</span><span class="nv">☟</span><span class="pi">,</span><span class="nv">✎</span><span class="pi">,</span><span class="nv">✄</span><span class="pi">,</span><span class="nv">♻</span><span class="pi">,</span><span class="nv">⚐</span><span class="pi">,</span><span class="nv">⚑</span><span class="pi">,</span><span class="nv">⚠</span><span class="pi">]</span> <span class="s2">"</span><span class="s">/xh"</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">*</span><span class="pi">,</span><span class="nv">×</span><span class="pi">,</span><span class="nv">✱</span><span class="pi">,</span><span class="nv">★</span><span class="pi">,</span><span class="nv">☆</span><span class="pi">,</span><span class="nv">✩</span><span class="pi">,</span><span class="nv">✧</span><span class="pi">,</span><span class="nv">❋</span><span class="pi">,</span><span class="nv">❊</span><span class="pi">,</span><span class="nv">❉</span><span class="pi">,</span><span class="nv">❈</span><span class="pi">,</span><span class="nv">❅</span><span class="pi">,</span><span class="nv">✿</span><span class="pi">,</span><span class="nv">✲</span><span class="pi">]</span> </code></pre></div></div> <h2 id="六设置输入方案">六、设置输入方案</h2> <p>在第一次按 <code class="highlighter-rouge">⌘ + 反引号(~)</code> 设置输入法时实际上我们可以看到很多的输入方案,而事实上很多方案我们根本用不上;想要删除和修改方案可以调整 <code class="highlighter-rouge">default.custom.yaml</code> 中的 <code class="highlighter-rouge">schema_list</code> 字段</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">patch</span><span class="pi">:</span> <span class="na">menu</span><span class="pi">:</span> <span class="na">page_size</span><span class="pi">:</span> <span class="m">8</span> <span class="na">schema_list</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">schema</span><span class="pi">:</span> <span class="s">luna_pinyin_simp</span> <span class="c1"># 朙月拼音 简化字</span> <span class="pi">-</span> <span class="na">schema</span><span class="pi">:</span> <span class="s">luna_pinyin</span> <span class="c1"># 朙月拼音</span> <span class="pi">-</span> <span class="na">schema</span><span class="pi">:</span> <span class="s">luna_pinyin_fluency</span> <span class="c1"># 语句流</span> <span class="c1"># - schema: double_pinyin # 自然碼雙拼</span> <span class="c1"># - schema: double_pinyin_flypy # 小鹤雙拼</span> <span class="c1"># - schema: double_pinyin_pyjj # 拼音加加双拼</span> <span class="c1"># - schema: wubi_pinyin # 五笔拼音混合輸入</span> </code></pre></div></div> <p><strong>实际上我只能用上第一个…毕竟写了好几年代码还得看键盘的人也只能这样了…</strong></p> <h2 id="七调整特殊键行为">七、调整特殊键行为</h2> <p>在刚安装完以后发现在中文输入法状态下输入英文,按 <code class="highlighter-rouge">shift</code> 键后字符上屏,然后还得回车一下,这就很让我难受…最后找到了这篇 <a href="http://gist.github.com/lotem/2981316">Gist</a>,目前将大写锁定、<code class="highlighter-rouge">shift</code> 键调整为了跟搜狗一致的配置,有需要调整的可以自行编辑 <code class="highlighter-rouge">default.custom.yaml</code> 中的 <code class="highlighter-rouge">ascii_composer/switch_key</code> 部分</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># capslock 键切换英文并输出大写</span> <span class="s">ascii_composer/good_old_caps_lock</span><span class="pi">:</span> <span class="no">true</span> <span class="c1"># 输入法中英文状态快捷键</span> <span class="s">ascii_composer/switch_key</span><span class="pi">:</span> <span class="na">Caps_Lock</span><span class="pi">:</span> <span class="s">commit_code</span> <span class="na">Control_L</span><span class="pi">:</span> <span class="s">noop</span> <span class="na">Control_R</span><span class="pi">:</span> <span class="s">noop</span> <span class="c1"># 按下左 shift 英文字符直接上屏,不需要再次回车,输入法保持英文状态</span> <span class="na">Shift_L</span><span class="pi">:</span> <span class="s">commit_code</span> <span class="na">Shift_R</span><span class="pi">:</span> <span class="s">noop</span> </code></pre></div></div> <h2 id="八自定义词库">八、自定义词库</h2> <p>Rime 默认的词库稍为有点弱,我们可以下载一些搜狗词库来进行扩展;不过搜狗词库格式默认是无法解析的,好在有人开发了工具可以方便的将搜狗细胞词库转化为 Rime 的格式(工具<a href="http://github.com/studyzy/imewlconverter/releases">点击这里</a>下载);目前该工具只支持 Windows(也有些别人写的 py 脚本啥的,但是我没用),所以词库转换这种操作还得需要一个 Windows 虚拟机;</p> <p>转换过程很简单,先从<a href="http://pinyin.sogou.com/dict/">搜狗词库</a>下载一系列的 <code class="highlighter-rouge">scel</code> 文件,然后批量选中,接着调整一下输入和输出格式点击转换,最后保存成一个 <code class="highlighter-rouge">txt</code> 文本</p> <p><img src="http://cdn.oss.link/markdown/jtv97.png" alt="input-setting" /></p> <p><img src="http://cdn.oss.link/markdown/p7qha.png" alt="convert" /></p> <p>光有这个文本还不够,我们要将它塞到词库的 <code class="highlighter-rouge">yaml</code> 配置里,所以新建一个词库配置文件 <code class="highlighter-rouge">luna_pinyin.sougou.dict.yaml</code>,然后写上头部说明(<strong>注意最后三个点后面加一个换行</strong>)</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Rime dictionary</span> <span class="c1"># encoding: utf-8</span> <span class="c1"># 搜狗词库 目前包含如下:</span> <span class="c1"># IT计算机 实用IT词汇 亲戚称呼 化学品名 数字时间 数学词汇 淘宝词库 编程语言 软件专业 颜色名称 程序猿词库 开发专用词库 搜狗标准词库</span> <span class="c1"># 摄影专业名词 计算机专业词库 计算机词汇大全 保险词汇 最详细的全国地名大全 饮食大全 常见花卉名称 房地产词汇大全 中国传统节日大全 财经金融词汇大全</span> <span class="nn">---</span> <span class="na">name</span><span class="pi">:</span> <span class="s">luna_pinyin.sougou</span> <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1.0"</span> <span class="na">sort</span><span class="pi">:</span> <span class="s">by_weight</span> <span class="na">use_preset_vocabulary</span><span class="pi">:</span> <span class="no">true</span> <span class="nn">...</span> </code></pre></div></div> <p>接着只需要把生成好的词库 <code class="highlighter-rouge">txt</code> 文件内容粘贴到三个点下面既可;但是词库太多的话你会发现这个文本有好几十 M,一般编辑器打开都会卡死,解决这种情况只需要用命令行 <code class="highlighter-rouge">cat</code> 一下就行</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>sougou.txt <span class="o">&gt;&gt;</span> luna_pinyin.sougou.dict.yaml </code></pre></div></div> <p>最后修改 <code class="highlighter-rouge">luna_pinyin.extended.dict.yaml</code> 中的 <code class="highlighter-rouge">import_tables</code> 字段,加入刚刚新建的词库既可</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">name</span><span class="pi">:</span> <span class="s">luna_pinyin.extended</span> <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">2016.06.26"</span> <span class="na">sort</span><span class="pi">:</span> <span class="s">by_weight</span> <span class="c1">#字典初始排序,可選original或by_weight</span> <span class="na">use_preset_vocabulary</span><span class="pi">:</span> <span class="no">true</span> <span class="c1">#此處爲明月拼音擴充詞庫(基本)默認鏈接載入的詞庫,有朙月拼音官方詞庫、明月拼音擴充詞庫(漢語大詞典)、明月拼音擴充詞庫(詩詞)、明月拼音擴充詞庫(含西文的詞彙)。如果不需要加載某个詞庫請將其用「#」註釋掉。</span> <span class="c1">#雙拼不支持 luna_pinyin.cn_en 詞庫,請用戶手動禁用。</span> <span class="na">import_tables</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">luna_pinyin</span> <span class="c1"># 加入搜狗词库</span> <span class="pi">-</span> <span class="s">luna_pinyin.sougou</span> <span class="pi">-</span> <span class="s">luna_pinyin.poetry</span> <span class="pi">-</span> <span class="s">luna_pinyin.cn_en</span> <span class="pi">-</span> <span class="s">luna_pinyin.kaomoji</span> </code></pre></div></div> <h2 id="九定制特殊单词">九、定制特殊单词</h2> <p>由于长期撸码,24 小时离不开命令行,偶尔在中文输入法下输入了一些命令导致汉字直接出现在 terminal 上就很尴尬…这时候我们可以在 <code class="highlighter-rouge">luna_pinyin.cn_en.dict.yaml</code> 加入一些我们自己的专属词库,比如这样</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="na">name</span><span class="pi">:</span> <span class="s">luna_pinyin.cn_en</span> <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">2017.9.13"</span> <span class="na">sort</span><span class="pi">:</span> <span class="s">by_weight</span> <span class="na">use_preset_vocabulary</span><span class="pi">:</span> <span class="no">true</span> <span class="nn">...</span> <span class="s">git</span><span class="err"> </span><span class="s">git</span> <span class="s">ls</span><span class="err"> </span><span class="s">ls</span> <span class="s">cd</span><span class="err"> </span><span class="s">cd</span> <span class="s">pwd</span><span class="err"> </span><span class="s">pwd</span> <span class="s">git ps</span><span class="err"> </span><span class="s">gitps</span> <span class="s">kubernetes</span><span class="err"> </span><span class="s">kubernetes</span> <span class="s">kubernetes</span><span class="err"> </span><span class="s">kuber</span> <span class="s">kubectl</span><span class="err"> </span><span class="s">kubectl</span> <span class="s">kubectl</span><span class="err"> </span><span class="s">kubec</span> <span class="s">docker</span><span class="err"> </span><span class="s">docker</span> <span class="s">docker</span><span class="err"> </span><span class="s">dock</span> <span class="s">ipvs</span><span class="err"> </span><span class="s">ipvs</span> <span class="s">ps</span><span class="err"> </span><span class="s">ps</span> <span class="s">bash</span><span class="err"> </span><span class="s">bash</span> <span class="s">source</span><span class="err"> </span><span class="s">source</span> <span class="s">source</span><span class="err"> </span><span class="s">sou</span> <span class="s">rm</span><span class="err"> </span><span class="s">rm</span> </code></pre></div></div> <p>配置后如果我在中文输入法下输入 git 则会自动匹配 git 这个单词,避免错误的键入中文字符;<strong>需要注意的是第一列代表上屏的字符,第二列代表输入的单词,即 “当输入第二列时候选词为第一列”;两列之间要用 tag 制表符隔开,记住不是空格</strong></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Sat, 23 Mar 2019 21:03:43 +0800 /2019/03/23/oh-my-rime/ /2019/03/23/oh-my-rime/ Mac Mac Ubuntu 设置多个源 <h2 id="一源起">一、源起</h2> <p>使用 Ubuntu 作为生产容器系统好久了,但是 apt 源问题一致有点困扰: <strong>由于众所周知的原因,官方源执行 <code class="highlighter-rouge">apt update</code> 等命令会非常慢;而国内有很多镜像服务,但是某些偶尔也会抽风(比如清华大源),最后的结果就是日常修改 apt 源…</strong>Google 查了了好久发现事实上 apt 源是支持 <code class="highlighter-rouge">mirror</code> 协议的,从而自动选择可用的一个</p> <h2 id="二使用-mirror-协议">二、使用 mirror 协议</h2> <p>废话不说多直接上代码,编辑 <code class="highlighter-rouge">/etc/apt/sources.list</code>,替换为如下内容</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#------------------------------------------------------------------------------#</span> <span class="c"># OFFICIAL UBUNTU REPOS #</span> <span class="c">#------------------------------------------------------------------------------#</span> <span class="c">###### Ubuntu Main Repos</span> deb mirror://mirrors.ubuntu.com/mirrors.txt bionic main restricted universe multiverse deb-src mirror://mirrors.ubuntu.com/mirrors.txt bionic main restricted universe multiverse <span class="c">###### Ubuntu Update Repos</span> deb mirror://mirrors.ubuntu.com/mirrors.txt bionic-security main restricted universe multiverse deb mirror://mirrors.ubuntu.com/mirrors.txt bionic-updates main restricted universe multiverse deb mirror://mirrors.ubuntu.com/mirrors.txt bionic-backports main restricted universe multiverse deb-src mirror://mirrors.ubuntu.com/mirrors.txt bionic-security main restricted universe multiverse deb-src mirror://mirrors.ubuntu.com/mirrors.txt bionic-updates main restricted universe multiverse deb-src mirror://mirrors.ubuntu.com/mirrors.txt bionic-backports main restricted universe multiverse </code></pre></div></div> <p>当使用 <code class="highlighter-rouge">mirror</code> 协议后,执行 <code class="highlighter-rouge">apt update</code> 时会首先<strong>通过 http 访问</strong> <code class="highlighter-rouge">mirrors.ubuntu.com/mirrors.txt</code> 文本;文本内容实际上就是当前可用的镜像源列表,如下所示</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://ftp.sjtu.edu.cn/ubuntu/ http://mirrors.nju.edu.cn/ubuntu/ http://mirrors.nwafu.edu.cn/ubuntu/ http://mirrors.sohu.com/ubuntu/ http://mirrors.aliyun.com/ubuntu/ http://mirrors.shu.edu.cn/ubuntu/ http://mirrors.cqu.edu.cn/ubuntu/ http://mirrors.huaweicloud.com/repository/ubuntu/ http://mirrors.cn99.com/ubuntu/ http://mirrors.yun-idc.com/ubuntu/ http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ http://mirrors.ustc.edu.cn/ubuntu/ http://mirrors.njupt.edu.cn/ubuntu/ http://mirror.lzu.edu.cn/ubuntu/ http://archive.ubuntu.com/ubuntu/ </code></pre></div></div> <p>得到列表后 apt 会自动选择一个(选择规则暂不清楚,国外有文章说是选择最快的,但是不清楚这个最快是延迟还是网速)进行下载;<strong>同时根据地区不通,官方也提供指定国家的 <code class="highlighter-rouge">mirror.txt</code></strong>,比如中国的实际上可以设置为 <code class="highlighter-rouge">mirrors.ubuntu.com/CN.txt</code>(我测试跟官方一样,推测可能是使用了类似 DNS 选优的策略)</p> <h2 id="三自定义-mirror-地址">三、自定义 mirror 地址</h2> <p>现在已经解决了能同时使用多个源的问题,但是有些时候你会发现源的可用性检测并不是很精准,比如某个源只有 40k 的下载速度…不巧你某个下载还命中了,这就很尴尬;<strong>所以有时候我们可能需要自定义 <code class="highlighter-rouge">mirror.txt</code> 这个源列表</strong>,经过测试证明<strong>只需要开启一个标准的 <code class="highlighter-rouge">http server</code> 能返回一个文本即可,不过需要注意只能是 <code class="highlighter-rouge">http</code>,而不是 <code class="highlighter-rouge">http</code></strong>;所以我们首先下载一下这个文本,把不想要的删掉;然后弄个 nginx,甚至 <code class="highlighter-rouge">python -m http.server</code> 把文本文件暴露出去就可以;我比较懒…扔 CDN 上了: http://oss.link/config/apt-mirrors.txt</p> <p>关于源的精简,我建议将一些 <code class="highlighter-rouge">edu</code> 的删掉,因为敏感时期他们很不稳定;优选阿里云、网易、华为这种大公司的,比较有名的清华大的什么的可以留着,其他的可以考虑都删掉</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 19 Mar 2019 21:43:23 +0800 /2019/03/19/how-to-set-multiple-apt-mirrors-for-ubuntu/ /2019/03/19/how-to-set-multiple-apt-mirrors-for-ubuntu/ Linux Linux Kubernetes 1.13.4 搭建 <blockquote> <p>年后回来有点懒,也有点忙;1.13 出来好久了,周末还是决定折腾一下吧</p> </blockquote> <h2 id="一环境准备">一、环境准备</h2> <p>老样子,安装环境为 5 台 Ubuntu 18.04.2 LTS 虚拟机,其他详细信息如下</p> <table> <thead> <tr> <th>System OS</th> <th>IP Address</th> <th>Docker</th> <th>Kernel</th> <th>Application</th> </tr> </thead> <tbody> <tr> <td>Ubuntu 18.04.2 LTS</td> <td>192.168.1.51</td> <td>18.09.2</td> <td>4.15.0-46-generic</td> <td>k8s-master、etcd</td> </tr> <tr> <td>Ubuntu 18.04.2 LTS</td> <td>192.168.1.52</td> <td>18.09.2</td> <td>4.15.0-46-generic</td> <td>k8s-master、etcd</td> </tr> <tr> <td>Ubuntu 18.04.2 LTS</td> <td>192.168.1.53</td> <td>18.09.2</td> <td>4.15.0-46-generic</td> <td>k8s-master、etcd</td> </tr> <tr> <td>Ubuntu 18.04.2 LTS</td> <td>192.168.1.54</td> <td>18.09.2</td> <td>4.15.0-46-generic</td> <td>k8s-node</td> </tr> <tr> <td>Ubuntu 18.04.2 LTS</td> <td>192.168.1.55</td> <td>18.09.2</td> <td>4.15.0-46-generic</td> <td>k8s-node</td> </tr> </tbody> </table> <p><strong>所有配置生成将在第一个节点上完成,第一个节点与其他节点 root 用户免密码登录,用于分发文件;为了方便搭建弄了一点小脚本,仓库地址 <a href="http://github.com/mritd/ktool">ktool</a>,本文后续所有脚本、配置都可以在此仓库找到;关于 <a href="http://github.com/cloudflare/cfssl">cfssl</a> 等基本工具使用,本文不再阐述</strong></p> <h2 id="二安装-etcd">二、安装 Etcd</h2> <h3 id="21生成证书">2.1、生成证书</h3> <p>Etcd 仍然开启 TLS 认证,所以先使用 cfssl 生成相关证书</p> <ul> <li>etcd-root-ca-csr.json</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd-root-ca"</span><span class="p">,</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd Security"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Beijing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Beijing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"ca"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"expiry"</span><span class="p">:</span><span class="w"> </span><span class="s2">"87600h"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>etcd-gencert.json</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"signing"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"usages"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"signing"</span><span class="p">,</span><span class="w"> </span><span class="s2">"key encipherment"</span><span class="p">,</span><span class="w"> </span><span class="s2">"server auth"</span><span class="p">,</span><span class="w"> </span><span class="s2">"client auth"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"expiry"</span><span class="p">:</span><span class="w"> </span><span class="s2">"87600h"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>etcd-csr.json</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd Security"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Beijing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Beijing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"etcd"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"127.0.0.1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w"> </span><span class="s2">"192.168.1.51"</span><span class="p">,</span><span class="w"> </span><span class="s2">"192.168.1.52"</span><span class="p">,</span><span class="w"> </span><span class="s2">"192.168.1.53"</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>接下来执行生成即可;<strong>我建议在生产环境在证书内预留几个 IP,已防止意外故障迁移时还需要重新生成证书;证书默认期限为 10 年(包括 CA 证书),有需要加强安全性的可以适当减小</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cfssl gencert <span class="nt">--initca</span><span class="o">=</span><span class="nb">true </span>etcd-root-ca-csr.json | cfssljson <span class="nt">--bare</span> etcd-root-ca cfssl gencert <span class="nt">--ca</span> etcd-root-ca.pem <span class="nt">--ca-key</span> etcd-root-ca-key.pem <span class="nt">--config</span> etcd-gencert.json etcd-csr.json | cfssljson <span class="nt">--bare</span> etcd </code></pre></div></div> <h3 id="22安装-etcd">2.2、安装 Etcd</h3> <h4 id="221安装脚本">2.2.1、安装脚本</h4> <p>安装 Etcd 只需要将二进制文件放在可执行目录下,然后修改配置增加 systemd service 配置文件即可;为了安全性起见最好使用单独的用户启动 Etcd</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span> <span class="nb">set</span> <span class="nt">-e</span> <span class="nv">ETCD_DEFAULT_VERSION</span><span class="o">=</span><span class="s2">"3.3.12"</span> <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">""</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nv">ETCD_VERSION</span><span class="o">=</span><span class="nv">$1</span> <span class="k">else </span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[33mWARNING: ETCD_VERSION is blank,use default version: </span><span class="k">${</span><span class="nv">ETCD_DEFAULT_VERSION</span><span class="k">}</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nv">ETCD_VERSION</span><span class="o">=</span><span class="k">${</span><span class="nv">ETCD_DEFAULT_VERSION</span><span class="k">}</span> <span class="k">fi</span> <span class="c"># 下载 Etcd 二进制文件</span> <span class="k">function </span>download<span class="o">(){</span> <span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="s2">"etcd-v</span><span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="s2">-linux-amd64.tar.gz"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span>wget http://github.com/coreos/etcd/releases/download/v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span>/etcd-v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="nt">-linux-amd64</span>.tar.gz <span class="nb">tar</span> <span class="nt">-zxvf</span> etcd-v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="nt">-linux-amd64</span>.tar.gz <span class="k">fi</span> <span class="o">}</span> <span class="c"># 为 Etcd 创建单独的用户</span> <span class="k">function </span>preinstall<span class="o">(){</span> getent group etcd <span class="o">&gt;</span>/dev/null <span class="o">||</span> groupadd <span class="nt">-r</span> etcd getent passwd etcd <span class="o">&gt;</span>/dev/null <span class="o">||</span> useradd <span class="nt">-r</span> <span class="nt">-g</span> etcd <span class="nt">-d</span> /var/lib/etcd <span class="nt">-s</span> /sbin/nologin <span class="nt">-c</span> <span class="s2">"etcd user"</span> etcd <span class="o">}</span> <span class="c"># 安装(复制文件)</span> <span class="k">function </span><span class="nb">install</span><span class="o">(){</span> <span class="c"># 释放 Etcd 二进制文件</span> <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy etcd...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">tar</span> <span class="nt">-zxvf</span> etcd-v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="nt">-linux-amd64</span>.tar.gz <span class="nb">cp </span>etcd-v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="nt">-linux-amd64</span>/etcd<span class="k">*</span> /usr/local/bin <span class="nb">rm</span> <span class="nt">-rf</span> etcd-v<span class="k">${</span><span class="nv">ETCD_VERSION</span><span class="k">}</span><span class="nt">-linux-amd64</span> <span class="c"># 复制 配置文件 到 /etc/etcd(目录内文件结构在下面)</span> <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy etcd config...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">cp</span> <span class="nt">-r</span> conf /etc/etcd <span class="nb">chown</span> <span class="nt">-R</span> etcd:etcd /etc/etcd <span class="nb">chmod</span> <span class="nt">-R</span> 755 /etc/etcd/ssl <span class="c"># 复制 systemd service 配置</span> <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy etcd systemd config...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">cp </span>systemd/<span class="k">*</span>.service /lib/systemd/system systemctl daemon-reload <span class="o">}</span> <span class="c"># 创建 Etcd 存储目录(如需要更改,请求改 /etc/etcd/etcd.conf 配置文件)</span> <span class="k">function </span>postinstall<span class="o">(){</span> <span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-d</span> <span class="s2">"/var/lib/etcd"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nb">mkdir</span> /var/lib/etcd <span class="nb">chown</span> <span class="nt">-R</span> etcd:etcd /var/lib/etcd <span class="k">fi</span> <span class="o">}</span> <span class="c"># 依次执行</span> download preinstall <span class="nb">install </span>postinstall </code></pre></div></div> <h4 id="222配置文件">2.2.2、配置文件</h4> <p><strong>关于配置文件目录结构如下(请自行复制证书)</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>conf ├── etcd.conf ├── etcd.conf.cluster.example ├── etcd.conf.single.example └── ssl ├── etcd-key.pem ├── etcd.pem ├── etcd-root-ca-key.pem └── etcd-root-ca.pem 1 directory, 7 files </code></pre></div></div> <ul> <li>etcd.conf</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># [member]</span> <span class="nv">ETCD_NAME</span><span class="o">=</span>etcd1 <span class="nv">ETCD_DATA_DIR</span><span class="o">=</span><span class="s2">"/var/lib/etcd/data"</span> <span class="nv">ETCD_WAL_DIR</span><span class="o">=</span><span class="s2">"/var/lib/etcd/wal"</span> <span class="nv">ETCD_SNAPSHOT_COUNT</span><span class="o">=</span><span class="s2">"100"</span> <span class="nv">ETCD_HEARTBEAT_INTERVAL</span><span class="o">=</span><span class="s2">"100"</span> <span class="nv">ETCD_ELECTION_TIMEOUT</span><span class="o">=</span><span class="s2">"1000"</span> <span class="nv">ETCD_LISTEN_PEER_URLS</span><span class="o">=</span><span class="s2">"http://192.168.1.51:2380"</span> <span class="nv">ETCD_LISTEN_CLIENT_URLS</span><span class="o">=</span><span class="s2">"http://192.168.1.51:2379,http://127.0.0.1:2379"</span> <span class="nv">ETCD_MAX_SNAPSHOTS</span><span class="o">=</span><span class="s2">"5"</span> <span class="nv">ETCD_MAX_WALS</span><span class="o">=</span><span class="s2">"5"</span> <span class="c">#ETCD_CORS=""</span> <span class="c"># [cluster]</span> <span class="nv">ETCD_INITIAL_ADVERTISE_PEER_URLS</span><span class="o">=</span><span class="s2">"http://192.168.1.51:2380"</span> <span class="c"># if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."</span> <span class="nv">ETCD_INITIAL_CLUSTER</span><span class="o">=</span><span class="s2">"etcd1=http://192.168.1.51:2380,etcd2=http://192.168.1.52:2380,etcd3=http://192.168.1.53:2380"</span> <span class="nv">ETCD_INITIAL_CLUSTER_STATE</span><span class="o">=</span><span class="s2">"new"</span> <span class="nv">ETCD_INITIAL_CLUSTER_TOKEN</span><span class="o">=</span><span class="s2">"etcd-cluster"</span> <span class="nv">ETCD_ADVERTISE_CLIENT_URLS</span><span class="o">=</span><span class="s2">"http://192.168.1.51:2379"</span> <span class="c">#ETCD_DISCOVERY=""</span> <span class="c">#ETCD_DISCOVERY_SRV=""</span> <span class="c">#ETCD_DISCOVERY_FALLBACK="proxy"</span> <span class="c">#ETCD_DISCOVERY_PROXY=""</span> <span class="c">#ETCD_STRICT_RECONFIG_CHECK="false"</span> <span class="c">#ETCD_AUTO_COMPACTION_RETENTION="0"</span> <span class="c"># [proxy]</span> <span class="c">#ETCD_PROXY="off"</span> <span class="c">#ETCD_PROXY_FAILURE_WAIT="5000"</span> <span class="c">#ETCD_PROXY_REFRESH_INTERVAL="30000"</span> <span class="c">#ETCD_PROXY_DIAL_TIMEOUT="1000"</span> <span class="c">#ETCD_PROXY_WRITE_TIMEOUT="5000"</span> <span class="c">#ETCD_PROXY_READ_TIMEOUT="0"</span> <span class="c"># [security]</span> <span class="nv">ETCD_CERT_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd.pem"</span> <span class="nv">ETCD_KEY_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd-key.pem"</span> <span class="nv">ETCD_CLIENT_CERT_AUTH</span><span class="o">=</span><span class="s2">"true"</span> <span class="nv">ETCD_TRUSTED_CA_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd-root-ca.pem"</span> <span class="nv">ETCD_AUTO_TLS</span><span class="o">=</span><span class="s2">"true"</span> <span class="nv">ETCD_PEER_CERT_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd.pem"</span> <span class="nv">ETCD_PEER_KEY_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd-key.pem"</span> <span class="nv">ETCD_PEER_CLIENT_CERT_AUTH</span><span class="o">=</span><span class="s2">"true"</span> <span class="nv">ETCD_PEER_TRUSTED_CA_FILE</span><span class="o">=</span><span class="s2">"/etc/etcd/ssl/etcd-root-ca.pem"</span> <span class="nv">ETCD_PEER_AUTO_TLS</span><span class="o">=</span><span class="s2">"true"</span> <span class="c"># [logging]</span> <span class="c">#ETCD_DEBUG="false"</span> <span class="c"># examples for -log-package-levels etcdserver=WARNING,security=DEBUG</span> <span class="c">#ETCD_LOG_PACKAGE_LEVELS=""</span> </code></pre></div></div> <ul> <li>etcd.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Etcd Server <span class="nv">After</span><span class="o">=</span>network.target <span class="nv">After</span><span class="o">=</span>network-online.target <span class="nv">Wants</span><span class="o">=</span>network-online.target <span class="o">[</span>Service] <span class="nv">Type</span><span class="o">=</span>notify <span class="nv">WorkingDirectory</span><span class="o">=</span>/var/lib/etcd/ <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/etcd/etcd.conf <span class="nv">User</span><span class="o">=</span>etcd <span class="c"># set GOMAXPROCS to number of processors</span> <span class="nv">ExecStart</span><span class="o">=</span>/bin/bash <span class="nt">-c</span> <span class="s2">"GOMAXPROCS=</span><span class="si">$(</span><span class="nb">nproc</span><span class="si">)</span><span class="s2"> /usr/local/bin/etcd --name=</span><span class="se">\"</span><span class="k">${</span><span class="nv">ETCD_NAME</span><span class="k">}</span><span class="se">\"</span><span class="s2"> --data-dir=</span><span class="se">\"</span><span class="k">${</span><span class="nv">ETCD_DATA_DIR</span><span class="k">}</span><span class="se">\"</span><span class="s2"> --listen-client-urls=</span><span class="se">\"</span><span class="k">${</span><span class="nv">ETCD_LISTEN_CLIENT_URLS</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">LimitNOFILE</span><span class="o">=</span>65536 <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <p>最后三台机器依次修改 <code class="highlighter-rouge">IP</code>、<code class="highlighter-rouge">ETCD_NAME</code> 然后启动即可,<strong>生产环境请不要忘记修改集群 Token 为真实随机字符串 (<code class="highlighter-rouge">ETCD_INITIAL_CLUSTER_TOKEN</code> 变量)</strong>启动后可以通过以下命令测试集群联通性</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker1.node ➜ ~ <span class="nb">export </span><span class="nv">ETCDCTL_API</span><span class="o">=</span>3 docker1.node ➜ ~ etcdctl member list 238b72cdd26e304f, started, etcd2, http://192.168.1.52:2380, http://192.168.1.52:2379 8034142cf01c5d1c, started, etcd3, http://192.168.1.53:2380, http://192.168.1.53:2379 8da171dbef9ded69, started, etcd1, http://192.168.1.51:2380, http://192.168.1.51:2379 </code></pre></div></div> <h2 id="三安装-kubernetes">三、安装 Kubernetes</h2> <h3 id="31生成证书及配置">3.1、生成证书及配置</h3> <h4 id="311生成证书">3.1.1、生成证书</h4> <p>新版本已经越来越趋近全面 TLS + RBAC 配置,<strong>所以本次安装将会启动大部分 TLS + RBAC 配置,包括 <code class="highlighter-rouge">kube-controler-manager</code>、<code class="highlighter-rouge">kube-scheduler</code> 组件不再连接本地 <code class="highlighter-rouge">kube-apiserver</code> 的 8080 非认证端口,<code class="highlighter-rouge">kubelet</code> 等组件 API 端点关闭匿名访问,启动 RBAC 认证等</strong>;为了满足这些认证,需要签署以下证书</p> <ul> <li>k8s-root-ca-csr.json 集群 CA 根证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubernetes"</span><span class="p">,</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">4096</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubernetes"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"ca"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"expiry"</span><span class="p">:</span><span class="w"> </span><span class="s2">"87600h"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>k8s-gencert.json 用于生成其他证书的标准配置</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"signing"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"default"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"expiry"</span><span class="p">:</span><span class="w"> </span><span class="s2">"87600h"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"profiles"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"kubernetes"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"usages"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"signing"</span><span class="p">,</span><span class="w"> </span><span class="s2">"key encipherment"</span><span class="p">,</span><span class="w"> </span><span class="s2">"server auth"</span><span class="p">,</span><span class="w"> </span><span class="s2">"client auth"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"expiry"</span><span class="p">:</span><span class="w"> </span><span class="s2">"87600h"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>kube-apiserver-csr.json apiserver TLS 认证端口需要的证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubernetes"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"127.0.0.1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"10.254.0.1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w"> </span><span class="s2">"*.master.kubernetes.node"</span><span class="p">,</span><span class="w"> </span><span class="s2">"kubernetes"</span><span class="p">,</span><span class="w"> </span><span class="s2">"kubernetes.default"</span><span class="p">,</span><span class="w"> </span><span class="s2">"kubernetes.default.svc"</span><span class="p">,</span><span class="w"> </span><span class="s2">"kubernetes.default.svc.cluster"</span><span class="p">,</span><span class="w"> </span><span class="s2">"kubernetes.default.svc.cluster.local"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"kubernetes"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>kube-controller-manager-csr.json controller manager 连接 apiserver 需要使用的证书,同时本身 <code class="highlighter-rouge">10257</code> 端口也会使用此证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-controller-manager"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"127.0.0.1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w"> </span><span class="s2">"*.master.kubernetes.node"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-controller-manager"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>kube-scheduler-csr.json scheduler 连接 apiserver 需要使用的证书,同时本身 <code class="highlighter-rouge">10259</code> 端口也会使用此证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-scheduler"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"127.0.0.1"</span><span class="p">,</span><span class="w"> </span><span class="s2">"localhost"</span><span class="p">,</span><span class="w"> </span><span class="s2">"*.master.kubernetes.node"</span><span class="w"> </span><span class="p">],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-scheduler"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>kube-proxy-csr.json proxy 组件连接 apiserver 需要使用的证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-proxy"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kube-proxy"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>kubelet-api-admin-csr.json apiserver 反向连接 kubelet 组件 <code class="highlighter-rouge">10250</code> 端口需要使用的证书(例如执行 <code class="highlighter-rouge">kubectl logs</code>)</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kubelet-api-admin"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:kubelet-api-admin"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <ul> <li>admin-csr.json 集群管理员(kubectl)连接 apiserver 需要使用的证书</li> </ul> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"CN"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:masters"</span><span class="p">,</span><span class="w"> </span><span class="nl">"hosts"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w"> </span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"algo"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rsa"</span><span class="p">,</span><span class="w"> </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">2048</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"names"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"C"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CN"</span><span class="p">,</span><span class="w"> </span><span class="nl">"ST"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"L"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BeiJing"</span><span class="p">,</span><span class="w"> </span><span class="nl">"O"</span><span class="p">:</span><span class="w"> </span><span class="s2">"system:masters"</span><span class="p">,</span><span class="w"> </span><span class="nl">"OU"</span><span class="p">:</span><span class="w"> </span><span class="s2">"System"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p><strong>注意: 请不要修改证书配置的 <code class="highlighter-rouge">CN</code>、<code class="highlighter-rouge">O</code> 字段,这两个字段名称比较特殊,大多数为 <code class="highlighter-rouge">system:</code> 开头,实际上是为了匹配 RBAC 规则,具体请参考 <a href="http://kubernetes.io/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings">Default Roles and Role Bindings</a></strong></p> <p>最后使用如下命令生成即可:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cfssl gencert <span class="nt">--initca</span><span class="o">=</span><span class="nb">true </span>k8s-root-ca-csr.json | cfssljson <span class="nt">--bare</span> k8s-root-ca <span class="k">for </span>targetName <span class="k">in </span>kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet-api-admin admin<span class="p">;</span> <span class="k">do </span>cfssl gencert <span class="nt">--ca</span> k8s-root-ca.pem <span class="nt">--ca-key</span> k8s-root-ca-key.pem <span class="nt">--config</span> k8s-gencert.json <span class="nt">--profile</span> kubernetes <span class="nv">$targetName</span><span class="nt">-csr</span>.json | cfssljson <span class="nt">--</span> bare <span class="nv">$targetName</span> <span class="k">done</span> </code></pre></div></div> <h4 id="312生成配置文件">3.1.2、生成配置文件</h4> <p>集群搭建需要预先生成一系列配置文件,生成配置需要预先安装 <code class="highlighter-rouge">kubectl</code> 命令,请自行根据文档安装 <a href="http://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-binary-using-curl">Install kubectl binary using curl</a>;其中配置文件及其作用如下:</p> <ul> <li><code class="highlighter-rouge">bootstrap.kubeconfig</code> kubelet TLS Bootstarp 引导阶段需要使用的配置文件</li> <li><code class="highlighter-rouge">kube-controller-manager.kubeconfig</code> controller manager 组件开启安全端口及 RBAC 认证所需配置</li> <li><code class="highlighter-rouge">kube-scheduler.kubeconfig</code> scheduler 组件开启安全端口及 RBAC 认证所需配置</li> <li><code class="highlighter-rouge">kube-proxy.kubeconfig</code> proxy 组件连接 apiserver 所需配置文件</li> <li><code class="highlighter-rouge">audit-policy.yaml</code> apiserver RBAC 审计日志配置文件</li> <li><code class="highlighter-rouge">bootstrap.secret.yaml</code> kubelet TLS Bootstarp 引导阶段使用 Bootstrap Token 方式引导,需要预先创建此 Token</li> </ul> <p>生成这些配置文件的脚本如下</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 指定 apiserver 地址</span> <span class="nv">KUBE_APISERVER</span><span class="o">=</span><span class="s2">"http://127.0.0.1:6443"</span> <span class="c"># 生成 Bootstrap Token</span> <span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="o">=</span><span class="si">$(</span><span class="nb">head</span> <span class="nt">-c</span> 6 /dev/urandom | <span class="nb">md5sum</span> | <span class="nb">head</span> <span class="nt">-c</span> 6<span class="si">)</span> <span class="nv">BOOTSTRAP_TOKEN_SECRET</span><span class="o">=</span><span class="si">$(</span><span class="nb">head</span> <span class="nt">-c</span> 16 /dev/urandom | <span class="nb">md5sum</span> | <span class="nb">head</span> <span class="nt">-c</span> 16<span class="si">)</span> <span class="nv">BOOTSTRAP_TOKEN</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="k">}</span><span class="s2">.</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_SECRET</span><span class="k">}</span><span class="s2">"</span> <span class="nb">echo</span> <span class="s2">"Bootstrap Tokne: </span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN</span><span class="k">}</span><span class="s2">"</span> <span class="c"># 生成 kubelet tls bootstrap 配置</span> <span class="nb">echo</span> <span class="s2">"Create kubelet bootstrapping kubeconfig..."</span> kubectl config set-cluster kubernetes <span class="se">\</span> <span class="nt">--certificate-authority</span><span class="o">=</span>k8s-root-ca.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--server</span><span class="o">=</span><span class="k">${</span><span class="nv">KUBE_APISERVER</span><span class="k">}</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig kubectl config set-credentials <span class="s2">"system:bootstrap:</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="k">}</span><span class="s2">"</span> <span class="se">\</span> <span class="nt">--token</span><span class="o">=</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN</span><span class="k">}</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig kubectl config set-context default <span class="se">\</span> <span class="nt">--cluster</span><span class="o">=</span>kubernetes <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span><span class="s2">"system:bootstrap:</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="k">}</span><span class="s2">"</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig kubectl config use-context default <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig <span class="c"># 生成 kube-controller-manager 配置文件</span> <span class="nb">echo</span> <span class="s2">"Create kube-controller-manager kubeconfig..."</span> kubectl config set-cluster kubernetes <span class="se">\</span> <span class="nt">--certificate-authority</span><span class="o">=</span>k8s-root-ca.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--server</span><span class="o">=</span><span class="k">${</span><span class="nv">KUBE_APISERVER</span><span class="k">}</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-controller-manager.kubeconfig kubectl config set-credentials <span class="s2">"system:kube-controller-manager"</span> <span class="se">\</span> <span class="nt">--client-certificate</span><span class="o">=</span>kube-controller-manager.pem <span class="se">\</span> <span class="nt">--client-key</span><span class="o">=</span>kube-controller-manager-key.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-controller-manager.kubeconfig kubectl config set-context default <span class="se">\</span> <span class="nt">--cluster</span><span class="o">=</span>kubernetes <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span>system:kube-controller-manager <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-controller-manager.kubeconfig kubectl config use-context default <span class="nt">--kubeconfig</span><span class="o">=</span>kube-controller-manager.kubeconfig <span class="c"># 生成 kube-scheduler 配置文件</span> <span class="nb">echo</span> <span class="s2">"Create kube-scheduler kubeconfig..."</span> kubectl config set-cluster kubernetes <span class="se">\</span> <span class="nt">--certificate-authority</span><span class="o">=</span>k8s-root-ca.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--server</span><span class="o">=</span><span class="k">${</span><span class="nv">KUBE_APISERVER</span><span class="k">}</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-scheduler.kubeconfig kubectl config set-credentials <span class="s2">"system:kube-scheduler"</span> <span class="se">\</span> <span class="nt">--client-certificate</span><span class="o">=</span>kube-scheduler.pem <span class="se">\</span> <span class="nt">--client-key</span><span class="o">=</span>kube-scheduler-key.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-scheduler.kubeconfig kubectl config set-context default <span class="se">\</span> <span class="nt">--cluster</span><span class="o">=</span>kubernetes <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span>system:kube-scheduler <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-scheduler.kubeconfig kubectl config use-context default <span class="nt">--kubeconfig</span><span class="o">=</span>kube-scheduler.kubeconfig <span class="c"># 生成 kube-proxy 配置文件</span> <span class="nb">echo</span> <span class="s2">"Create kube-proxy kubeconfig..."</span> kubectl config set-cluster kubernetes <span class="se">\</span> <span class="nt">--certificate-authority</span><span class="o">=</span>k8s-root-ca.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--server</span><span class="o">=</span><span class="k">${</span><span class="nv">KUBE_APISERVER</span><span class="k">}</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-proxy.kubeconfig kubectl config set-credentials <span class="s2">"system:kube-proxy"</span> <span class="se">\</span> <span class="nt">--client-certificate</span><span class="o">=</span>kube-proxy.pem <span class="se">\</span> <span class="nt">--client-key</span><span class="o">=</span>kube-proxy-key.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-proxy.kubeconfig kubectl config set-context default <span class="se">\</span> <span class="nt">--cluster</span><span class="o">=</span>kubernetes <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span>system:kube-proxy <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>kube-proxy.kubeconfig kubectl config use-context default <span class="nt">--kubeconfig</span><span class="o">=</span>kube-proxy.kubeconfig <span class="c"># 生成 apiserver RBAC 审计配置文件 </span> <span class="nb">cat</span> <span class="o">&gt;&gt;</span> audit-policy.yaml <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh"> # Log all requests at the Metadata level. apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata </span><span class="no">EOF </span><span class="c"># 生成 tls bootstrap token secret 配置文件</span> <span class="nb">cat</span> <span class="o">&gt;&gt;</span> bootstrap.secret.yaml <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh"> apiVersion: v1 kind: Secret metadata: # Name MUST be of form "bootstrap-token-&lt;token id&gt;" name: bootstrap-token-</span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="k">}</span><span class="sh"> namespace: kube-system # Type MUST be 'bootstrap.kubernetes.io/token' type: bootstrap.kubernetes.io/token stringData: # Human readable description. Optional. description: "The default bootstrap token." # Token ID and secret. Required. token-id: </span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_ID</span><span class="k">}</span><span class="sh"> token-secret: </span><span class="k">${</span><span class="nv">BOOTSTRAP_TOKEN_SECRET</span><span class="k">}</span><span class="sh"> # Expiration. Optional. expiration: </span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span><span class="s1">'+2 day'</span> <span class="nt">-u</span> +<span class="s2">"%Y-%m-%dT%H:%M:%SZ"</span><span class="si">)</span><span class="sh"> # Allowed usages. usage-bootstrap-authentication: "true" usage-bootstrap-signing: "true" # Extra groups to authenticate the token as. Must start with "system:bootstrappers:" # auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress </span><span class="no">EOF </span></code></pre></div></div> <h3 id="32处理-ipvs-及依赖">3.2、处理 ipvs 及依赖</h3> <p>新版本目前 <code class="highlighter-rouge">kube-proxy</code> 组件全部采用 ipvs 方式负载,所以为了 <code class="highlighter-rouge">kube-proxy</code> 能正常工作需要预先处理一下 ipvs 配置以及相关依赖(每台 node 都要处理)</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">&gt;&gt;</span> /etc/sysctl.conf <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh"> net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 </span><span class="no">EOF </span>sysctl <span class="nt">-p</span> <span class="nb">cat</span> <span class="o">&gt;&gt;</span> /etc/modules <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh"> ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp </span><span class="no">EOF </span>apt <span class="nb">install</span> <span class="nt">-y</span> conntrack ipvsadm </code></pre></div></div> <h3 id="33部署-master">3.3、部署 Master</h3> <h4 id="331安装脚本">3.3.1、安装脚本</h4> <p>master 节点上需要三个组件: <code class="highlighter-rouge">kube-apiserver</code>、<code class="highlighter-rouge">kube-controller-manager</code>、<code class="highlighter-rouge">kube-scheduler</code></p> <p><strong>安装流程整体为以下几步</strong></p> <ul> <li><strong>创建单独的 <code class="highlighter-rouge">kube</code> 用户</strong></li> <li><strong>复制相关二进制文件到 <code class="highlighter-rouge">/usr/bin</code>,可以采用 <code class="highlighter-rouge">all in one</code> 的 <code class="highlighter-rouge">hyperkube</code></strong></li> <li><strong>复制配置文件到 <code class="highlighter-rouge">/etc/kubernetes</code></strong></li> <li><strong>复制证书文件到 <code class="highlighter-rouge">/etc/kubernetes/ssl</code></strong></li> <li><strong>修改配置并启动</strong></li> </ul> <p>安装脚本如下所示:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">KUBE_DEFAULT_VERSION</span><span class="o">=</span><span class="s2">"1.13.4"</span> <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">""</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nv">KUBE_VERSION</span><span class="o">=</span><span class="nv">$1</span> <span class="k">else </span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[33mWARNING: KUBE_VERSION is blank,use default version: </span><span class="k">${</span><span class="nv">KUBE_DEFAULT_VERSION</span><span class="k">}</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nv">KUBE_VERSION</span><span class="o">=</span><span class="k">${</span><span class="nv">KUBE_DEFAULT_VERSION</span><span class="k">}</span> <span class="k">fi</span> <span class="c"># 下载 hyperkube</span> <span class="k">function </span>download_k8s<span class="o">(){</span> <span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="s2">"hyperkube_v</span><span class="k">${</span><span class="nv">KUBE_VERSION</span><span class="k">}</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span>wget http://storage.googleapis.com/kubernetes-release/release/v<span class="k">${</span><span class="nv">KUBE_VERSION</span><span class="k">}</span>/bin/linux/amd64/hyperkube <span class="nt">-O</span> hyperkube_v<span class="k">${</span><span class="nv">KUBE_VERSION</span><span class="k">}</span> <span class="nb">chmod</span> +x hyperkube_v<span class="k">${</span><span class="nv">KUBE_VERSION</span><span class="k">}</span> <span class="k">fi</span> <span class="o">}</span> <span class="c"># 创建专用用户 kube</span> <span class="k">function </span>preinstall<span class="o">(){</span> getent group kube <span class="o">&gt;</span>/dev/null <span class="o">||</span> groupadd <span class="nt">-r</span> kube getent passwd kube <span class="o">&gt;</span>/dev/null <span class="o">||</span> useradd <span class="nt">-r</span> <span class="nt">-g</span> kube <span class="nt">-d</span> / <span class="nt">-s</span> /sbin/nologin <span class="nt">-c</span> <span class="s2">"Kubernetes user"</span> kube <span class="o">}</span> <span class="c"># 复制可执行文件和配置以及证书</span> <span class="k">function </span>install_k8s<span class="o">(){</span> <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy hyperkube...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">cp </span>hyperkube_v<span class="k">${</span><span class="nv">KUBE_VERSION</span><span class="k">}</span> /usr/bin/hyperkube <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Create symbolic link...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="o">(</span><span class="nb">cd</span> /usr/bin <span class="o">&amp;&amp;</span> hyperkube <span class="nt">--make-symlinks</span><span class="o">)</span> <span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy kubernetes config...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">cp</span> <span class="nt">-r</span> conf /etc/kubernetes <span class="k">if</span> <span class="o">[</span> <span class="nt">-d</span> <span class="s2">"/etc/kubernetes/ssl"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nb">chown</span> <span class="nt">-R</span> kube:kube /etc/kubernetes/ssl <span class="k">fi </span><span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\0</span><span class="s2">33[32mINFO: Copy kubernetes systemd config...</span><span class="se">\0</span><span class="s2">33[0m"</span> <span class="nb">cp </span>systemd/<span class="k">*</span>.service /lib/systemd/system systemctl daemon-reload <span class="o">}</span> <span class="c"># 创建必要的目录并修改权限</span> <span class="k">function </span>postinstall<span class="o">(){</span> <span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-d</span> <span class="s2">"/var/log/kube-audit"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nb">mkdir</span> /var/log/kube-audit <span class="k">fi if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-d</span> <span class="s2">"/var/lib/kubelet"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nb">mkdir</span> /var/lib/kubelet <span class="k">fi if</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-d</span> <span class="s2">"/usr/libexec"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then </span><span class="nb">mkdir</span> /usr/libexec <span class="k">fi </span><span class="nb">chown</span> <span class="nt">-R</span> kube:kube /etc/kubernetes /var/log/kube-audit /var/lib/kubelet /usr/libexec <span class="o">}</span> <span class="c"># 执行</span> download_k8s preinstall install_k8s postinstall </code></pre></div></div> <p><strong>hyperkube 是一个多合一的可执行文件,通过 <code class="highlighter-rouge">--make-symlinks</code> 会在当前目录生成 kubernetes 各个组件的软连接</strong></p> <p>被复制的 conf 目录结构如下(最终被复制到 <code class="highlighter-rouge">/etc/kubernetes</code>)</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">.</span> ├── apiserver ├── audit-policy.yaml ├── bootstrap.kubeconfig ├── bootstrap.secret.yaml ├── controller-manager ├── kube-controller-manager.kubeconfig ├── kubelet ├── kube-proxy.kubeconfig ├── kube-scheduler.kubeconfig ├── proxy ├── scheduler └── ssl ├── admin-key.pem ├── admin.pem ├── k8s-root-ca-key.pem ├── k8s-root-ca.pem ├── kube-apiserver-key.pem ├── kube-apiserver.pem ├── kube-controller-manager-key.pem ├── kube-controller-manager.pem ├── kubelet-api-admin-key.pem ├── kubelet-api-admin.pem ├── kube-proxy-key.pem ├── kube-proxy.pem ├── kube-scheduler-key.pem └── kube-scheduler.pem 1 directory, 25 files </code></pre></div></div> <h4 id="332配置文件">3.3.2、配置文件</h4> <p>以下为相关配置文件内容</p> <p><strong>systemd 配置如下</strong></p> <ul> <li>kube-apiserver.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Kubernetes API Server <span class="nv">Documentation</span><span class="o">=</span>http://github.com/GoogleCloudPlatform/kubernetes <span class="nv">After</span><span class="o">=</span>network.target <span class="nv">After</span><span class="o">=</span>etcd.service <span class="o">[</span>Service] <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/kubernetes/apiserver <span class="nv">User</span><span class="o">=</span>kube <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/kube-apiserver <span class="se">\</span> <span class="nv">$KUBE_LOGTOSTDERR</span> <span class="se">\</span> <span class="nv">$KUBE_LOG_LEVEL</span> <span class="se">\</span> <span class="nv">$KUBE_ETCD_SERVERS</span> <span class="se">\</span> <span class="nv">$KUBE_API_ADDRESS</span> <span class="se">\</span> <span class="nv">$KUBE_API_PORT</span> <span class="se">\</span> <span class="nv">$KUBELET_PORT</span> <span class="se">\</span> <span class="nv">$KUBE_ALLOW_PRIV</span> <span class="se">\</span> <span class="nv">$KUBE_SERVICE_ADDRESSES</span> <span class="se">\</span> <span class="nv">$KUBE_ADMISSION_CONTROL</span> <span class="se">\</span> <span class="nv">$KUBE_API_ARGS</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">Type</span><span class="o">=</span>notify <span class="nv">LimitNOFILE</span><span class="o">=</span>65536 <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <ul> <li>kube-controller-manager.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Kubernetes Controller Manager <span class="nv">Documentation</span><span class="o">=</span>http://github.com/GoogleCloudPlatform/kubernetes <span class="o">[</span>Service] <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/kubernetes/controller-manager <span class="nv">User</span><span class="o">=</span>kube <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/kube-controller-manager <span class="se">\</span> <span class="nv">$KUBE_LOGTOSTDERR</span> <span class="se">\</span> <span class="nv">$KUBE_LOG_LEVEL</span> <span class="se">\</span> <span class="nv">$KUBE_MASTER</span> <span class="se">\</span> <span class="nv">$KUBE_CONTROLLER_MANAGER_ARGS</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">LimitNOFILE</span><span class="o">=</span>65536 <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <ul> <li>kube-scheduler.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Kubernetes Scheduler Plugin <span class="nv">Documentation</span><span class="o">=</span>http://github.com/GoogleCloudPlatform/kubernetes <span class="o">[</span>Service] <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/kubernetes/scheduler <span class="nv">User</span><span class="o">=</span>kube <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/kube-scheduler <span class="se">\</span> <span class="nv">$KUBE_LOGTOSTDERR</span> <span class="se">\</span> <span class="nv">$KUBE_LOG_LEVEL</span> <span class="se">\</span> <span class="nv">$KUBE_MASTER</span> <span class="se">\</span> <span class="nv">$KUBE_SCHEDULER_ARGS</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">LimitNOFILE</span><span class="o">=</span>65536 <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <p><strong>核心配置文件</strong></p> <ul> <li>apiserver</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###</span> <span class="c"># kubernetes system config</span> <span class="c">#</span> <span class="c"># The following values are used to configure the kube-apiserver</span> <span class="c">#</span> <span class="c"># The address on the local server to listen to.</span> <span class="nv">KUBE_API_ADDRESS</span><span class="o">=</span><span class="s2">"--advertise-address=192.168.1.51 --bind-address=0.0.0.0"</span> <span class="c"># The port on the local server to listen on.</span> <span class="nv">KUBE_API_PORT</span><span class="o">=</span><span class="s2">"--secure-port=6443"</span> <span class="c"># Port minions listen on</span> <span class="c"># KUBELET_PORT="--kubelet-port=10250"</span> <span class="c"># Comma separated list of nodes in the etcd cluster</span> <span class="nv">KUBE_ETCD_SERVERS</span><span class="o">=</span><span class="s2">"--etcd-servers=http://192.168.1.51:2379,http://192.168.1.52:2379,http://192.168.1.53:2379"</span> <span class="c"># Address range to use for services</span> <span class="nv">KUBE_SERVICE_ADDRESSES</span><span class="o">=</span><span class="s2">"--service-cluster-ip-range=10.254.0.0/16"</span> <span class="c"># default admission control policies</span> <span class="nv">KUBE_ADMISSION_CONTROL</span><span class="o">=</span><span class="s2">"--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota"</span> <span class="c"># Add your own!</span> <span class="nv">KUBE_API_ARGS</span><span class="o">=</span><span class="s2">" --allow-privileged=true </span><span class="se">\</span><span class="s2"> --anonymous-auth=false </span><span class="se">\</span><span class="s2"> --alsologtostderr </span><span class="se">\</span><span class="s2"> --apiserver-count=3 </span><span class="se">\</span><span class="s2"> --audit-log-maxage=30 </span><span class="se">\</span><span class="s2"> --audit-log-maxbackup=3 </span><span class="se">\</span><span class="s2"> --audit-log-maxsize=100 </span><span class="se">\</span><span class="s2"> --audit-log-path=/var/log/kube-audit/audit.log </span><span class="se">\</span><span class="s2"> --audit-policy-file=/etc/kubernetes/audit-policy.yaml </span><span class="se">\</span><span class="s2"> --authorization-mode=Node,RBAC </span><span class="se">\</span><span class="s2"> --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --enable-bootstrap-token-auth </span><span class="se">\</span><span class="s2"> --enable-garbage-collector </span><span class="se">\</span><span class="s2"> --enable-logs-handler </span><span class="se">\</span><span class="s2"> --endpoint-reconciler-type=lease </span><span class="se">\</span><span class="s2"> --etcd-cafile=/etc/etcd/ssl/etcd-root-ca.pem </span><span class="se">\</span><span class="s2"> --etcd-certfile=/etc/etcd/ssl/etcd.pem </span><span class="se">\</span><span class="s2"> --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem </span><span class="se">\</span><span class="s2"> --etcd-compaction-interval=0s </span><span class="se">\</span><span class="s2"> --event-ttl=168h0m0s </span><span class="se">\</span><span class="s2"> --kubelet-http=true </span><span class="se">\</span><span class="s2"> --kubelet-certificate-authority=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --kubelet-client-certificate=/etc/kubernetes/ssl/kubelet-api-admin.pem </span><span class="se">\</span><span class="s2"> --kubelet-client-key=/etc/kubernetes/ssl/kubelet-api-admin-key.pem </span><span class="se">\</span><span class="s2"> --kubelet-timeout=3s </span><span class="se">\</span><span class="s2"> --runtime-config=api/all=true </span><span class="se">\</span><span class="s2"> --service-node-port-range=30000-50000 </span><span class="se">\</span><span class="s2"> --service-account-key-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem </span><span class="se">\</span><span class="s2"> --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem </span><span class="se">\</span><span class="s2"> --v=2"</span> </code></pre></div></div> <p>配置解释:</p> <table> <thead> <tr> <th>选项</th> <th>作用</th> </tr> </thead> <tbody> <tr> <td><code class="highlighter-rouge">--client-ca-file</code></td> <td>定义客户端 CA</td> </tr> <tr> <td><code class="highlighter-rouge">--endpoint-reconciler-type</code></td> <td>master endpoint 策略</td> </tr> <tr> <td><code class="highlighter-rouge">--kubelet-client-certificate</code>、<code class="highlighter-rouge">--kubelet-client-key</code></td> <td>master 反向连接 kubelet 使用的证书</td> </tr> <tr> <td><code class="highlighter-rouge">--service-account-key-file</code></td> <td>service account 签名 key(用于有效性验证)</td> </tr> <tr> <td><code class="highlighter-rouge">--tls-cert-file</code>、<code class="highlighter-rouge">--tls-private-key-file</code></td> <td>master apiserver <code class="highlighter-rouge">6443</code> 端口证书</td> </tr> </tbody> </table> <ul> <li>controller-manager</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###</span> <span class="c"># The following values are used to configure the kubernetes controller-manager</span> <span class="c"># defaults from config and apiserver should be adequate</span> <span class="c"># Add your own!</span> <span class="nv">KUBE_CONTROLLER_MANAGER_ARGS</span><span class="o">=</span><span class="s2">" --address=127.0.0.1 </span><span class="se">\</span><span class="s2"> --authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig </span><span class="se">\</span><span class="s2"> --authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig </span><span class="se">\</span><span class="s2"> --bind-address=0.0.0.0 </span><span class="se">\</span><span class="s2"> --cluster-name=kubernetes </span><span class="se">\</span><span class="s2"> --cluster-signing-cert-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --cluster-signing-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem </span><span class="se">\</span><span class="s2"> --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --controllers=*,bootstrapsigner,tokencleaner </span><span class="se">\</span><span class="s2"> --deployment-controller-sync-period=10s </span><span class="se">\</span><span class="s2"> --experimental-cluster-signing-duration=87600h0m0s </span><span class="se">\</span><span class="s2"> --enable-garbage-collector=true </span><span class="se">\</span><span class="s2"> --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig </span><span class="se">\</span><span class="s2"> --leader-elect=true </span><span class="se">\</span><span class="s2"> --node-monitor-grace-period=20s </span><span class="se">\</span><span class="s2"> --node-monitor-period=5s </span><span class="se">\</span><span class="s2"> --port=10252 </span><span class="se">\</span><span class="s2"> --pod-eviction-timeout=2m0s </span><span class="se">\</span><span class="s2"> --requestheader-client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --terminated-pod-gc-threshold=50 </span><span class="se">\</span><span class="s2"> --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem </span><span class="se">\</span><span class="s2"> --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem </span><span class="se">\</span><span class="s2"> --root-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --secure-port=10257 </span><span class="se">\</span><span class="s2"> --service-cluster-ip-range=10.254.0.0/16 </span><span class="se">\</span><span class="s2"> --service-account-private-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem </span><span class="se">\</span><span class="s2"> --use-service-account-credentials=true </span><span class="se">\</span><span class="s2"> --v=2"</span> </code></pre></div></div> <p>controller manager 将不安全端口 <code class="highlighter-rouge">10252</code> 绑定到 127.0.0.1 确保 <code class="highlighter-rouge">kuebctl get cs</code> 有正确返回;将安全端口 <code class="highlighter-rouge">10257</code> 绑定到 0.0.0.0 公开,提供服务调用;<strong>由于 controller manager 开始连接 apiserver 的 <code class="highlighter-rouge">6443</code> 认证端口,所以需要 <code class="highlighter-rouge">--use-service-account-credentials</code> 选项来让 controller manager 创建单独的 service account(默认 <code class="highlighter-rouge">system:kube-controller-manager</code> 用户没有那么高权限)</strong></p> <ul> <li>scheduler</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###</span> <span class="c"># kubernetes scheduler config</span> <span class="c"># default config should be adequate</span> <span class="c"># Add your own!</span> <span class="nv">KUBE_SCHEDULER_ARGS</span><span class="o">=</span><span class="s2">" --address=127.0.0.1 </span><span class="se">\</span><span class="s2"> --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig </span><span class="se">\</span><span class="s2"> --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig </span><span class="se">\</span><span class="s2"> --bind-address=0.0.0.0 </span><span class="se">\</span><span class="s2"> --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig </span><span class="se">\</span><span class="s2"> --requestheader-client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --secure-port=10259 </span><span class="se">\</span><span class="s2"> --leader-elect=true </span><span class="se">\</span><span class="s2"> --port=10251 </span><span class="se">\</span><span class="s2"> --tls-cert-file=/etc/kubernetes/ssl/kube-scheduler.pem </span><span class="se">\</span><span class="s2"> --tls-private-key-file=/etc/kubernetes/ssl/kube-scheduler-key.pem </span><span class="se">\</span><span class="s2"> --v=2"</span> </code></pre></div></div> <p>shceduler 同 controller manager 一样将不安全端口绑定在本地,安全端口对外公开</p> <p><strong>最后在三台节点上调整一下 IP 配置,启动即可</strong></p> <h3 id="34部署-node">3.4、部署 Node</h3> <h4 id="341安装脚本">3.4.1、安装脚本</h4> <p>node 安装与 master 安装过程一致,这里不再阐述</p> <h4 id="342配置文件">3.4.2、配置文件</h4> <p><strong>systemd 配置文件</strong></p> <ul> <li>kubelet.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Kubernetes Kubelet Server <span class="nv">Documentation</span><span class="o">=</span>http://github.com/GoogleCloudPlatform/kubernetes <span class="nv">After</span><span class="o">=</span>docker.service <span class="nv">Requires</span><span class="o">=</span>docker.service <span class="o">[</span>Service] <span class="nv">WorkingDirectory</span><span class="o">=</span>/var/lib/kubelet <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/kubernetes/kubelet <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/kubelet <span class="se">\</span> <span class="nv">$KUBE_LOGTOSTDERR</span> <span class="se">\</span> <span class="nv">$KUBE_LOG_LEVEL</span> <span class="se">\</span> <span class="nv">$KUBELET_API_SERVER</span> <span class="se">\</span> <span class="nv">$KUBELET_ADDRESS</span> <span class="se">\</span> <span class="nv">$KUBELET_PORT</span> <span class="se">\</span> <span class="nv">$KUBELET_HOSTNAME</span> <span class="se">\</span> <span class="nv">$KUBE_ALLOW_PRIV</span> <span class="se">\</span> <span class="nv">$KUBELET_ARGS</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">KillMode</span><span class="o">=</span>process <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <ul> <li>kube-proxy.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>Kubernetes Kube-Proxy Server <span class="nv">Documentation</span><span class="o">=</span>http://github.com/GoogleCloudPlatform/kubernetes <span class="nv">After</span><span class="o">=</span>network.target <span class="o">[</span>Service] <span class="nv">EnvironmentFile</span><span class="o">=</span>-/etc/kubernetes/proxy <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/kube-proxy <span class="se">\</span> <span class="nv">$KUBE_LOGTOSTDERR</span> <span class="se">\</span> <span class="nv">$KUBE_LOG_LEVEL</span> <span class="se">\</span> <span class="nv">$KUBE_MASTER</span> <span class="se">\</span> <span class="nv">$KUBE_PROXY_ARGS</span> <span class="nv">Restart</span><span class="o">=</span>on-failure <span class="nv">LimitNOFILE</span><span class="o">=</span>65536 <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <p><strong>核心配置文件</strong></p> <ul> <li>kubelet</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###</span> <span class="c"># kubernetes kubelet (minion) config</span> <span class="c"># The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)</span> <span class="nv">KUBELET_ADDRESS</span><span class="o">=</span><span class="s2">"--node-ip=192.168.1.54"</span> <span class="c"># The port for the info server to serve on</span> <span class="c"># KUBELET_PORT="--port=10250"</span> <span class="c"># You may leave this blank to use the actual hostname</span> <span class="nv">KUBELET_HOSTNAME</span><span class="o">=</span><span class="s2">"--hostname-override=docker4.node"</span> <span class="c"># location of the api-server</span> <span class="c"># KUBELET_API_SERVER=""</span> <span class="c"># Add your own!</span> <span class="nv">KUBELET_ARGS</span><span class="o">=</span><span class="s2">" --address=0.0.0.0 </span><span class="se">\</span><span class="s2"> --allow-privileged </span><span class="se">\</span><span class="s2"> --anonymous-auth=false </span><span class="se">\</span><span class="s2"> --authorization-mode=Webhook </span><span class="se">\</span><span class="s2"> --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig </span><span class="se">\</span><span class="s2"> --client-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --network-plugin=cni </span><span class="se">\</span><span class="s2"> --cgroup-driver=cgroupfs </span><span class="se">\</span><span class="s2"> --cert-dir=/etc/kubernetes/ssl </span><span class="se">\</span><span class="s2"> --cluster-dns=10.254.0.2 </span><span class="se">\</span><span class="s2"> --cluster-domain=cluster.local </span><span class="se">\</span><span class="s2"> --cni-conf-dir=/etc/cni/net.d </span><span class="se">\</span><span class="s2"> --eviction-soft=imagefs.available&lt;15%,memory.available&lt;512Mi,nodefs.available&lt;15%,nodefs.inodesFree&lt;10% </span><span class="se">\</span><span class="s2"> --eviction-soft-grace-period=imagefs.available=3m,memory.available=1m,nodefs.available=3m,nodefs.inodesFree=1m </span><span class="se">\</span><span class="s2"> --eviction-hard=imagefs.available&lt;10%,memory.available&lt;256Mi,nodefs.available&lt;10%,nodefs.inodesFree&lt;5% </span><span class="se">\</span><span class="s2"> --eviction-max-pod-grace-period=30 </span><span class="se">\</span><span class="s2"> --image-gc-high-threshold=80 </span><span class="se">\</span><span class="s2"> --image-gc-low-threshold=70 </span><span class="se">\</span><span class="s2"> --image-pull-progress-deadline=30s </span><span class="se">\</span><span class="s2"> --kube-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi </span><span class="se">\</span><span class="s2"> --kubeconfig=/etc/kubernetes/kubelet.kubeconfig </span><span class="se">\</span><span class="s2"> --max-pods=100 </span><span class="se">\</span><span class="s2"> --minimum-image-ttl-duration=720h0m0s </span><span class="se">\</span><span class="s2"> --node-labels=node.kubernetes.io/k8s-node=true </span><span class="se">\</span><span class="s2"> --pod-infra-container-image=gcr.azk8s.cn/google_containers/pause-amd64:3.1 </span><span class="se">\</span><span class="s2"> --port=10250 </span><span class="se">\</span><span class="s2"> --read-only-port=0 </span><span class="se">\</span><span class="s2"> --rotate-certificates </span><span class="se">\</span><span class="s2"> --rotate-server-certificates </span><span class="se">\</span><span class="s2"> --resolv-conf=/run/systemd/resolve/resolv.conf </span><span class="se">\</span><span class="s2"> --system-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi </span><span class="se">\</span><span class="s2"> --fail-swap-on=false </span><span class="se">\</span><span class="s2"> --v=2"</span> </code></pre></div></div> <p><strong>当 kubelet 组件设置了 <code class="highlighter-rouge">--rotate-certificates</code>,<code class="highlighter-rouge">--rotate-server-certificates</code> 后会自动更新其使用的相关证书,同时指定 <code class="highlighter-rouge">--authorization-mode=Webhook</code> 后 <code class="highlighter-rouge">10250</code> 端口 RBAC 授权将会委托给 apiserver</strong></p> <ul> <li>proxy</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">###</span> <span class="c"># kubernetes proxy config</span> <span class="c"># default config should be adequate</span> <span class="c"># Add your own!</span> <span class="nv">KUBE_PROXY_ARGS</span><span class="o">=</span><span class="s2">" --bind-address=0.0.0.0 </span><span class="se">\</span><span class="s2"> --cleanup-ipvs=true </span><span class="se">\</span><span class="s2"> --cluster-cidr=10.254.0.0/16 </span><span class="se">\</span><span class="s2"> --hostname-override=docker4.node </span><span class="se">\</span><span class="s2"> --healthz-bind-address=0.0.0.0 </span><span class="se">\</span><span class="s2"> --healthz-port=10256 </span><span class="se">\</span><span class="s2"> --masquerade-all=true </span><span class="se">\</span><span class="s2"> --proxy-mode=ipvs </span><span class="se">\</span><span class="s2"> --ipvs-min-sync-period=5s </span><span class="se">\</span><span class="s2"> --ipvs-sync-period=5s </span><span class="se">\</span><span class="s2"> --ipvs-scheduler=wrr </span><span class="se">\</span><span class="s2"> --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig </span><span class="se">\</span><span class="s2"> --logtostderr=true </span><span class="se">\</span><span class="s2"> --v=2"</span> </code></pre></div></div> <p>由于 <code class="highlighter-rouge">kubelet</code> 组件是采用 TLS Bootstrap 启动,所以需要预先创建相关配置</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 创建用于 tls bootstrap 的 token secret</span> kubectl create <span class="nt">-f</span> bootstrap.secret.yaml <span class="c"># 为了能让 kubelet 实现自动更新证书,需要配置相关 clusterrolebinding</span> <span class="c"># 允许 kubelet tls bootstrap 创建 csr 请求</span> kubectl create clusterrolebinding create-csrs-for-bootstrapping <span class="se">\</span> <span class="nt">--clusterrole</span><span class="o">=</span>system:node-bootstrapper <span class="se">\</span> <span class="nt">--group</span><span class="o">=</span>system:bootstrappers <span class="c"># 自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求</span> kubectl create clusterrolebinding auto-approve-csrs-for-group <span class="se">\</span> <span class="nt">--clusterrole</span><span class="o">=</span>system:certificates.k8s.io:certificatesigningrequests:nodeclient <span class="se">\</span> <span class="nt">--group</span><span class="o">=</span>system:bootstrappers <span class="c"># 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求</span> kubectl create clusterrolebinding auto-approve-renewals-for-nodes <span class="se">\</span> <span class="nt">--clusterrole</span><span class="o">=</span>system:certificates.k8s.io:certificatesigningrequests:selfnodeclient <span class="se">\</span> <span class="nt">--group</span><span class="o">=</span>system:nodes <span class="c"># 在 kubelet server 开启 api 认证的情况下,apiserver 反向访问 kubelet 10250 需要此授权(eg: kubectl logs)</span> kubectl create clusterrolebinding system:kubelet-api-admin <span class="se">\</span> <span class="nt">--clusterrole</span><span class="o">=</span>system:kubelet-api-admin <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span>system:kubelet-api-admin </code></pre></div></div> <h4 id="343nginx-代理">3.4.3、Nginx 代理</h4> <p>为了保证 apiserver 的 HA,需要在每个 node 上部署 nginx 来反向代理(tcp)所有 apiserver;然后 kubelet、kube-proxy 组件连接本地 <code class="highlighter-rouge">127.0.0.1:6443</code> 访问 apiserver,以确保任何 master 挂掉以后 node 都不会受到影响</p> <ul> <li>nginx.conf</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>error_log stderr notice<span class="p">;</span> worker_processes auto<span class="p">;</span> events <span class="o">{</span> multi_accept on<span class="p">;</span> use epoll<span class="p">;</span> worker_connections 1024<span class="p">;</span> <span class="o">}</span> stream <span class="o">{</span> upstream kube_apiserver <span class="o">{</span> least_conn<span class="p">;</span> server 192.168.1.51:6443<span class="p">;</span> server 192.168.1.52:6443<span class="p">;</span> server 192.168.1.53:6443<span class="p">;</span> <span class="o">}</span> server <span class="o">{</span> listen 0.0.0.0:6443<span class="p">;</span> proxy_pass kube_apiserver<span class="p">;</span> proxy_timeout 10m<span class="p">;</span> proxy_connect_timeout 1s<span class="p">;</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <ul> <li>nginx-proxy.service</li> </ul> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>Unit] <span class="nv">Description</span><span class="o">=</span>kubernetes apiserver docker wrapper <span class="nv">Wants</span><span class="o">=</span>docker.socket <span class="nv">After</span><span class="o">=</span>docker.service <span class="o">[</span>Service] <span class="nv">User</span><span class="o">=</span>root <span class="nv">PermissionsStartOnly</span><span class="o">=</span><span class="nb">true </span><span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/docker run <span class="nt">-p</span> 127.0.0.1:6443:6443 <span class="se">\</span> <span class="nt">-v</span> /etc/nginx:/etc/nginx <span class="se">\</span> <span class="nt">--name</span> nginx-proxy <span class="se">\</span> <span class="nt">--net</span><span class="o">=</span>host <span class="se">\</span> <span class="nt">--restart</span><span class="o">=</span>on-failure:5 <span class="se">\</span> <span class="nt">--memory</span><span class="o">=</span>512M <span class="se">\</span> nginx:1.14.2-alpine <span class="nv">ExecStartPre</span><span class="o">=</span>-/usr/bin/docker <span class="nb">rm</span> <span class="nt">-f</span> nginx-proxy <span class="nv">ExecStop</span><span class="o">=</span>/usr/bin/docker stop nginx-proxy <span class="nv">Restart</span><span class="o">=</span>always <span class="nv">RestartSec</span><span class="o">=</span>15s <span class="nv">TimeoutStartSec</span><span class="o">=</span>30s <span class="o">[</span>Install] <span class="nv">WantedBy</span><span class="o">=</span>multi-user.target </code></pre></div></div> <p>然后在每个 node 上先启动 nginx-proxy,接着启动 kubelet 与 kube-proxy 即可(master 上的 kubelet、kube-proxy 只需要修改 ip 和 node name)</p> <h4 id="344kubelet-server-证书">3.4.4、kubelet server 证书</h4> <p><strong>注意: 新版本 kubelet server 的证书自动签发已经被关闭(看 issue 好像是由于安全原因),所以对于 kubelet server 的证书仍需要手动签署</strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker1.node ➜ ~ kubectl get csr NAME AGE REQUESTOR CONDITION csr-99l77 10s system:node:docker4.node Pending node-csr-aGwaNKorMc0MZBYOuJsJGCB8Bg8ds97rmE3oKBTV-_E 11s system:bootstrap:5d820b Approved,Issued docker1.node ➜ ~ kubectl certificate approve csr-99l77 certificatesigningrequest.certificates.k8s.io/csr-99l77 approved </code></pre></div></div> <h3 id="35部署-calico">3.5、部署 Calico</h3> <p>当 node 全部启动后,由于网络组件(CNI)未安装会显示为 NotReady 状态;下面将部署 Calico 作为网络组件,完成跨节点网络通讯;具体安装文档可以参考 <a href="http://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/calico#installing-with-the-etcd-datastore">Installing with the etcd datastore</a></p> <p>以下为 calico 的配置文件</p> <ul> <li>calico.yaml</li> </ul> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span> <span class="c1"># Source: calico/templates/calico-etcd-secrets.yaml</span> <span class="c1"># The following contains k8s Secrets for use with a TLS enabled etcd cluster.</span> <span class="c1"># For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span> <span class="na">type</span><span class="pi">:</span> <span class="s">Opaque</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-etcd-secrets</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">data</span><span class="pi">:</span> <span class="c1"># Populate the following with etcd TLS configuration if desired, but leave blank if</span> <span class="c1"># not using TLS for etcd.</span> <span class="c1"># The keys below should be uncommented and the values populated with the base64</span> <span class="c1"># encoded contents of each file that would be associated with the TLS data.</span> <span class="c1"># Example command for encoding a file contents: cat &lt;file&gt; | base64 -w 0</span> <span class="na">etcd-key</span><span class="pi">:</span> <span class="s">LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdGtOVlV5QWtOOWxDKy9EbzlCRkt0em5IZlFJKzJMK2crclkwLzNoOExJTEFoWUtXCm1XdVNNQUFjbyt4clVtaTFlUGIzcmRKR0p1NEhmRXFmalYvakhvN0haOGxteXd0S29Ed254aU9jZDRlRXltcXEKTEFVYzZ5RWU4dXFGZ2pLVHE4SjV2Z1F1cGp0ZlZnZXRPdVVsWWtWbUNKMWtpUW0yVk5WRnRWZ0Fqck1xSy9POApJTXN6RWRYU3BDc1Zwb0kzaUpoVHJSRng4ZzRXc2hwNG1XMzhMWDVJYVVoMWZaSGVMWm1sRURpclBWMGRTNmFWCmJscUk2aUFwanVBc3hYWjFlVTdmOVZWK01PVmNVc3A4cDAxNmJzS3R6VTJGSnB6ZlM3c1BlbGpKZGgzZmVOdk8KRVl1aDlsU0c1VGNKUHBuTTZ0R0ppaHpEWCt4dnNGa3d5MVJSVlFJREFRQUJBb0lCQUYwRXVqd2xVRGFzakJJVwpubDFKb2U4bTd0ZXUyTEk0QW9sUmluVERZZVE1aXRYWWt0R1Q0OVRaaWNSak9WYWlsOU0zZjZwWGdYUUcwUTB1CjdJVHpaZTlIZ1I5SDIwMU80dlFxSDBaeEVENjBqQ0hlRkNGSkxyd1ZlRDBUVWJYajZCZWx0Z296Q2pmT1gxYUIKcm5nN1VEdjZIUnZTYitlOGJEQ1pjKzBjRDVURG4vUWV0R1dtUmpJZ1FhMmlUT2MzSzFiaHo2RTl5Nk9qWkFTMQpiai9NL1dOd20yNHRxQTJEeWdjcGVmUGFnTWtFNm9uYXBFVHhZdi83QmNqcUhtdVd6WE1wMzd6VGpPckwxVDdmClhrbHdFMUYrMDRhRDR6dDZycEdmN0lqSUdvRkEvT2ZrRGZiYkRjN2NsaDJ1SkNMTVE5MGpuSkxMTGRSV3dQRW4KMkkyY3IvVUNnWUVBN3BjT29VV3RwdDJjWGIzSnl3Tkh4aXl1bEc4V1JENjBlQ1MrUXFnQUZndU5JWFJlMEREUwovSWY0M1BhaVB3TjhBS216ZTRKbGsxM3Rnd29qdi9RWVFVblJzZi9PbnpUUlFoWVJXT2lxSE5lSmFvOUxFU0VDClcxNXNmUjhnYzd0dFdPZ0loZkhudmdCR0QvYmUzS1NWVjdUY0lndVVjV3RzeHhLdjZ4LzJNdHNDZ1lFQXc1QVIKWk9HNUp4UGVNV3FVRUR3QjJuQmt6WEtGblpNSEJXV2FOeHpEaTI0NmZEVWM2T1hSTTJJanh2cmVkc3JKQjBXMwovelNDeFdUbkRmL3RJY1lKMjRuTmNsMUNDS2hTNVE5bVZxanZ3dE1SaEF1Uk5VSFJSVjZLNS91V1hHQzAzekR3CkEvMUFSd3lZSHNHTlJVOFRNNnpNRFcxL0x5djZNZ2pnOFBIamk0OENnWUVBa3JwelZOcjFJRm5KZ0J6bnJPSW4Ka2NpSTFPQThZVnZ1d0xSWURjWWp4MnJ6TUUvUXYxaEhhT1oyTmUyM2VlazZxVzJ6NDVFZHhyTk5EZmwrWXQ1Swp6RndKaWQ0M3c5RkhuOHpTZmtzWDB3VDZqWDN5UEdhQWZKQmxSODJNdDUvY2I0RERQUnkzMkRGeTVQNTlzRlBIClJGa0Z5Q28yOEVtUWJCMGg4d2VFOFdFQ2dZQm1IeUptS3RWVUNiVDYyeXZzZWxtQlp6WE1ieVJGRDlVWHhXSE4KcTlDVlMvOXdndy9Rc3NvVzZnWEN6NWhDTWt6ZDVsTmFDbUxMajVCMHFCTjlrbnZ0VDcyZ0hnRHdvbTEvUGhaego1STRuajY3UzVITjBleVU3ODAzWUxISHRWWGErSWtFRDVFaWZrWDBTZW9JNkVqdjF2U05sVTZ1WngzNUVpSXhtClpmb3NFd0tCZ0dQMmpsK0lPcFV5Y2NEL25EbUJWa05CWHoydWhncU8yYjE4d0hSOGdiSXoyVTRBZnpreXVkWUcKZzQvRjJZZVdCSEdNeTc5N0I2c0hjQTdQUWNNdUFuRk11MG9UNkMvanpDSHpoK2VaaS8wdHJRTHJGeWFFaGVuWgpnazduUTdHNHhROWZLZmVTeFcyUlNNUUR0MTZULzNOTitTOEZCTjJmZEliY3V4QWs0WjVHCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==</span> <span class="na">etcd-cert</span><span class="pi">:</span> <span class="s">LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZFekNDQXZ1Z0F3SUJBZ0lVRGJqcTdVc2ViY2toZXRZb1RPNnRsc1N1c1k0d0RRWUpLb1pJaHZjTkFRRU4KQlFBd2J6RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeERUQUxCZ05WQkFvVEJHVjBZMlF4RmpBVUJnTlZCQXNURFdWMFkyUWdVMlZqZFhKcGRIa3hGVEFUCkJnTlZCQU1UREdWMFkyUXRjbTl2ZEMxallUQWVGdzB4T1RBek1UWXdNelV4TURCYUZ3MHlPVEF6TVRNd016VXgKTURCYU1HY3hDekFKQmdOVkJBWVRBa05PTVJBd0RnWURWUVFJRXdkQ1pXbHFhVzVuTVJBd0RnWURWUVFIRXdkQwpaV2xxYVc1bk1RMHdDd1lEVlFRS0V3UmxkR05rTVJZd0ZBWURWUVFMRXcxbGRHTmtJRk5sWTNWeWFYUjVNUTB3CkN3WURWUVFERXdSbGRHTmtNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXRrTlYKVXlBa045bEMrL0RvOUJGS3R6bkhmUUkrMkwrZytyWTAvM2g4TElMQWhZS1dtV3VTTUFBY28reHJVbWkxZVBiMwpyZEpHSnU0SGZFcWZqVi9qSG83SFo4bG15d3RLb0R3bnhpT2NkNGVFeW1xcUxBVWM2eUVlOHVxRmdqS1RxOEo1CnZnUXVwanRmVmdldE91VWxZa1ZtQ0oxa2lRbTJWTlZGdFZnQWpyTXFLL084SU1zekVkWFNwQ3NWcG9JM2lKaFQKclJGeDhnNFdzaHA0bVczOExYNUlhVWgxZlpIZUxabWxFRGlyUFYwZFM2YVZibHFJNmlBcGp1QXN4WFoxZVU3Zgo5VlYrTU9WY1VzcDhwMDE2YnNLdHpVMkZKcHpmUzdzUGVsakpkaDNmZU52T0VZdWg5bFNHNVRjSlBwbk02dEdKCmloekRYK3h2c0Zrd3kxUlJWUUlEQVFBQm80R3VNSUdyTUE0R0ExVWREd0VCL3dRRUF3SUZvREFkQmdOVkhTVUUKRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEQVlEVlIwVEFRSC9CQUl3QURBZEJnTlZIUTRFRmdRVQpFKzVsWWN1LzhieHJ2WjNvUnRSMmEvOVBJRkF3SHdZRFZSMGpCQmd3Rm9BVTJaVWM3R2hGaG1PQXhzRlZ3VEEyCm5lZFJIdmN3TEFZRFZSMFJCQ1V3STRJSmJHOWpZV3hvYjNOMGh3Ui9BQUFCaHdUQXFBRXpod1RBcUFFMGh3VEEKcUFFMU1BMEdDU3FHU0liM0RRRUJEUVVBQTRJQ0FRQUx3Vkc2QW93cklwZzQvYlRwWndWL0pBUWNLSnJGdm52VApabDVDdzIzNDI4UzJLLzIwaXphaStEWUR1SXIwQ0ZCa2xGOXVsK05ROXZMZ1lqcE0rOTNOY3I0dXhUTVZsRUdZCjloc3NyT1FZZVBGUHhBS1k3RGd0K2RWUGwrWlg4MXNWRzJkU3ZBbm9Kd3dEVWt5U0VUY0g5NkszSlNKS2dXZGsKaTYxN21GYnMrTlcxdngrL0JNN2pVU3ZRUzhRb3JGQVE3SlcwYzZ3R2V4RFEzZExvTXJuR3Vocjd0V0E0WjhwawpPaE12cWdhWUZYSThNUm4yemlLV0R6QXNsa0hGd1RZdWhCNURMSEt0RUVwcWhxbGh1RThwTkZMaVVSV2xQWWhlCmpDNnVKZ0hBZDltcSswd2pyTmxqKzlWaDJoZUJWNldXZEROVTZaR2tpR003RW9YbDM1OWdUTzJPUkNLUk5vZ0YKRVplR25HcjJQNDhKbnZjTnFmZzNPdUtYd24wRDVYYllSWjFuYnR5WG9mMFByUUhEU21wUFVPMWNiZUJjSWVtcQpEVWozK0MrRzBRS1FLQlZDTXJzNXJIVlVWVkJZZzk5ZW1sRE1zUE5TZm9JWDQwTVFCeTdKMnpxRVV5M0sxcGlaCkhwT0lZT1RrWDRhczhqcGYxMnkxSXoxRVZydE1xek83d294VmMwdHRZYWN5NzUrVzZuS1hlWjBaand5aTVYSzUKZGduSVhmZW51RUNlWFNDdWZCSmUxVklzaXVWZ3cyRjlUNk5zRDhnQ3A5SlhTamJ1SXpiM3ArNU9uZzM2ZnBRdQpXZVBCY0dQVXE5cGEwZUtOUGJXNjlDUHdtUTQ2cjg0T3hTTURHWC9CMElqNUtNUnZUMmhPUXBqTVpSblc5OUxFCjRMbUJuUTg1Wmc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==</span> <span class="na">etcd-ca</span><span class="pi">:</span> <span class="s">LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZyakNDQTVhZ0F3SUJBZ0lVWXVIKzIxYlNaU2hncVYxWkx3a2o4RmpZbUl3d0RRWUpLb1pJaHZjTkFRRU4KQlFBd2J6RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeERUQUxCZ05WQkFvVEJHVjBZMlF4RmpBVUJnTlZCQXNURFdWMFkyUWdVMlZqZFhKcGRIa3hGVEFUCkJnTlZCQU1UREdWMFkyUXRjbTl2ZEMxallUQWVGdzB4T1RBek1UWXdNelV4TURCYUZ3MHlPVEF6TVRNd016VXgKTURCYU1HOHhDekFKQmdOVkJBWVRBa05PTVJBd0RnWURWUVFJRXdkQ1pXbHFhVzVuTVJBd0RnWURWUVFIRXdkQwpaV2xxYVc1bk1RMHdDd1lEVlFRS0V3UmxkR05rTVJZd0ZBWURWUVFMRXcxbGRHTmtJRk5sWTNWeWFYUjVNUlV3CkV3WURWUVFERXd4bGRHTmtMWEp2YjNRdFkyRXdnZ0lpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElDRHdBd2dnSUsKQW9JQ0FRRGFLK0s4WStqZkdOY2pOeUloeUhXSE5adWxVZzVKZFpOVU9GOHFXbXJMa0NuY2ZWdVF3dmI4cDFwLwpSSjBFOWo0OFBhZ1RJT3U2TU81R24zejFrZGpHRk9jOVZwMlZjYWJEQzJLWWJvRzdVQ0RmTWkzR1MzUnhUejVkCnh0MG1Ya2liVkMvc01NU2RrRm1mU2FCSXBoKzAyTnMwZURyMzNtUWxTdURlTWozNHJaTkVwMzRnUUk0eElTejAKbXhXR0dWNzcwUE9ScVgrZUthTEpiclp3anFFcnpHMEtEVUlBM0ZuTFdRMnp4b0VwN3JZby9LaGRiOHdETE1kbQp6VXNOZHI0T1F4MFBVRXA4akRUU2lFODkydDQ4KytsOHJ0MW4vTHFRc1FhVncrQlQrMTRvRHdIVkFaRXZ2ZnMwCmZkZ0QvU2RINGJRdHNhT21BdFByQldseU5aMUxIZkR2djMraXFzNk83UXpWUTFCK1c5cFRxdUZ2YUxWN3R1S3UKSXNlUFlseFdjV2E2M0hGbFkxVVJ6M0owaGtrZEZ1dkhUc0dhZDVpaWVrb0dUcFdTN2dVdCtTeWVJT2FhMldHLwp4Y1NiUWE0Y2xiZThuUHV2c1ZFVDhqZ0d0NGVLT25yRVJId0hMb2VleEpsSjdUdnhHNHpOTHZsc2FOL29iRzFDClUzMXczZ2d1SXpzRk5yallsUFdSZ0hSdXdPTlE5anlkM2dqVmNYUFdHTFJISUdYbjNhUDluT3A0OE9WWDhzbXoKOGIwS0V4UVpEQWUyS0tjWEg5a1ZiUFJQSWlLeGpXelV5aDMzQlRNejlPczZHcWM0Zk05c1hxbGRhVzBGd3g4MQpJaklScWx5a3VOSXNDWGhMUzhlNmVtdUNYMTVDZGNKb0ZmdXRuTENvV1B4Umg5OEF4UUlEQVFBQm8wSXdRREFPCkJnTlZIUThCQWY4RUJBTUNBUVl3RHdZRFZSMFRBUUgvQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVMlpVYzdHaEYKaG1PQXhzRlZ3VEEybmVkUkh2Y3dEUVlKS29aSWh2Y05BUUVOQlFBRGdnSUJBSjh3bVNJMVBsQm8zcE1RWC9DOQpRS1RrR0xvVUhGdWprdFoxM1FYeXQ1LzFSeVB2WG1lLy90N3FHR2I5RmJZSm9BYTRTd3JSZkYzZmh3UDZaS0FnCnNYSEliR2gwc014UTdqVmQwMUNMWkoxQmZFNGZtTVlaQUlEWGpTcTNqbHJXZWcxL2hWTFN2dXRuUEFWSXc1SWwKZUdXRTMyOVJ2b2d2dXV6dUsxY2xwZFpIL2p3UlZjUUFUK0xvT2xFZ3Rkd293c0xpaWx3WE95eEZLZDd1UDk3bgozTFZUekFNN3Flell4SUVMQVlUUUN5eTdpeEIxNXlJV1UrUWhreUFtWXJoNEN6VUNNUjQreDlpaGZ6UnlOQkxLCmRBRTdwcjdyUEM4WFQ0YWh2SkJCZTg1THViTVdVRmprcEF5cklQODYyYkFCOCtKSXNFdXNZVGdQakUrMGhteTkKT0NIU2x4Q25GQVdPUXcwQ05Kb3AxWGpHU0RZOXlXL1NNWS83T3B0QlBhT3VWTzVwZTg3VmVXRFFtYmlpdnc3MQo4cFhDQnN6ZWNsdjJZKzdscTRnL0FaQkViVXRvLzV4UXJCbmZGKy9hZFFOQzY4aG4yYzZWa3czYTVDR0ZMN0p2CjhWdFNmeFEzZnFUci9TdzlJbkVKVWpuc0Y3R0xINzZMWXZIU05WeldhMkhiVFNlTnQ0RUlpdlEwb2d0b2hzY0kKSHlrZlpRQ3Z6ZnBSZi9TODFiRDNnU29jQ3NzR2crdVpVU0FMdVhBRDE4RkRXNzg2LzRCckcrMzVLOVBLNktUZwpoWGN4WmRHd3V1RWx0aTRBNWx4OHNrZExPSkZ6TUJPWFJNU2Jsc0dna3pGK2JNRkMrMHV3WW1WK0VTRUdwdy9NCm93WUN1dHh2a3ltL2NOcEk1bjFhanpEcQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==</span> <span class="nn">---</span> <span class="c1"># Source: calico/templates/calico-config.yaml</span> <span class="c1"># This ConfigMap is used to configure a self-hosted Calico installation.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">data</span><span class="pi">:</span> <span class="c1"># Configure this with the location of your etcd cluster.</span> <span class="na">etcd_endpoints</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://192.168.1.51:2379,http://192.168.1.52:2379,http://192.168.1.53:2379"</span> <span class="c1"># If you're using TLS enabled etcd uncomment the following.</span> <span class="c1"># You must also populate the Secret below with these files.</span> <span class="na">etcd_ca</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/calico-secrets/etcd-ca"</span> <span class="na">etcd_cert</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/calico-secrets/etcd-cert"</span> <span class="na">etcd_key</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/calico-secrets/etcd-key"</span> <span class="c1"># Typha is disabled.</span> <span class="na">typha_service_name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">none"</span> <span class="c1"># Configure the Calico backend to use.</span> <span class="na">calico_backend</span><span class="pi">:</span> <span class="s2">"</span><span class="s">bird"</span> <span class="c1"># Configure the MTU to use</span> <span class="na">veth_mtu</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1440"</span> <span class="c1"># The CNI network configuration to install on each node. The special</span> <span class="c1"># values in this config will be automatically populated.</span> <span class="na">cni_network_config</span><span class="pi">:</span> <span class="pi">|-</span> <span class="s">{</span> <span class="s">"name": "k8s-pod-network",</span> <span class="s">"cniVersion": "0.3.0",</span> <span class="s">"plugins": [</span> <span class="s">{</span> <span class="s">"type": "calico",</span> <span class="s">"log_level": "info",</span> <span class="s">"etcd_endpoints": "__ETCD_ENDPOINTS__",</span> <span class="s">"etcd_key_file": "__ETCD_KEY_FILE__",</span> <span class="s">"etcd_cert_file": "__ETCD_CERT_FILE__",</span> <span class="s">"etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",</span> <span class="s">"mtu": __CNI_MTU__,</span> <span class="s">"ipam": {</span> <span class="s">"type": "calico-ipam"</span> <span class="s">},</span> <span class="s">"policy": {</span> <span class="s">"type": "k8s"</span> <span class="s">},</span> <span class="s">"kubernetes": {</span> <span class="s">"kubeconfig": "__KUBECONFIG_FILEPATH__"</span> <span class="s">}</span> <span class="s">},</span> <span class="s">{</span> <span class="s">"type": "portmap",</span> <span class="s">"snat": true,</span> <span class="s">"capabilities": {"portMappings": true}</span> <span class="s">}</span> <span class="s">]</span> <span class="s">}</span> <span class="s">---</span> <span class="c1"># Source: calico/templates/rbac.yaml</span> <span class="c1"># Include a clusterrole for the kube-controllers component,</span> <span class="c1"># and bind it to the calico-kube-controllers serviceaccount.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1beta1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">rules</span><span class="pi">:</span> <span class="c1"># Pods are monitored for changing labels.</span> <span class="c1"># The node controller monitors Kubernetes nodes.</span> <span class="c1"># Namespace and serviceaccount labels are used for policy.</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">pods</span> <span class="pi">-</span> <span class="s">nodes</span> <span class="pi">-</span> <span class="s">namespaces</span> <span class="pi">-</span> <span class="s">serviceaccounts</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">watch</span> <span class="pi">-</span> <span class="s">list</span> <span class="c1"># Watch for changes to Kubernetes NetworkPolicies.</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">networking.k8s.io"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">networkpolicies</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">watch</span> <span class="pi">-</span> <span class="s">list</span> <span class="nn">---</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1beta1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">roleRef</span><span class="pi">:</span> <span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">subjects</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="nn">---</span> <span class="c1"># Include a clusterrole for the calico-node DaemonSet,</span> <span class="c1"># and bind it to the calico-node serviceaccount.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1beta1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">rules</span><span class="pi">:</span> <span class="c1"># The CNI plugin needs to get pods, nodes, and namespaces.</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">pods</span> <span class="pi">-</span> <span class="s">nodes</span> <span class="pi">-</span> <span class="s">namespaces</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">get</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">endpoints</span> <span class="pi">-</span> <span class="s">services</span> <span class="na">verbs</span><span class="pi">:</span> <span class="c1"># Used to discover service IPs for advertisement.</span> <span class="pi">-</span> <span class="s">watch</span> <span class="pi">-</span> <span class="s">list</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">nodes/status</span> <span class="na">verbs</span><span class="pi">:</span> <span class="c1"># Needed for clearing NodeNetworkUnavailable flag.</span> <span class="pi">-</span> <span class="s">patch</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1beta1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">roleRef</span><span class="pi">:</span> <span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">subjects</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="nn">---</span> <span class="nn">---</span> <span class="c1"># Source: calico/templates/calico-node.yaml</span> <span class="c1"># This manifest installs the calico/node container, as well</span> <span class="c1"># as the Calico CNI plugins and network config on</span> <span class="c1"># each master and worker node in a Kubernetes cluster.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">DaemonSet</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">matchLabels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">updateStrategy</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s">RollingUpdate</span> <span class="na">rollingUpdate</span><span class="pi">:</span> <span class="na">maxUnavailable</span><span class="pi">:</span> <span class="m">1</span> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">annotations</span><span class="pi">:</span> <span class="c1"># This, along with the CriticalAddonsOnly toleration below,</span> <span class="c1"># marks the pod as a critical add-on, ensuring it gets</span> <span class="c1"># priority scheduling and that its resources are reserved</span> <span class="c1"># if it ever gets evicted.</span> <span class="s">scheduler.alpha.kubernetes.io/critical-pod</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s">beta.kubernetes.io/os</span><span class="pi">:</span> <span class="s">linux</span> <span class="na">hostNetwork</span><span class="pi">:</span> <span class="no">true</span> <span class="na">tolerations</span><span class="pi">:</span> <span class="c1"># Make sure calico-node gets scheduled on all nodes.</span> <span class="pi">-</span> <span class="na">effect</span><span class="pi">:</span> <span class="s">NoSchedule</span> <span class="na">operator</span><span class="pi">:</span> <span class="s">Exists</span> <span class="c1"># Mark the pod as a critical add-on for rescheduling.</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">CriticalAddonsOnly</span> <span class="na">operator</span><span class="pi">:</span> <span class="s">Exists</span> <span class="pi">-</span> <span class="na">effect</span><span class="pi">:</span> <span class="s">NoExecute</span> <span class="na">operator</span><span class="pi">:</span> <span class="s">Exists</span> <span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="c1"># Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force</span> <span class="c1"># deletion": http://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.</span> <span class="na">terminationGracePeriodSeconds</span><span class="pi">:</span> <span class="m">0</span> <span class="na">initContainers</span><span class="pi">:</span> <span class="c1"># This container installs the Calico CNI binaries</span> <span class="c1"># and CNI network config file on each node.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">install-cni</span> <span class="na">image</span><span class="pi">:</span> <span class="s">calico/cni:v3.6.0</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">/install-cni.sh"</span><span class="pi">]</span> <span class="na">env</span><span class="pi">:</span> <span class="c1"># Name of the CNI config file to create.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CNI_CONF_NAME</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10-calico.conflist"</span> <span class="c1"># The CNI network config to install on each node.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CNI_NETWORK_CONFIG</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">cni_network_config</span> <span class="c1"># The location of the Calico etcd cluster.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_ENDPOINTS</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_endpoints</span> <span class="c1"># CNI MTU Config variable</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CNI_MTU</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">veth_mtu</span> <span class="c1"># Prevents the container from sleeping forever.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">SLEEP</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">false"</span> <span class="na">volumeMounts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/host/opt/cni/bin</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cni-bin-dir</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/host/etc/cni/net.d</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cni-net-dir</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/calico-secrets</span> <span class="na">name</span><span class="pi">:</span> <span class="s">etcd-certs</span> <span class="na">containers</span><span class="pi">:</span> <span class="c1"># Runs calico/node container on each Kubernetes node. This</span> <span class="c1"># container programs network policy and routes on each</span> <span class="c1"># host.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">image</span><span class="pi">:</span> <span class="s">calico/node:v3.6.0</span> <span class="na">env</span><span class="pi">:</span> <span class="c1"># The location of the Calico etcd cluster.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_ENDPOINTS</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_endpoints</span> <span class="c1"># Location of the CA certificate for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_CA_CERT_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_ca</span> <span class="c1"># Location of the client key for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_KEY_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_key</span> <span class="c1"># Location of the client certificate for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_CERT_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_cert</span> <span class="c1"># Set noderef for node controller.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CALICO_K8S_NODE_REF</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">fieldRef</span><span class="pi">:</span> <span class="na">fieldPath</span><span class="pi">:</span> <span class="s">spec.nodeName</span> <span class="c1"># Choose the backend to use.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CALICO_NETWORKING_BACKEND</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">calico_backend</span> <span class="c1"># Cluster type to identify the deployment type</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CLUSTER_TYPE</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">k8s,bgp"</span> <span class="c1"># Auto-detect the BGP IP address.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">IP</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">autodetect"</span> <span class="c1"># Enable IPIP</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CALICO_IPV4POOL_IPIP</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Always"</span> <span class="c1"># Set MTU for tunnel device used if ipip is enabled</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FELIX_IPINIPMTU</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">veth_mtu</span> <span class="c1"># The default IPv4 pool to create on startup if none exists. Pod IPs will be</span> <span class="c1"># chosen from this range. Changing this value after installation will have</span> <span class="c1"># no effect. This should fall within `--cluster-cidr`.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CALICO_IPV4POOL_CIDR</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10.20.0.0/16"</span> <span class="c1"># Disable file logging so `kubectl logs` works.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">CALICO_DISABLE_FILE_LOGGING</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="c1"># Set Felix endpoint to host default action to ACCEPT.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FELIX_DEFAULTENDPOINTTOHOSTACTION</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ACCEPT"</span> <span class="c1"># Disable IPv6 on Kubernetes.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FELIX_IPV6SUPPORT</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">false"</span> <span class="c1"># Set Felix logging to "info"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FELIX_LOGSEVERITYSCREEN</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">info"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">FELIX_HEALTHENABLED</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">IP_AUTODETECTION_METHOD</span> <span class="na">value</span><span class="pi">:</span> <span class="s">can-reach=192.168.1.51</span> <span class="na">securityContext</span><span class="pi">:</span> <span class="na">privileged</span><span class="pi">:</span> <span class="no">true</span> <span class="na">resources</span><span class="pi">:</span> <span class="na">requests</span><span class="pi">:</span> <span class="na">cpu</span><span class="pi">:</span> <span class="s">250m</span> <span class="na">livenessProbe</span><span class="pi">:</span> <span class="na">httpGet</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/liveness</span> <span class="na">port</span><span class="pi">:</span> <span class="m">9099</span> <span class="na">host</span><span class="pi">:</span> <span class="s">localhost</span> <span class="na">periodSeconds</span><span class="pi">:</span> <span class="m">10</span> <span class="na">initialDelaySeconds</span><span class="pi">:</span> <span class="m">10</span> <span class="na">failureThreshold</span><span class="pi">:</span> <span class="m">6</span> <span class="na">readinessProbe</span><span class="pi">:</span> <span class="na">exec</span><span class="pi">:</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">/bin/calico-node</span> <span class="pi">-</span> <span class="s">-bird-ready</span> <span class="pi">-</span> <span class="s">-felix-ready</span> <span class="na">periodSeconds</span><span class="pi">:</span> <span class="m">10</span> <span class="na">volumeMounts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/lib/modules</span> <span class="na">name</span><span class="pi">:</span> <span class="s">lib-modules</span> <span class="na">readOnly</span><span class="pi">:</span> <span class="no">true</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/run/xtables.lock</span> <span class="na">name</span><span class="pi">:</span> <span class="s">xtables-lock</span> <span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/var/run/calico</span> <span class="na">name</span><span class="pi">:</span> <span class="s">var-run-calico</span> <span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/var/lib/calico</span> <span class="na">name</span><span class="pi">:</span> <span class="s">var-lib-calico</span> <span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/calico-secrets</span> <span class="na">name</span><span class="pi">:</span> <span class="s">etcd-certs</span> <span class="na">volumes</span><span class="pi">:</span> <span class="c1"># Used by calico/node.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">lib-modules</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/lib/modules</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">var-run-calico</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/var/run/calico</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">var-lib-calico</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/var/lib/calico</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">xtables-lock</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/run/xtables.lock</span> <span class="na">type</span><span class="pi">:</span> <span class="s">FileOrCreate</span> <span class="c1"># Used to install CNI.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cni-bin-dir</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/opt/cni/bin</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cni-net-dir</span> <span class="na">hostPath</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/etc/cni/net.d</span> <span class="c1"># Mount in the etcd TLS secrets with mode 400.</span> <span class="c1"># See http://kubernetes.io/docs/concepts/configuration/secret/</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">etcd-certs</span> <span class="na">secret</span><span class="pi">:</span> <span class="na">secretName</span><span class="pi">:</span> <span class="s">calico-etcd-secrets</span> <span class="na">defaultMode</span><span class="pi">:</span> <span class="m">0400</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-node</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="nn">---</span> <span class="c1"># Source: calico/templates/calico-kube-controllers.yaml</span> <span class="c1"># This manifest deploys the Calico Kubernetes controllers.</span> <span class="c1"># See http://github.com/projectcalico/kube-controllers</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">extensions/v1beta1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">annotations</span><span class="pi">:</span> <span class="s">scheduler.alpha.kubernetes.io/critical-pod</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span> <span class="na">spec</span><span class="pi">:</span> <span class="c1"># The controllers can only have a single active instance.</span> <span class="na">replicas</span><span class="pi">:</span> <span class="m">1</span> <span class="na">strategy</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s">Recreate</span> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s">beta.kubernetes.io/os</span><span class="pi">:</span> <span class="s">linux</span> <span class="c1"># The controllers must run in the host network namespace so that</span> <span class="c1"># it isn't governed by policy that would prevent it from working.</span> <span class="na">hostNetwork</span><span class="pi">:</span> <span class="no">true</span> <span class="na">tolerations</span><span class="pi">:</span> <span class="c1"># Mark the pod as a critical add-on for rescheduling.</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">CriticalAddonsOnly</span> <span class="na">operator</span><span class="pi">:</span> <span class="s">Exists</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">node-role.kubernetes.io/master</span> <span class="na">effect</span><span class="pi">:</span> <span class="s">NoSchedule</span> <span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">image</span><span class="pi">:</span> <span class="s">calico/kube-controllers:v3.6.0</span> <span class="na">env</span><span class="pi">:</span> <span class="c1"># The location of the Calico etcd cluster.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_ENDPOINTS</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_endpoints</span> <span class="c1"># Location of the CA certificate for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_CA_CERT_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_ca</span> <span class="c1"># Location of the client key for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_KEY_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_key</span> <span class="c1"># Location of the client certificate for etcd.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ETCD_CERT_FILE</span> <span class="na">valueFrom</span><span class="pi">:</span> <span class="na">configMapKeyRef</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-config</span> <span class="na">key</span><span class="pi">:</span> <span class="s">etcd_cert</span> <span class="c1"># Choose which controllers to run.</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">ENABLED_CONTROLLERS</span> <span class="na">value</span><span class="pi">:</span> <span class="s">policy,namespace,serviceaccount,workloadendpoint,node</span> <span class="na">volumeMounts</span><span class="pi">:</span> <span class="c1"># Mount in the etcd TLS secrets.</span> <span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/calico-secrets</span> <span class="na">name</span><span class="pi">:</span> <span class="s">etcd-certs</span> <span class="na">readinessProbe</span><span class="pi">:</span> <span class="na">exec</span><span class="pi">:</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">/usr/bin/check-status</span> <span class="pi">-</span> <span class="s">-r</span> <span class="na">volumes</span><span class="pi">:</span> <span class="c1"># Mount in the etcd TLS secrets with mode 400.</span> <span class="c1"># See http://kubernetes.io/docs/concepts/configuration/secret/</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">etcd-certs</span> <span class="na">secret</span><span class="pi">:</span> <span class="na">secretName</span><span class="pi">:</span> <span class="s">calico-etcd-secrets</span> <span class="na">defaultMode</span><span class="pi">:</span> <span class="m">0400</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">calico-kube-controllers</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> </code></pre></div></div> <p><strong>需要注意的是我们添加了 <code class="highlighter-rouge">IP_AUTODETECTION_METHOD</code> 变量,这个变量会设置 calcio 获取 node ip 的方式;默认情况下采用 <a href="http://docs.projectcalico.org/v3.6/reference/node/configuration#ip-autodetection-methods">first-found</a> 方式获取,即获取第一个有效网卡的 IP 作为 node ip;在某些多网卡机器上可能会出现问题;这里将值设置为 <code class="highlighter-rouge">can-reach=192.168.1.51</code>,即使用第一个能够访问 master <code class="highlighter-rouge">192.168.1.51</code> 的网卡地址作为 node ip</strong></p> <p>最后执行创建即可,创建成功后如下所示</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker1.node ➜ ~ kubectl get pod <span class="nt">-o</span> wide <span class="nt">-n</span> kube-system NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-65bc6b9f9-cn27f 1/1 Running 0 85s 192.168.1.53 docker3.node &lt;none&gt; &lt;none&gt; calico-node-c5nl8 1/1 Running 0 85s 192.168.1.53 docker3.node &lt;none&gt; &lt;none&gt; calico-node-fqknv 1/1 Running 0 85s 192.168.1.51 docker1.node &lt;none&gt; &lt;none&gt; calico-node-ldfzs 1/1 Running 0 85s 192.168.1.55 docker5.node &lt;none&gt; &lt;none&gt; calico-node-ngjxc 1/1 Running 0 85s 192.168.1.52 docker2.node &lt;none&gt; &lt;none&gt; calico-node-vj8np 1/1 Running 0 85s 192.168.1.54 docker4.node &lt;none&gt; &lt;none&gt; </code></pre></div></div> <p>此时所有 node 应当变为 Ready 状态</p> <h3 id="35部署-dns">3.5、部署 DNS</h3> <p>其他组件全部完成后我们应当部署集群 DNS 使 service 等能够正常解析;集群 DNS 这里采用 coredns,具体安装文档参考 <a href="http://github.com/coredns/deployment/tree/master/kubernetes">coredns/deployment</a>;coredns 完整配置如下</p> <ul> <li>coredns.yaml</li> </ul> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">kubernetes.io/bootstrapping</span><span class="pi">:</span> <span class="s">rbac-defaults</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:coredns</span> <span class="na">rules</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">"</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">endpoints</span> <span class="pi">-</span> <span class="s">services</span> <span class="pi">-</span> <span class="s">pods</span> <span class="pi">-</span> <span class="s">namespaces</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">list</span> <span class="pi">-</span> <span class="s">watch</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">"</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">nodes</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">get</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">annotations</span><span class="pi">:</span> <span class="s">rbac.authorization.kubernetes.io/autoupdate</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">kubernetes.io/bootstrapping</span><span class="pi">:</span> <span class="s">rbac-defaults</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:coredns</span> <span class="na">roleRef</span><span class="pi">:</span> <span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:coredns</span> <span class="na">subjects</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ConfigMap</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">data</span><span class="pi">:</span> <span class="na">Corefile</span><span class="pi">:</span> <span class="pi">|</span> <span class="s">.:53 {</span> <span class="s">errors</span> <span class="s">health</span> <span class="s">kubernetes cluster.local in-addr.arpa ip6.arpa {</span> <span class="s">pods insecure</span> <span class="s">upstream</span> <span class="s">fallthrough in-addr.arpa ip6.arpa</span> <span class="s">}</span> <span class="s">prometheus :9153</span> <span class="s">forward . /etc/resolv.conf</span> <span class="s">cache 30</span> <span class="s">loop</span> <span class="s">reload</span> <span class="s">loadbalance</span> <span class="s">}</span> <span class="s">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="s">kubernetes.io/name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">CoreDNS"</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">replicas</span><span class="pi">:</span> <span class="m">2</span> <span class="na">strategy</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="s">RollingUpdate</span> <span class="na">rollingUpdate</span><span class="pi">:</span> <span class="na">maxUnavailable</span><span class="pi">:</span> <span class="m">1</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">matchLabels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">priorityClassName</span><span class="pi">:</span> <span class="s">system-cluster-critical</span> <span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">tolerations</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s2">"</span><span class="s">CriticalAddonsOnly"</span> <span class="na">operator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Exists"</span> <span class="na">nodeSelector</span><span class="pi">:</span> <span class="s">beta.kubernetes.io/os</span><span class="pi">:</span> <span class="s">linux</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">image</span><span class="pi">:</span> <span class="s">coredns/coredns:1.3.1</span> <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">IfNotPresent</span> <span class="na">resources</span><span class="pi">:</span> <span class="na">limits</span><span class="pi">:</span> <span class="na">memory</span><span class="pi">:</span> <span class="s">170Mi</span> <span class="na">requests</span><span class="pi">:</span> <span class="na">cpu</span><span class="pi">:</span> <span class="s">100m</span> <span class="na">memory</span><span class="pi">:</span> <span class="s">70Mi</span> <span class="na">args</span><span class="pi">:</span> <span class="pi">[</span> <span class="s2">"</span><span class="s">-conf"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">/etc/coredns/Corefile"</span> <span class="pi">]</span> <span class="na">volumeMounts</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">config-volume</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/etc/coredns</span> <span class="na">readOnly</span><span class="pi">:</span> <span class="no">true</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">53</span> <span class="na">name</span><span class="pi">:</span> <span class="s">dns</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">UDP</span> <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">53</span> <span class="na">name</span><span class="pi">:</span> <span class="s">dns-tcp</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">9153</span> <span class="na">name</span><span class="pi">:</span> <span class="s">metrics</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="na">securityContext</span><span class="pi">:</span> <span class="na">allowPrivilegeEscalation</span><span class="pi">:</span> <span class="no">false</span> <span class="na">capabilities</span><span class="pi">:</span> <span class="na">add</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">NET_BIND_SERVICE</span> <span class="na">drop</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">all</span> <span class="na">readOnlyRootFilesystem</span><span class="pi">:</span> <span class="no">true</span> <span class="na">livenessProbe</span><span class="pi">:</span> <span class="na">httpGet</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/health</span> <span class="na">port</span><span class="pi">:</span> <span class="m">8080</span> <span class="na">scheme</span><span class="pi">:</span> <span class="s">HTTP</span> <span class="na">initialDelaySeconds</span><span class="pi">:</span> <span class="m">60</span> <span class="na">timeoutSeconds</span><span class="pi">:</span> <span class="m">5</span> <span class="na">successThreshold</span><span class="pi">:</span> <span class="m">1</span> <span class="na">failureThreshold</span><span class="pi">:</span> <span class="m">5</span> <span class="na">readinessProbe</span><span class="pi">:</span> <span class="na">httpGet</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/health</span> <span class="na">port</span><span class="pi">:</span> <span class="m">8080</span> <span class="na">scheme</span><span class="pi">:</span> <span class="s">HTTP</span> <span class="na">dnsPolicy</span><span class="pi">:</span> <span class="s">Default</span> <span class="na">volumes</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">config-volume</span> <span class="na">configMap</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">coredns</span> <span class="na">items</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s">Corefile</span> <span class="na">path</span><span class="pi">:</span> <span class="s">Corefile</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">annotations</span><span class="pi">:</span> <span class="s">prometheus.io/port</span><span class="pi">:</span> <span class="s2">"</span><span class="s">9153"</span> <span class="s">prometheus.io/scrape</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="s">kubernetes.io/cluster-service</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="s">kubernetes.io/name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">CoreDNS"</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns</span> <span class="na">clusterIP</span><span class="pi">:</span> <span class="s">10.254.0.2</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">dns</span> <span class="na">port</span><span class="pi">:</span> <span class="m">53</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">UDP</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">dns-tcp</span> <span class="na">port</span><span class="pi">:</span> <span class="m">53</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">metrics</span> <span class="na">port</span><span class="pi">:</span> <span class="m">9153</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> </code></pre></div></div> <h3 id="35部署-dns-自动扩容">3.5、部署 DNS 自动扩容</h3> <p>在大规模集群的情况下,可能需要集群 DNS 自动扩容,具体文档请参考 <a href="http://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns-horizontal-autoscaler">DNS Horizontal Autoscaler</a>,DNS 扩容算法可参考 <a href="http://github.com/kubernetes-incubator/cluster-proportional-autoscaler/">Github</a>,如有需要请自行修改;以下为具体配置</p> <ul> <li>dns-horizontal-autoscaler.yaml</li> </ul> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Copyright 2016 The Kubernetes Authors.</span> <span class="c1">#</span> <span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span> <span class="c1"># you may not use this file except in compliance with the License.</span> <span class="c1"># You may obtain a copy of the License at</span> <span class="c1">#</span> <span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span> <span class="c1">#</span> <span class="c1"># Unless required by applicable law or agreed to in writing, software</span> <span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span> <span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span> <span class="c1"># See the License for the specific language governing permissions and</span> <span class="c1"># limitations under the License.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">addonmanager.kubernetes.io/mode</span><span class="pi">:</span> <span class="s">Reconcile</span> <span class="nn">---</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:kube-dns-autoscaler</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">addonmanager.kubernetes.io/mode</span><span class="pi">:</span> <span class="s">Reconcile</span> <span class="na">rules</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">nodes"</span><span class="pi">]</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">list"</span><span class="pi">]</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">replicationcontrollers/scale"</span><span class="pi">]</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">get"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">update"</span><span class="pi">]</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">extensions"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">deployments/scale"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">replicasets/scale"</span><span class="pi">]</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">get"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">update"</span><span class="pi">]</span> <span class="c1"># Remove the configmaps rule once below issue is fixed:</span> <span class="c1"># kubernetes-incubator/cluster-proportional-autoscaler#16</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">configmaps"</span><span class="pi">]</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">get"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">create"</span><span class="pi">]</span> <span class="nn">---</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:kube-dns-autoscaler</span> <span class="na">labels</span><span class="pi">:</span> <span class="s">addonmanager.kubernetes.io/mode</span><span class="pi">:</span> <span class="s">Reconcile</span> <span class="na">subjects</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">roleRef</span><span class="pi">:</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:kube-dns-autoscaler</span> <span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="s">kubernetes.io/cluster-service</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="s">addonmanager.kubernetes.io/mode</span><span class="pi">:</span> <span class="s">Reconcile</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">matchLabels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">k8s-app</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> <span class="na">annotations</span><span class="pi">:</span> <span class="s">scheduler.alpha.kubernetes.io/critical-pod</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">priorityClassName</span><span class="pi">:</span> <span class="s">system-cluster-critical</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">autoscaler</span> <span class="na">image</span><span class="pi">:</span> <span class="s">gcr.azk8s.cn/google_containers/cluster-proportional-autoscaler-amd64:1.1.2-r2</span> <span class="na">resources</span><span class="pi">:</span> <span class="na">requests</span><span class="pi">:</span> <span class="na">cpu</span><span class="pi">:</span> <span class="s2">"</span><span class="s">20m"</span> <span class="na">memory</span><span class="pi">:</span> <span class="s2">"</span><span class="s">10Mi"</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">/cluster-proportional-autoscaler</span> <span class="pi">-</span> <span class="s">--namespace=kube-system</span> <span class="pi">-</span> <span class="s">--configmap=kube-dns-autoscaler</span> <span class="c1"># Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base</span> <span class="pi">-</span> <span class="s">--target=Deployment/coredns</span> <span class="c1"># When cluster is using large nodes(with more cores), "coresPerReplica" should dominate.</span> <span class="c1"># If using small nodes, "nodesPerReplica" should dominate.</span> <span class="pi">-</span> <span class="s">--default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}}</span> <span class="pi">-</span> <span class="s">--logtostderr=true</span> <span class="pi">-</span> <span class="s">--v=2</span> <span class="na">tolerations</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">key</span><span class="pi">:</span> <span class="s2">"</span><span class="s">CriticalAddonsOnly"</span> <span class="na">operator</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Exists"</span> <span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">kube-dns-autoscaler</span> </code></pre></div></div> <h2 id="四其他">四、其他</h2> <h3 id="41集群测试">4.1、集群测试</h3> <p>为测试集群工作正常,我们创建一个 deployment 和一个 service,用于测试联通性和 DNS 工作是否正常;测试配置如下</p> <ul> <li>test.yaml</li> </ul> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">test</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">test</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">replicas</span><span class="pi">:</span> <span class="m">5</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">matchLabels</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">test</span> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">test</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">test</span> <span class="na">image</span><span class="pi">:</span> <span class="s">nginx:1.14.2-alpine</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="m">80</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">test-service</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">selector</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">test</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">nginx</span> <span class="na">port</span><span class="pi">:</span> <span class="m">80</span> <span class="na">nodePort</span><span class="pi">:</span> <span class="m">30001</span> <span class="na">targetPort</span><span class="pi">:</span> <span class="m">80</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="na">type</span><span class="pi">:</span> <span class="s">NodePort</span> </code></pre></div></div> <p>测试方式很简单,进入某一个 pod ping 其他 pod ip 确认网络是否正常,直接访问 service 名称测试 DNS 是否工作正常,这里不再演示</p> <h3 id="42其他说明">4.2、其他说明</h3> <p>此次搭建开启了大部分认证,限于篇幅原因没有将每个选项作用做完整解释,推荐搭建完成后仔细阅读以下 <code class="highlighter-rouge">--help</code> 中的描述(官方文档页面有时候更新不完整);目前 apiserver 仍然保留了 8080 端口(因为直接使用 kubectl 方便),但是在高安全性环境请关闭 8080 端口,因为即使绑定在 127.0.0.1 上,对于任何能够登录 master 机器的用户仍然能够不经验证操作整个集群</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Sat, 16 Mar 2019 17:36:52 +0800 /2019/03/16/set-up-kubernetes-1.13.4-cluster/ /2019/03/16/set-up-kubernetes-1.13.4-cluster/ Kubernetes Kubernetes Kubernetes sample-cli-plugin 源码分析 <blockquote> <p>写这篇文章的目的是为了继续上篇 <a href="/2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/">Kubernetes 1.12 新的插件机制</a> 中最后部分对 <code class="highlighter-rouge">Golang 的插件辅助库</code> 说明;以及为后续使用 Golang 编写自己的 Kubernetes 插件做一个基础铺垫;顺边说一下 <strong>sample-cli-plugin 这个项目是官方为 Golang 开发者编写的一个用于快速切换配置文件中 Namespace 的一个插件样例</strong></p> </blockquote> <h2 id="一基础准备">一、基础准备</h2> <p>在开始分析源码之前,<strong>我们假设读者已经熟悉 Golang 语言,至少对基本语法、指针、依赖管理工具有一定认知</strong>;下面介绍一下 <a href="http://github.com/kubernetes/sample-cli-plugin">sample-cli-plugin</a> 这个项目一些基础核心的依赖:</p> <h3 id="11cobra-终端库">1.1、Cobra 终端库</h3> <p>这是一个强大的 Golang 的 command line interface 库,其支持用非常简单的代码创建出符合 Unix 风格的 cli 程序;甚至官方提供了用于创建 cli 工程脚手架的 cli 命令工具;Cobra 官方 Github 地址 <a href="http://github.com/spf13/cobra">点击这里</a>,具体用法请自行 Google,以下只做一个简单的命令定义介绍(docker、kubernetes 终端 cli 都基于这个库)</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">#</span> <span class="n">每一个命令</span><span class="p">(</span><span class="n">不论是子命令还是主命令</span><span class="p">)</span><span class="n">都会是一个</span> <span class="n">cobra</span><span class="o">.</span><span class="n">Command</span> <span class="n">对象</span> <span class="k">var</span> <span class="n">lsCmd</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">{</span> <span class="c">// 一些命令帮助文档有关的描述信息</span> <span class="n">Use</span><span class="o">:</span> <span class="s">"ls"</span><span class="p">,</span> <span class="n">Short</span><span class="o">:</span> <span class="s">"A brief description of your command"</span><span class="p">,</span> <span class="n">Long</span><span class="o">:</span> <span class="s">`A longer description that spans multiple lines and likely contains examples and usage of using your command. For example: Cobra is a CLI library for Go that empowers applications. This application is a tool to generate the needed files to quickly create a Cobra application.`</span><span class="p">,</span> <span class="c">// 命令运行时真正执行逻辑,如果需要返回 Error 信息,我们一般设置 RunE</span> <span class="n">Run</span><span class="o">:</span> <span class="k">func</span><span class="p">(</span><span class="n">cmd</span> <span class="o">*</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">,</span> <span class="n">args</span> <span class="p">[]</span><span class="kt">string</span><span class="p">)</span> <span class="p">{</span> <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"ls called"</span><span class="p">)</span> <span class="p">},</span> <span class="p">}</span> <span class="c">// 为这个命令添加 flag,比如 `--help`、`-p`</span> <span class="c">// PersistentFlags() 方法添加的 flag 在所有子 command 也会生效</span> <span class="c">// Cobra 的 command 可以无限级联,比如 `kubectl get pod` 就是在 `kubectl` command 下增加了子 `get` command</span> <span class="n">lsCmd</span><span class="o">.</span><span class="n">PersistentFlags</span><span class="p">()</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"foo"</span><span class="p">,</span> <span class="s">""</span><span class="p">,</span> <span class="s">"A help for foo"</span><span class="p">)</span> <span class="c">// Flags() 方法添加的 flag 仅在直接调用此子命令时生效</span> <span class="n">lsCmd</span><span class="o">.</span><span class="n">Flags</span><span class="p">()</span><span class="o">.</span><span class="n">BoolP</span><span class="p">(</span><span class="s">"toggle"</span><span class="p">,</span> <span class="s">"t"</span><span class="p">,</span> <span class="no">false</span><span class="p">,</span> <span class="s">"Help message for toggle"</span><span class="p">)</span> </code></pre></div></div> <h3 id="12vendor-依赖">1.2、vendor 依赖</h3> <p>vendor 目录用于存放 Golang 的依赖库,sample-cli-plugin 这个项目采用 <a href="http://github.com/tools/godep">godep</a> 工具管理依赖;依赖配置信息被保存在 <code class="highlighter-rouge">Godeps/Godeps.json</code> 中,<strong>一般项目不会上传 vendor 目录,因为它的依赖信息已经在 Godeps.json 中存在,只需要在项目下使用 <code class="highlighter-rouge">godep restore</code> 命令恢复就可自动重新下载</strong>;这里上传了 vendor 目录的原因应该是为了方便开发者直接使用 <code class="highlighter-rouge">go get</code> 命令安装;顺边说一下在 Golang 新版本已经开始转换到 <code class="highlighter-rouge">go mod</code> 依赖管理工具,标志就是项目下会有 <code class="highlighter-rouge">go.mod</code> 文件</p> <h2 id="二源码分析">二、源码分析</h2> <h3 id="21环境搭建">2.1、环境搭建</h3> <p>这里准备一笔带过了,基本就是 clone 源码到 <code class="highlighter-rouge">$GOPATH/src/k8s.io/sample-cli-plugin</code> 目录,然后在 GoLand 中打开;目前我使用的 Go 版本为最新的 1.11.4;以下时导入源码后的截图</p> <p><img src="http://cdn.oss.link/markdown/sn8o8.png" alt="GoLand" /></p> <h3 id="22定位核心运行方法">2.2、定位核心运行方法</h3> <p>熟悉过 Cobra 库以后,再从整个项目包名上分析,首先想到的启动入口应该在 <code class="highlighter-rouge">cmd</code> 包下(一般 <code class="highlighter-rouge">cmd</code> 包下的文件都会编译成最终可执行文件名,Kubernetes 也是一样)</p> <p><img src="http://cdn.oss.link/markdown/rafeq.png" alt="main" /></p> <p>从以上截图中可以看出,首先通过 <code class="highlighter-rouge">cmd.NewCmdNamespace</code> 方法创建了一个 Command 对象 <code class="highlighter-rouge">root</code>,然后调用了 <code class="highlighter-rouge">root.Execute</code> 就结束了;那么也就说明 <code class="highlighter-rouge">root</code> 这个 Command 是唯一的核心命令对象,整个插件实现都在这个 <code class="highlighter-rouge">root</code> 里;所以我们需要查看一下这个 <code class="highlighter-rouge">cmd.NewCmdNamespace</code> 是如何对它初始化的,找到 Cobra 中的 <code class="highlighter-rouge">Run</code> 或者 <code class="highlighter-rouge">RunE</code> 设置</p> <p><img src="http://cdn.oss.link/markdown/77krg.png" alt="NewCmdNamespace" /></p> <p>定位到 <code class="highlighter-rouge">NewCmdNamespace</code> 方法以后,基本上就是标准的 Cobra 库的使用方式了;<strong>从截图上可以看到,<code class="highlighter-rouge">RunE</code> 设置的函数总共运行了 3 个动作: <code class="highlighter-rouge">o.Complete</code>、<code class="highlighter-rouge">o.Validate</code>、<code class="highlighter-rouge">o.Run</code></strong>;所以接下来我们主要分析这三个方法就行了</p> <h3 id="23namespaceoptions-结构体">2.3、NamespaceOptions 结构体</h3> <p>在分析上面说的这三个方法之前,我们还应当了解一下这个 <code class="highlighter-rouge">o</code> 是什么玩意</p> <p><img src="http://cdn.oss.link/markdown/4b3cc.png" alt="NamespaceOptions" /></p> <p>从源码中可以看到,<code class="highlighter-rouge">o</code> 这个对象由 <code class="highlighter-rouge">NewNamespaceOptions</code> 创建,而 <code class="highlighter-rouge">NewNamespaceOptions</code> 方法返回的实际上是一个 <code class="highlighter-rouge">NamespaceOptions</code> 结构体;接下来我们需要研究一下这个结构体都是由什么组成的,换句话说要基本大致上整明白结构体的基本结构,比如里面的属性都是干啥的</p> <h4 id="231genericclioptionsconfigflags">2.3.1、*genericclioptions.ConfigFlags</h4> <p>首先看下第一个属性 <code class="highlighter-rouge">configFlags</code>,它的实际类型是 <code class="highlighter-rouge">*genericclioptions.ConfigFlags</code>,点击查看以后如下</p> <p><img src="http://cdn.oss.link/markdown/li6s4.png" alt="genericclioptions.ConfigFlags" /></p> <p>从这些字段上来看,我们可以暂且模糊的推测出这应该是个基础配置型的字段,负责存储一些全局基本设置,比如 API Server 认证信息等</p> <h4 id="232apicontext">2.3.2、*api.Context</h4> <p>下面这两个 <code class="highlighter-rouge">resultingContext</code>、<code class="highlighter-rouge">resultingContextName</code> 就很好理解了,从名字上看就可以知道它们应该是用来存储结果集的 Context 信息的;当然这个 <code class="highlighter-rouge">*api.Context</code> 就是 Kubernetes 配置文件中 Context 的 Go 结构体</p> <h4 id="233userspecified">2.3.3、userSpecified*</h4> <p>这几个字段从名字上就可以区分出,他们应该用于存储用户设置的或者说是通过命令行选项输入的一些指定配置信息,比如 Cluster、Context 等</p> <h4 id="234rawconfig">2.3.4、rawConfig</h4> <p>rawConfig 这个变量名字有点子奇怪,不过它实际上是个 <code class="highlighter-rouge">api.Config</code>;里面保存了与 API Server 通讯的配置信息;<strong>至于为什么要有这玩意,是因为配置信息输入源有两个: cli 命令行选项(eg: <code class="highlighter-rouge">--namespace</code>)和用户配置文件(eg: <code class="highlighter-rouge">~/.kube/config</code>);最终这两个地方的配置合并后会存储在这个 rawConfig 里</strong></p> <h4 id="235listnamespaces">2.3.5、listNamespaces</h4> <p>这个变量实际上相当于一个 flag,用于存储插件是否使用了 <code class="highlighter-rouge">--list</code> 选项;在分析结构体这里没法看出来;不过只要稍稍的多看一眼代码就能看在 <code class="highlighter-rouge">NewCmdNamespace</code> 方法中有这么一行代码</p> <p><img src="http://cdn.oss.link/markdown/f07l3.png" alt="listNamespaces" /></p> <h3 id="24核心处理逻辑">2.4、核心处理逻辑</h3> <p>介绍完了结构体的基本属性,最后我们只需要弄明白在核心 Command 方法内运行的这三个核心方法就行了</p> <p><img src="http://cdn.oss.link/markdown/8lm4b.png" alt="core func" /></p> <h4 id="241namespaceoptionscomplete">2.4.1、*NamespaceOptions.Complete</h4> <p>这个方法代码稍微有点多,这里不会对每一行代码都做解释,只要大体明白都在干什么就行了;我们的目的是理解它,后续模仿它创造自己的插件;下面是代码截图</p> <p><img src="http://cdn.oss.link/markdown/qqf0f.png" alt="NamespaceOptions.Complete" /></p> <p>从截图上可以看到,首先弄出了 <code class="highlighter-rouge">rawConfig</code> 这个玩意,<code class="highlighter-rouge">rawConfig</code> 上面也提到了,它就是终端选项和用户配置文件的最终合并,至于为什么可以查看 <code class="highlighter-rouge">ToRawKubeConfigLoader().RawConfig()</code> 这两个方法的注释和实现即可;</p> <p>接下来就是各种获取插件执行所需要的变量信息,比如获取用户指定的 <code class="highlighter-rouge">Namespace</code>、<code class="highlighter-rouge">Cluster</code>、<code class="highlighter-rouge">Context</code> 等,其中还包含了一些必要的校验;比如不允许使用 <code class="highlighter-rouge">kubectl ns NS_NAME1 --namespace NS_NAME2</code> 这种操作(因为这么干很让人难以理解 “你到底是要切换到 <code class="highlighter-rouge">NS_NAME1</code> 还是 <code class="highlighter-rouge">NS_NAME2</code>”)</p> <p>最后从 <code class="highlighter-rouge">153</code> 行 <code class="highlighter-rouge">o.resultingContext = api.NewContext()</code> 开始就是创建最终的 <code class="highlighter-rouge">resultingContext</code> 对象,把获取到的用户指定的 <code class="highlighter-rouge">Namespace</code> 等各种信息赋值好,为下一步将其持久化到配置文件中做准备</p> <h4 id="242namespaceoptionsvalidate">2.4.2、*NamespaceOptions.Validate</h4> <p>这个方法看名字就知道,里面全是对最终结果的校验;比如检查一下 <code class="highlighter-rouge">rawConfig</code> 中的 <code class="highlighter-rouge">CurrentContext</code> 是否获取到了,看看命令行参数是否正确,确保你不会瞎鸡儿输入 <code class="highlighter-rouge">kubectl ns NS_NAME1 NS_NAME2</code> 这种命令</p> <p><img src="http://cdn.oss.link/markdown/frqpb.png" alt="NamespaceOptions.Validate" /></p> <h4 id="243namespaceoptionsrun">2.4.3、*NamespaceOptions.Run</h4> <p>第一步合并配置信息并获取到用户设置(输入)的配置,第二部做参数校验;可以说前面的两步操作都是为这一步做准备,<code class="highlighter-rouge">Run</code> 方法真正的做了配置文件写入、终端返回结果打印操作</p> <p><img src="http://cdn.oss.link/markdown/6tkjz.png" alt="NamespaceOptions.Run" /></p> <p>可以看到,<code class="highlighter-rouge">Run</code> 方法第一步就是更加谨慎的检查了一下参数是否正常,然后调用了 <code class="highlighter-rouge">o.setNamespace</code>;这个方法截图如下</p> <p><img src="http://cdn.oss.link/markdown/1jc3k.png" alt="NamespaceOptions.setNamespace" /></p> <p>这个 <code class="highlighter-rouge">setNamespace</code>是真正的做了配置文件写入动作的,实际写入方法就是 <code class="highlighter-rouge">clientcmd.ModifyConfig</code>;这个是 <code class="highlighter-rouge">Kubernetes</code> <code class="highlighter-rouge">client-go</code> 提供的方法,这些库的作用就是提供给我们非常方便的 API 操作;比如修改配置文件,你不需要关心配置文件在哪,你更不需要关系文件句柄是否被释放</p> <p>从 <code class="highlighter-rouge">o.setNamespace</code> 方法以后其实就没什么看头了,毕竟插件的核心功能就是快速修改 <code class="highlighter-rouge">Namespace</code>;下面的各种 <code class="highlighter-rouge">for</code> 循环遍历其实就是在做打印输出;比如当你没有设置 <code class="highlighter-rouge">Namespace</code> 而使用了 <code class="highlighter-rouge">--list</code> 选项,插件就通过这里帮你打印设置过那些 <code class="highlighter-rouge">Namespace</code></p> <h2 id="三插件总结">三、插件总结</h2> <p>分析完了这个官方的插件,然后想一下自己以后写插件可能的需求,最后对比一下,可以为以后写插件做个总结:</p> <ul> <li>我们最好也弄个 <code class="highlighter-rouge">xxxOptions</code> 这种结构体存存一些配置</li> <li>结构体内至少我们应当存储 <code class="highlighter-rouge">configFlags</code>、<code class="highlighter-rouge">rawConfig</code> 这两个基础配置信息</li> <li>结构体内其它参数都应当是跟自己实际业务有关的</li> <li>最后在在结构体上增加适当的方法完成自己的业务逻辑并保持好适当的校验</li> </ul> <p>转载请注明出n,本文采用 [CC4.0](http://c 1.12 新的插件机制](/2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/) 中最后部分对 <code class="highlighter-rouge">Golang 的插件辅助库</code> 说明;以及为后续使用 Golang 编写自己的 Kubernetes 插件做一个基础铺垫;顺边说一下 <strong>sample-cli-plugin 这个项目是官方为 Golang 开发者编写的一个用于快速切换配置文件中 Namespace 的一个插件样例</strong></p> Wed, 16 Jan 2019 12:16:42 +0800 /2019/01/16/understand-kubernetes-sample-cli-plugin-source-code/ /2019/01/16/understand-kubernetes-sample-cli-plugin-source-code/ Kubernetes Kubernetes Kubernetes 1.12 新的插件机制 <blockquote> <p>在很久以前的版本研究过 kubernetes 的插件机制,当时弄了一个快速切换 <code class="highlighter-rouge">namespace</code> 的小插件;最近把自己本机的 kubectl 升级到了 1.12,突然发现插件不能用了;撸了一下文档发现插件机制彻底改了…</p> </blockquote> <h2 id="一插件编写语言">一、插件编写语言</h2> <p>kubernetes 1.12 新的插件机制在编写语言上同以前一样,<strong>可以以任意语言编写,只要能弄一个可执行的文件出来就行</strong>,插件可以是一个 <code class="highlighter-rouge">bash</code>、<code class="highlighter-rouge">python</code> 脚本,也可以是 <code class="highlighter-rouge">Go</code> 等编译语言最终编译的二进制;以下是一个 Copy 自官方文档的 <code class="highlighter-rouge">bash</code> 编写的插件样例</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span> <span class="c"># optional argument handling</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"version"</span> <span class="o">]]</span> <span class="k">then </span><span class="nb">echo</span> <span class="s2">"1.0.0"</span> <span class="nb">exit </span>0 <span class="k">fi</span> <span class="c"># optional argument handling</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"config"</span> <span class="o">]]</span> <span class="k">then </span><span class="nb">echo</span> <span class="nv">$KUBECONFIG</span> <span class="nb">exit </span>0 <span class="k">fi </span><span class="nb">echo</span> <span class="s2">"I am a plugin named kubectl-foo"</span> </code></pre></div></div> <h2 id="二插件加载方式">二、插件加载方式</h2> <h3 id="21插件位置">2.1、插件位置</h3> <p>1.12 kubectl 插件最大的变化就是加载方式变了,由原来的放置在指定位置,还要为其编写 yaml 配置变成了现在的类似 git 扩展命令的方式: <strong>只要放置在 PATH 下,并以 <code class="highlighter-rouge">kubectl-</code> 开头的可执行文件都被认为是 <code class="highlighter-rouge">kubectl</code> 的插件</strong>;所以你可以随便弄个小脚本(比如上面的代码),然后改好名字赋予可执行权限,扔到 PATH 下即可</p> <p><img src="http://cdn.oss.link/markdown/s64v6.png" alt="test-plugin" /></p> <h3 id="22插件变量">2.2、插件变量</h3> <p>同以前不通,<strong>以前版本的执行插件时,<code class="highlighter-rouge">kubectl</code> 会向插件传递一些特定的与 <code class="highlighter-rouge">kubectl</code> 相关的变量,现在则只会传递标准变量;即 <code class="highlighter-rouge">kubectl</code> 能读到什么变量,插件就能读到,其他的私有化变量(比如 <code class="highlighter-rouge">KUBECTL_PLUGINS_CURRENT_NAMESPACE</code>)不会再提供</strong></p> <p><img src="http://cdn.oss.link/markdown/vs1c3.png" alt="plugin env" /></p> <p><strong>并且新版本的插件体系,所有选项(<code class="highlighter-rouge">flag</code>) 将全部交由插件本身处理,kubectl 不会再解析</strong>,比如下面的 <code class="highlighter-rouge">--help</code> 交给了自定义插件处理,由于脚本内没有处理这个选项,所以相当于选项无效了</p> <p><img src="http://cdn.oss.link/markdown/8ch88.png" alt="plugin flag" /></p> <p>还有就是 <strong>传递给插件的第一个参数永远是插件自己的绝对位置,比如这个 <code class="highlighter-rouge">test</code> 插件在执行时的 <code class="highlighter-rouge">$0</code> 是 <code class="highlighter-rouge">/usr/local/bin/kubectl-test</code></strong></p> <h3 id="23插件命名及查找">2.3、插件命名及查找</h3> <p>目前在插件命名及查找顺序上官方文档写的非常详尽,不给过对于普通使用者来说,实际上命名规则和查找与常规的 Linux 下的命令查找机制相同,只不过还做了增强;增强后的基本规则如下</p> <ul> <li><code class="highlighter-rouge">PATH</code> 优先匹配原则</li> <li>短横线 <code class="highlighter-rouge">-</code> 自动分割匹配以及智能转义</li> <li>以最精确匹配为首要目标</li> <li>查找失败自动转换参数</li> </ul> <p><code class="highlighter-rouge">PATH</code> 优先匹配原则跟传统的命令查找一致,即当多个路径下存在同名的插件时,则采用最先查找到的插件</p> <p><img src="http://cdn.oss.link/markdown/ljyp5.png" alt="plugin path" /></p> <p>当你的插件文件名中包含 <code class="highlighter-rouge">-</code> ,并且 <code class="highlighter-rouge">kubectl</code> 在无法精确找到插件时会尝试自动拼接命令来尝试匹配;如下所示,在没有找到 <code class="highlighter-rouge">kubectl-test</code> 这个命令时会尝试拼接参数查找</p> <p><img src="http://cdn.oss.link/markdown/l85bp.png" alt="auto merge" /></p> <p>由于以上这种查找机制,<strong>当命令中确实包含 <code class="highlighter-rouge">-</code> 时,必须进行转义以 <code class="highlighter-rouge">_</code> 替换,否则 <code class="highlighter-rouge">kubectl</code> 会提示命令未找到错误</strong>;替换后可直接使用 <code class="highlighter-rouge">kubectl 插件命令(包含-)</code> 执行,同时也支持以原始插件名称执行(使用 <code class="highlighter-rouge">_</code>)</p> <p><img src="http://cdn.oss.link/markdown/7vm0l.png" alt="name contains dash" /></p> <p>在复杂插件体系下,多个插件可能包含同样的前缀,此时将遵序最精确查找原则;即当两个插件 <code class="highlighter-rouge">kubectl-test-aaa</code>、<code class="highlighter-rouge">kubectl-test-aaa-bbb</code> 同时存在,并且执行 <code class="highlighter-rouge">kubectl test aaa bbb</code> 命令时,优先匹配最精确的插件 <code class="highlighter-rouge">kubectl-test-aaa-bbb</code>,<strong>而不是将 <code class="highlighter-rouge">bbb</code> 作为参数传递给 <code class="highlighter-rouge">kubectl-test-aaa</code> 插件</strong></p> <p><img src="http://cdn.oss.link/markdown/god8q.png" alt="precise search" /></p> <h3 id="24总结">2.4、总结</h3> <p>插件查找机制在一般情况下与传统 PATH 查找方式相同,同时 <code class="highlighter-rouge">kubectl</code> 实现了智能的 <code class="highlighter-rouge">-</code> 自动匹配查找、更精确的命令命中功能;这两种机制的实现主要为了方便编写插件的命令树(插件命令的子命令…),类似下面这种</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> ./plugin_command_tree kubectl-parent kubectl-parent-subcommand kubectl-parent-subcommand-subsubcommand </code></pre></div></div> <p>当出现多个位置有同名插件时,执行 <code class="highlighter-rouge">kubectl plugin list</code> 能够检测出哪些插件由于 PATH 查找顺序原因导致永远不会被执行问题</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl plugin list The following kubectl-compatible plugins are available: <span class="nb">test</span>/fixtures/pkg/kubectl/plugins/kubectl-foo /usr/local/bin/kubectl-foo - warning: /usr/local/bin/kubectl-foo is overshadowed by a similarly named plugin: <span class="nb">test</span>/fixtures/pkg/kubectl/plugins/kubectl-foo plugins/kubectl-invalid - warning: plugins/kubectl-invalid identified as a kubectl plugin, but it is not executable error: 2 plugin warnings were found </code></pre></div></div> <h3 id="三golang-的插件辅助库">三、Golang 的插件辅助库</h3> <p>由于插件机制的变更,导致其他语言编写的插件在实时获取某些配置信息、动态修改 <code class="highlighter-rouge">kubectl</code> 配置方面可能造成一定的阻碍;为此 kubernetes 提供了一个 <a href="http://github.com/kubernetes/cli-runtime">command line runtime package</a>,使用 Go 编写插件,配合这个库可以更加方便的解析和调整 <code class="highlighter-rouge">kubectl</code> 的配置信息</p> <p>官方为了演示如何使用这个 <a href="http://github.com/kubernetes/cli-runtime">cli-runtime</a> 库编写了一个 <code class="highlighter-rouge">namespace</code> 切换的插件(自己白写了…),仓库地址在 <a href="http://github.com/kubernetes/sample-cli-plugin">Github</a> 上,基本编译使用如下(直接 <code class="highlighter-rouge">go get</code> 后编译文件默认为目录名 <code class="highlighter-rouge">cmd</code>)</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ go get k8s.io/sample-cli-plugin/cmd ➜ ~ <span class="nb">sudo mv </span>gopath/bin/cmd /usr/local/bin/kubectl-ns ➜ ~ kubectl ns default ➜ ~ kubectl ns <span class="nt">--help</span> View or <span class="nb">set </span>the current namespace Usage: ns <span class="o">[</span>new-namespace] <span class="o">[</span>flags] Examples: <span class="c"># view the current namespace in your KUBECONFIG</span> kubectl ns <span class="c"># view all of the namespaces in use by contexts in your KUBECONFIG</span> kubectl ns <span class="nt">--list</span> <span class="c"># switch your current-context to one that contains the desired namespace</span> kubectl ns foo Flags: <span class="nt">--as</span> string Username to impersonate <span class="k">for </span>the operation <span class="nt">--as-group</span> stringArray Group to impersonate <span class="k">for </span>the operation, this flag can be repeated to specify multiple groups. <span class="nt">--cache-dir</span> string Default HTTP cache directory <span class="o">(</span>default <span class="s2">"/Users/mritd/.kube/http-cache"</span><span class="o">)</span> <span class="nt">--certificate-authority</span> string Path to a cert file <span class="k">for </span>the certificate authority <span class="nt">--client-certificate</span> string Path to a client certificate file <span class="k">for </span>TLS <span class="nt">--client-key</span> string Path to a client key file <span class="k">for </span>TLS <span class="nt">--cluster</span> string The name of the kubeconfig cluster to use <span class="nt">--context</span> string The name of the kubeconfig context to use <span class="nt">-h</span>, <span class="nt">--help</span> <span class="nb">help </span><span class="k">for </span>ns <span class="nt">--insecure-skip-tls-verify</span> If <span class="nb">true</span>, the server<span class="s1">'s certificate will not be checked for validity. This will make your http connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. --list if true, print the list of all namespaces in the current KUBECONFIG -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don'</span>t <span class="nb">timeout </span>requests. <span class="o">(</span>default <span class="s2">"0"</span><span class="o">)</span> <span class="nt">-s</span>, <span class="nt">--server</span> string The address and port of the Kubernetes API server <span class="nt">--token</span> string Bearer token <span class="k">for </span>authentication to the API server <span class="nt">--user</span> string The name of the kubeconfig user to use </code></pre></div></div> <p>限于篇幅原因,具体这个 <code class="highlighter-rouge">cli-runtime</code> 包怎么用请自行参考官方写的这个 <code class="highlighter-rouge">sample-cli-plugin</code> (其实并不怎么 “simple”…)</p> <p>本文参考文档:</p> <ul> <li><a href="http://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/">Extend kubectl with plugins</a></li> <li><a href="http://github.com/kubernetes/cli-runtime">cli-runtime</a></li> <li><a href="http://github.com/kubernetes/sample-cli-plugin">sample-cli-plugin</a></li> </ul> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Fri, 30 Nov 2018 00:05:34 +0800 /2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/ /2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/ Kubernetes Kubernetes