www.463.com 永利官网误乐域 / Wed, 16 Jan 2019 12:20:16 +0800 Wed, 16 Jan 2019 12:20:16 +0800 Jekyll v3.8.5 Kubernetes sample-cli-plugin 源码分析 <blockquote> <p>写这篇文章的目的是为了继续上篇 <a href="/2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/">Kubernetes 1.12 新的插件机制</a> 中最后部分对 <code class="highlighter-rouge">Golang 的插件辅助库</code> 说明;以及为后续使用 Golang 编写自己的 Kubernetes 插件做一个基础铺垫;顺边说一下 <strong>sample-cli-plugin 这个项目是官方为 Golang 开发者编写的一个用于快速切换配置文件中 Namespace 的一个插件样例</strong></p> </blockquote> <h2 id="一基础准备">一、基础准备</h2> <p>在开始分析源码之前,<strong>我们假设读者已经熟悉 Golang 语言,至少对基本语法、指针、依赖管理工具有一定认知</strong>;下面介绍一下 <a href="http://github.com/kubernetes/sample-cli-plugin">sample-cli-plugin</a> 这个项目一些基础核心的依赖:</p> <h3 id="11cobra-终端库">1.1、Cobra 终端库</h3> <p>这是一个强大的 Golang 的 command line interface 库,其支持用非常简单的代码创建出符合 Unix 风格的 cli 程序;甚至官方提供了用于创建 cli 工程脚手架的 cli 命令工具;Cobra 官方 Github 地址 <a href="http://github.com/spf13/cobra">点击这里</a>,具体用法请自行 Google,以下只做一个简单的命令定义介绍(docker、kubernetes 终端 cli 都基于这个库)</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">#</span><span class="x"> </span><span class="n">每一个命令</span><span class="p">(</span><span class="n">不论是子命令还是主命令</span><span class="p">)</span><span class="n">都会是一个</span><span class="x"> </span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="x"> </span><span class="n">对象</span><span class="x"> </span><span class="k">var</span><span class="x"> </span><span class="n">lsCmd</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">{</span><span class="x"> </span><span class="c">// 一些命令帮助文档有关的描述信息</span><span class="x"> </span><span class="n">Use</span><span class="o">:</span><span class="x"> </span><span class="s">"ls"</span><span class="p">,</span><span class="x"> </span><span class="n">Short</span><span class="o">:</span><span class="x"> </span><span class="s">"A brief description of your command"</span><span class="p">,</span><span class="x"> </span><span class="n">Long</span><span class="o">:</span><span class="x"> </span><span class="s">`A longer description that spans multiple lines and likely contains examples and usage of using your command. For example: Cobra is a CLI library for Go that empowers applications. This application is a tool to generate the needed files to quickly create a Cobra application.`</span><span class="p">,</span><span class="x"> </span><span class="c">// 命令运行时真正执行逻辑,如果需要返回 Error 信息,我们一般设置 RunE</span><span class="x"> </span><span class="n">Run</span><span class="o">:</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">cmd</span><span class="x"> </span><span class="o">*</span><span class="n">cobra</span><span class="o">.</span><span class="n">Command</span><span class="p">,</span><span class="x"> </span><span class="n">args</span><span class="x"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"ls called"</span><span class="p">)</span><span class="x"> </span><span class="p">},</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="c">// 为这个命令添加 flag,比如 `--help`、`-p`</span><span class="x"> </span><span class="c">// PersistentFlags() 方法添加的 flag 在所有子 command 也会生效</span><span class="x"> </span><span class="c">// Cobra 的 command 可以无限级联,比如 `kubectl get pod` 就是在 `kubectl` command 下增加了子 `get` command</span><span class="x"> </span><span class="n">lsCmd</span><span class="o">.</span><span class="n">PersistentFlags</span><span class="p">()</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"foo"</span><span class="p">,</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="s">"A help for foo"</span><span class="p">)</span><span class="x"> </span><span class="c">// Flags() 方法添加的 flag 仅在直接调用此子命令时生效</span><span class="x"> </span><span class="n">lsCmd</span><span class="o">.</span><span class="n">Flags</span><span class="p">()</span><span class="o">.</span><span class="n">BoolP</span><span class="p">(</span><span class="s">"toggle"</span><span class="p">,</span><span class="x"> </span><span class="s">"t"</span><span class="p">,</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x"> </span><span class="s">"Help message for toggle"</span><span class="p">)</span><span class="x"> </span></code></pre></div></div> <h3 id="12vendor-依赖">1.2、vendor 依赖</h3> <p>vendor 目录用于存放 Golang 的依赖库,sample-cli-plugin 这个项目采用 <a href="http://github.com/tools/godep">godep</a> 工具管理依赖;依赖配置信息被保存在 <code class="highlighter-rouge">Godeps/Godeps.json</code> 中,<strong>一般项目不会上传 vendor 目录,因为它的依赖信息已经在 Godeps.json 中存在,只需要在项目下使用 <code class="highlighter-rouge">godep restore</code> 命令恢复就可自动重新下载</strong>;这里上传了 vendor 目录的原因应该是为了方便开发者直接使用 <code class="highlighter-rouge">go get</code> 命令安装;顺边说一下在 Golang 新版本已经开始转换到 <code class="highlighter-rouge">go mod</code> 依赖管理工具,标志就是项目下会有 <code class="highlighter-rouge">go.mod</code> 文件</p> <h2 id="二源码分析">二、源码分析</h2> <h3 id="21环境搭建">2.1、环境搭建</h3> <p>这里准备一笔带过了,基本就是 clone 源码到 <code class="highlighter-rouge">$GOPATH/src/k8s.io/sample-cli-plugin</code> 目录,然后在 GoLand 中打开;目前我使用的 Go 版本为最新的 1.11.4;以下时导入源码后的截图</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/sn8o8.png" alt="GoLand" /></p> <h3 id="22定位核心运行方法">2.2、定位核心运行方法</h3> <p>熟悉过 Cobra 库以后,再从整个项目包名上分析,首先想到的启动入口应该在 <code class="highlighter-rouge">cmd</code> 包下(一般 <code class="highlighter-rouge">cmd</code> 包下的文件都会编译成最终可执行文件名,Kubernetes 也是一样)</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/rafeq.png" alt="main" /></p> <p>从以上截图中可以看出,首先通过 <code class="highlighter-rouge">cmd.NewCmdNamespace</code> 方法创建了一个 Command 对象 <code class="highlighter-rouge">root</code>,然后调用了 <code class="highlighter-rouge">root.Execute</code> 就结束了;那么也就说明 <code class="highlighter-rouge">root</code> 这个 Command 是唯一的核心命令对象,整个插件实现都在这个 <code class="highlighter-rouge">root</code> 里;所以我们需要查看一下这个 <code class="highlighter-rouge">cmd.NewCmdNamespace</code> 是如何对它初始化的,找到 Cobra 中的 <code class="highlighter-rouge">Run</code> 或者 <code class="highlighter-rouge">RunE</code> 设置</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/77krg.png" alt="NewCmdNamespace" /></p> <p>定位到 <code class="highlighter-rouge">NewCmdNamespace</code> 方法以后,基本上就是标准的 Cobra 库的使用方式了;<strong>从截图上可以看到,<code class="highlighter-rouge">RunE</code> 设置的函数总共运行了 3 个动作: <code class="highlighter-rouge">o.Complete</code>、<code class="highlighter-rouge">o.Validate</code>、<code class="highlighter-rouge">o.Run</code></strong>;所以接下来我们主要分析这三个方法就行了</p> <h3 id="23namespaceoptions-结构体">2.3、NamespaceOptions 结构体</h3> <p>在分析上面说的这三个方法之前,我们还应当了解一下这个 <code class="highlighter-rouge">o</code> 是什么玩意</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/4b3cc.png" alt="NamespaceOptions" /></p> <p>从源码中可以看到,<code class="highlighter-rouge">o</code> 这个对象由 <code class="highlighter-rouge">NewNamespaceOptions</code> 创建,而 <code class="highlighter-rouge">NewNamespaceOptions</code> 方法返回的实际上是一个 <code class="highlighter-rouge">NamespaceOptions</code> 结构体;接下来我们需要研究一下这个结构体都是由什么组成的,换句话说要基本大致上整明白结构体的基本结构,比如里面的属性都是干啥的</p> <h4 id="231genericclioptionsconfigflags">2.3.1、*genericclioptions.ConfigFlags</h4> <p>首先看下第一个属性 <code class="highlighter-rouge">configFlags</code>,它的实际类型是 <code class="highlighter-rouge">*genericclioptions.ConfigFlags</code>,点击查看以后如下</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/li6s4.png" alt="genericclioptions.ConfigFlags" /></p> <p>从这些字段上来看,我们可以暂且模糊的推测出这应该是个基础配置型的字段,负责存储一些全局基本设置,比如 API Server 认证信息等</p> <h4 id="232apicontext">2.3.2、*api.Context</h4> <p>下面这两个 <code class="highlighter-rouge">resultingContext</code>、<code class="highlighter-rouge">resultingContextName</code> 就很好理解了,从名字上看就可以知道它们应该是用来存储结果集的 Context 信息的;当然这个 <code class="highlighter-rouge">*api.Context</code> 就是 Kubernetes 配置文件中 Context 的 Go 结构体</p> <h4 id="233userspecified">2.3.3、userSpecified*</h4> <p>这几个字段从名字上就可以区分出,他们应该用于存储用户设置的或者说是通过命令行选项输入的一些指定配置信息,比如 Cluster、Context 等</p> <h4 id="234rawconfig">2.3.4、rawConfig</h4> <p>rawConfig 这个变量名字有点子奇怪,不过它实际上是个 <code class="highlighter-rouge">api.Config</code>;里面保存了与 API Server 通讯的配置信息;<strong>至于为什么要有这玩意,是因为配置信息输入源有两个: cli 命令行选项(eg: <code class="highlighter-rouge">--namespace</code>)和用户配置文件(eg: <code class="highlighter-rouge">~/.kube/config</code>);最终这两个地方的配置合并后会存储在这个 rawConfig 里</strong></p> <h4 id="235listnamespaces">2.3.5、listNamespaces</h4> <p>这个变量实际上相当于一个 flag,用于存储插件是否使用了 <code class="highlighter-rouge">--list</code> 选项;在分析结构体这里没法看出来;不过只要稍稍的多看一眼代码就能看在 <code class="highlighter-rouge">NewCmdNamespace</code> 方法中有这么一行代码</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/f07l3.png" alt="listNamespaces" /></p> <h3 id="24核心处理逻辑">2.4、核心处理逻辑</h3> <p>介绍完了结构体的基本属性,最后我们只需要弄明白在核心 Command 方法内运行的这三个核心方法就行了</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/8lm4b.png" alt="core func" /></p> <h4 id="241namespaceoptionscomplete">2.4.1、*NamespaceOptions.Complete</h4> <p>这个方法代码稍微有点多,这里不会对每一行代码都做解释,只要大体明白都在干什么就行了;我们的目的是理解它,后续模仿它创造自己的插件;下面是代码截图</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/qqf0f.png" alt="NamespaceOptions.Complete" /></p> <p>从截图上可以看到,首先弄出了 <code class="highlighter-rouge">rawConfig</code> 这个玩意,<code class="highlighter-rouge">rawConfig</code> 上面也提到了,它就是终端选项和用户配置文件的最终合并,至于为什么可以查看 <code class="highlighter-rouge">ToRawKubeConfigLoader().RawConfig()</code> 这两个方法的注释和实现即可;</p> <p>接下来就是各种获取插件执行所需要的变量信息,比如获取用户指定的 <code class="highlighter-rouge">Namespace</code>、<code class="highlighter-rouge">Cluster</code>、<code class="highlighter-rouge">Context</code> 等,其中还包含了一些必要的校验;比如不允许使用 <code class="highlighter-rouge">kubectl ns NS_NAME1 --namespace NS_NAME2</code> 这种操作(因为这么干很让人难以理解 “你到底是要切换到 <code class="highlighter-rouge">NS_NAME1</code> 还是 <code class="highlighter-rouge">NS_NAME2</code>”)</p> <p>最后从 <code class="highlighter-rouge">153</code> 行 <code class="highlighter-rouge">o.resultingContext = api.NewContext()</code> 开始就是创建最终的 <code class="highlighter-rouge">resultingContext</code> 对象,把获取到的用户指定的 <code class="highlighter-rouge">Namespace</code> 等各种信息赋值好,为下一步将其持久化到配置文件中做准备</p> <h4 id="242namespaceoptionsvalidate">2.4.2、*NamespaceOptions.Validate</h4> <p>这个方法看名字就知道,里面全是对最终结果的校验;比如检查一下 <code class="highlighter-rouge">rawConfig</code> 中的 <code class="highlighter-rouge">CurrentContext</code> 是否获取到了,看看命令行参数是否正确,确保你不会瞎鸡儿输入 <code class="highlighter-rouge">kubectl ns NS_NAME1 NS_NAME2</code> 这种命令</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/frqpb.png" alt="NamespaceOptions.Validate" /></p> <h4 id="243namespaceoptionsrun">2.4.3、*NamespaceOptions.Run</h4> <p>第一步合并配置信息并获取到用户设置(输入)的配置,第二部做参数校验;可以说前面的两步操作都是为这一步做准备,<code class="highlighter-rouge">Run</code> 方法真正的做了配置文件写入、终端返回结果打印操作</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/6tkjz.png" alt="NamespaceOptions.Run" /></p> <p>可以看到,<code class="highlighter-rouge">Run</code> 方法第一步就是更加谨慎的检查了一下参数是否正常,然后调用了 <code class="highlighter-rouge">o.setNamespace</code>;这个方法截图如下</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/1jc3k.png" alt="NamespaceOptions.setNamespace" /></p> <p>这个 <code class="highlighter-rouge">setNamespace</code>是真正的做了配置文件写入动作的,实际写入方法就是 <code class="highlighter-rouge">clientcmd.ModifyConfig</code>;这个是 <code class="highlighter-rouge">Kubernetes</code> <code class="highlighter-rouge">client-go</code> 提供的方法,这些库的作用就是提供给我们非常方便的 API 操作;比如修改配置文件,你不需要关心配置文件在哪,你更不需要关系文件句柄是否被释放</p> <p>从 <code class="highlighter-rouge">o.setNamespace</code> 方法以后其实就没什么看头了,毕竟插件的核心功能就是快速修改 <code class="highlighter-rouge">Namespace</code>;下面的各种 <code class="highlighter-rouge">for</code> 循环遍历其实就是在做打印输出;比如当你没有设置 <code class="highlighter-rouge">Namespace</code> 而使用了 <code class="highlighter-rouge">--list</code> 选项,插件就通过这里帮你打印设置过那些 <code class="highlighter-rouge">Namespace</code></p> <h2 id="三插件总结">三、插件总结</h2> <p>分析完了这个官方的插件,然后想一下自己以后写插件可能的需求,最后对比一下,可以为以后写插件做个总结:</p> <ul> <li>我们最好也弄个 <code class="highlighter-rouge">xxxOptions</code> 这种结构体存存一些配置</li> <li>结构体内至少我们应当存储 <code class="highlighter-rouge">configFlags</code>、<code class="highlighter-rouge">rawConfig</code> 这两个基础配置信息</li> <li>结构体内其它参数都应当是跟自己实际业务有关的</li> <li>最后在在结构体上增加适当的方法完成自己的业务逻辑并保持好适当的校验</li> </ul> <p>转载请注明出n,本文采用 [CC4.0](http://c 1.12 新的插件机制](/2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/) 中最后部分对 <code class="highlighter-rouge">Golang 的插件辅助库</code> 说明;以及为后续使用 Golang 编写自己的 Kubernetes 插件做一个基础铺垫;顺边说一下 <strong>sample-cli-plugin 这个项目是官方为 Golang 开发者编写的一个用于快速切换配置文件中 Namespace 的一个插件样例</strong></p> Wed, 16 Jan 2019 12:16:42 +0800 /2019/01/16/understand-kubernetes-sample-cli-plugin-source-code/ /2019/01/16/understand-kubernetes-sample-cli-plugin-source-code/ Kubernetes Kubernetes Kubernetes 1.12 新的插件机制 <blockquote> <p>在很久以前的版本研究过 kubernetes 的插件机制,当时弄了一个快速切换 <code class="highlighter-rouge">namespace</code> 的小插件;最近把自己本机的 kubectl 升级到了 1.12,突然发现插件不能用了;撸了一下文档发现插件机制彻底改了…</p> </blockquote> <h2 id="一插件编写语言">一、插件编写语言</h2> <p>kubernetes 1.12 新的插件机制在编写语言上同以前一样,<strong>可以以任意语言编写,只要能弄一个可执行的文件出来就行</strong>,插件可以是一个 <code class="highlighter-rouge">bash</code>、<code class="highlighter-rouge">python</code> 脚本,也可以是 <code class="highlighter-rouge">Go</code> 等编译语言最终编译的二进制;以下是一个 Copy 自官方文档的 <code class="highlighter-rouge">bash</code> 编写的插件样例</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span> <span class="c"># optional argument handling</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"version"</span> <span class="o">]]</span> <span class="k">then </span><span class="nb">echo</span> <span class="s2">"1.0.0"</span> <span class="nb">exit </span>0 <span class="k">fi</span> <span class="c"># optional argument handling</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"config"</span> <span class="o">]]</span> <span class="k">then </span><span class="nb">echo</span> <span class="nv">$KUBECONFIG</span> <span class="nb">exit </span>0 <span class="k">fi </span><span class="nb">echo</span> <span class="s2">"I am a plugin named kubectl-foo"</span> </code></pre></div></div> <h2 id="二插件加载方式">二、插件加载方式</h2> <h3 id="21插件位置">2.1、插件位置</h3> <p>1.12 kubectl 插件最大的变化就是加载方式变了,由原来的放置在指定位置,还要为其编写 yaml 配置变成了现在的类似 git 扩展命令的方式: <strong>只要放置在 PATH 下,并以 <code class="highlighter-rouge">kubectl-</code> 开头的可执行文件都被认为是 <code class="highlighter-rouge">kubectl</code> 的插件</strong>;所以你可以随便弄个小脚本(比如上面的代码),然后改好名字赋予可执行权限,扔到 PATH 下即可</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/s64v6.png" alt="test-plugin" /></p> <h3 id="22插件变量">2.2、插件变量</h3> <p>同以前不通,<strong>以前版本的执行插件时,<code class="highlighter-rouge">kubectl</code> 会向插件传递一些特定的与 <code class="highlighter-rouge">kubectl</code> 相关的变量,现在则只会传递标准变量;即 <code class="highlighter-rouge">kubectl</code> 能读到什么变量,插件就能读到,其他的私有化变量(比如 <code class="highlighter-rouge">KUBECTL_PLUGINS_CURRENT_NAMESPACE</code>)不会再提供</strong></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/vs1c3.png" alt="plugin env" /></p> <p><strong>并且新版本的插件体系,所有选项(<code class="highlighter-rouge">flag</code>) 将全部交由插件本身处理,kubectl 不会再解析</strong>,比如下面的 <code class="highlighter-rouge">--help</code> 交给了自定义插件处理,由于脚本内没有处理这个选项,所以相当于选项无效了</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/8ch88.png" alt="plugin flag" /></p> <p>还有就是 <strong>传递给插件的第一个参数永远是插件自己的绝对位置,比如这个 <code class="highlighter-rouge">test</code> 插件在执行时的 <code class="highlighter-rouge">$0</code> 是 <code class="highlighter-rouge">/usr/local/bin/kubectl-test</code></strong></p> <h3 id="23插件命名及查找">2.3、插件命名及查找</h3> <p>目前在插件命名及查找顺序上官方文档写的非常详尽,不给过对于普通使用者来说,实际上命名规则和查找与常规的 Linux 下的命令查找机制相同,只不过还做了增强;增强后的基本规则如下</p> <ul> <li><code class="highlighter-rouge">PATH</code> 优先匹配原则</li> <li>短横线 <code class="highlighter-rouge">-</code> 自动分割匹配以及智能转义</li> <li>以最精确匹配为首要目标</li> <li>查找失败自动转换参数</li> </ul> <p><code class="highlighter-rouge">PATH</code> 优先匹配原则跟传统的命令查找一致,即当多个路径下存在同名的插件时,则采用最先查找到的插件</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/ljyp5.png" alt="plugin path" /></p> <p>当你的插件文件名中包含 <code class="highlighter-rouge">-</code> ,并且 <code class="highlighter-rouge">kubectl</code> 在无法精确找到插件时会尝试自动拼接命令来尝试匹配;如下所示,在没有找到 <code class="highlighter-rouge">kubectl-test</code> 这个命令时会尝试拼接参数查找</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/l85bp.png" alt="auto merge" /></p> <p>由于以上这种查找机制,<strong>当命令中确实包含 <code class="highlighter-rouge">-</code> 时,必须进行转义以 <code class="highlighter-rouge">_</code> 替换,否则 <code class="highlighter-rouge">kubectl</code> 会提示命令未找到错误</strong>;替换后可直接使用 <code class="highlighter-rouge">kubectl 插件命令(包含-)</code> 执行,同时也支持以原始插件名称执行(使用 <code class="highlighter-rouge">_</code>)</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/7vm0l.png" alt="name contains dash" /></p> <p>在复杂插件体系下,多个插件可能包含同样的前缀,此时将遵序最精确查找原则;即当两个插件 <code class="highlighter-rouge">kubectl-test-aaa</code>、<code class="highlighter-rouge">kubectl-test-aaa-bbb</code> 同时存在,并且执行 <code class="highlighter-rouge">kubectl test aaa bbb</code> 命令时,优先匹配最精确的插件 <code class="highlighter-rouge">kubectl-test-aaa-bbb</code>,<strong>而不是将 <code class="highlighter-rouge">bbb</code> 作为参数传递给 <code class="highlighter-rouge">kubectl-test-aaa</code> 插件</strong></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/god8q.png" alt="precise search" /></p> <h3 id="24总结">2.4、总结</h3> <p>插件查找机制在一般情况下与传统 PATH 查找方式相同,同时 <code class="highlighter-rouge">kubectl</code> 实现了智能的 <code class="highlighter-rouge">-</code> 自动匹配查找、更精确的命令命中功能;这两种机制的实现主要为了方便编写插件的命令树(插件命令的子命令…),类似下面这种</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">ls</span> ./plugin_command_tree kubectl-parent kubectl-parent-subcommand kubectl-parent-subcommand-subsubcommand </code></pre></div></div> <p>当出现多个位置有同名插件时,执行 <code class="highlighter-rouge">kubectl plugin list</code> 能够检测出哪些插件由于 PATH 查找顺序原因导致永远不会被执行问题</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kubectl plugin list The following kubectl-compatible plugins are available: <span class="nb">test</span>/fixtures/pkg/kubectl/plugins/kubectl-foo /usr/local/bin/kubectl-foo - warning: /usr/local/bin/kubectl-foo is overshadowed by a similarly named plugin: <span class="nb">test</span>/fixtures/pkg/kubectl/plugins/kubectl-foo plugins/kubectl-invalid - warning: plugins/kubectl-invalid identified as a kubectl plugin, but it is not executable error: 2 plugin warnings were found </code></pre></div></div> <h3 id="三golang-的插件辅助库">三、Golang 的插件辅助库</h3> <p>由于插件机制的变更,导致其他语言编写的插件在实时获取某些配置信息、动态修改 <code class="highlighter-rouge">kubectl</code> 配置方面可能造成一定的阻碍;为此 kubernetes 提供了一个 <a href="http://github.com/kubernetes/cli-runtime">command line runtime package</a>,使用 Go 编写插件,配合这个库可以更加方便的解析和调整 <code class="highlighter-rouge">kubectl</code> 的配置信息</p> <p>官方为了演示如何使用这个 <a href="http://github.com/kubernetes/cli-runtime">cli-runtime</a> 库编写了一个 <code class="highlighter-rouge">namespace</code> 切换的插件(自己白写了…),仓库地址在 <a href="http://github.com/kubernetes/sample-cli-plugin">Github</a> 上,基本编译使用如下(直接 <code class="highlighter-rouge">go get</code> 后编译文件默认为目录名 <code class="highlighter-rouge">cmd</code>)</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ go get k8s.io/sample-cli-plugin/cmd ➜ ~ <span class="nb">sudo mv </span>gopath/bin/cmd /usr/local/bin/kubectl-ns ➜ ~ kubectl ns default ➜ ~ kubectl ns <span class="nt">--help</span> View or <span class="nb">set </span>the current namespace Usage: ns <span class="o">[</span>new-namespace] <span class="o">[</span>flags] Examples: <span class="c"># view the current namespace in your KUBECONFIG</span> kubectl ns <span class="c"># view all of the namespaces in use by contexts in your KUBECONFIG</span> kubectl ns <span class="nt">--list</span> <span class="c"># switch your current-context to one that contains the desired namespace</span> kubectl ns foo Flags: <span class="nt">--as</span> string Username to impersonate <span class="k">for </span>the operation <span class="nt">--as-group</span> stringArray Group to impersonate <span class="k">for </span>the operation, this flag can be repeated to specify multiple groups. <span class="nt">--cache-dir</span> string Default HTTP cache directory <span class="o">(</span>default <span class="s2">"/Users/mritd/.kube/http-cache"</span><span class="o">)</span> <span class="nt">--certificate-authority</span> string Path to a cert file <span class="k">for </span>the certificate authority <span class="nt">--client-certificate</span> string Path to a client certificate file <span class="k">for </span>TLS <span class="nt">--client-key</span> string Path to a client key file <span class="k">for </span>TLS <span class="nt">--cluster</span> string The name of the kubeconfig cluster to use <span class="nt">--context</span> string The name of the kubeconfig context to use <span class="nt">-h</span>, <span class="nt">--help</span> <span class="nb">help </span><span class="k">for </span>ns <span class="nt">--insecure-skip-tls-verify</span> If <span class="nb">true</span>, the server<span class="s1">'s certificate will not be checked for validity. This will make your http connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. --list if true, print the list of all namespaces in the current KUBECONFIG -n, --namespace string If present, the namespace scope for this CLI request --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don'</span>t <span class="nb">timeout </span>requests. <span class="o">(</span>default <span class="s2">"0"</span><span class="o">)</span> <span class="nt">-s</span>, <span class="nt">--server</span> string The address and port of the Kubernetes API server <span class="nt">--token</span> string Bearer token <span class="k">for </span>authentication to the API server <span class="nt">--user</span> string The name of the kubeconfig user to use </code></pre></div></div> <p>限于篇幅原因,具体这个 <code class="highlighter-rouge">cli-runtime</code> 包怎么用请自行参考官方写的这个 <code class="highlighter-rouge">sample-cli-plugin</code> (其实并不怎么 “simple”…)</p> <p>本文参考文档:</p> <ul> <li><a href="http://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/">Extend kubectl with plugins</a></li> <li><a href="http://github.com/kubernetes/cli-runtime">cli-runtime</a></li> <li><a href="http://github.com/kubernetes/sample-cli-plugin">sample-cli-plugin</a></li> </ul> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Fri, 30 Nov 2018 00:05:34 +0800 /2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/ /2018/11/30/kubectl-plugin-new-solution-on-kubernetes-1.12/ Kubernetes Kubernetes Go 编写的一些常用小工具 <blockquote> <p>迫于 Github 上 Star 的项目有点多,今天整理一下一些有意思的 Go 编写的小工具;大多数为终端下的实用工具,装逼的比如天气预报啥的就不写了</p> </blockquote> <h3 id="syncthing">syncthing</h3> <p>强大的文件同步工具,构建私人同步盘 👉 <a href="http://github.com/syncthing、syncthing">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/er3tj.jpg" alt="syncthing" /></p> <h3 id="fzf">fzf</h3> <p>一个强大的终端文件浏览器 👉 <a href="http://github.com/junegunn/fzf">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/ihhqy.jpg" alt="fzf" /></p> <h3 id="hey">hey</h3> <p>http 负载测试工具,简单好用 👉 <a href="http://github.com/rakyll/hey">Github</a></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Usage: hey <span class="o">[</span>options...] &lt;url&gt; Options: <span class="nt">-n</span> Number of requests to run. Default is 200. <span class="nt">-c</span> Number of requests to run concurrently. Total number of requests cannot be smaller than the concurrency level. Default is 50. <span class="nt">-q</span> Rate limit, <span class="k">in </span>queries per second <span class="o">(</span>QPS<span class="o">)</span><span class="nb">.</span> Default is no rate limit. <span class="nt">-z</span> Duration of application to send requests. When duration is reached, application stops and exits. If duration is specified, n is ignored. Examples: <span class="nt">-z</span> 10s <span class="nt">-z</span> 3m. <span class="nt">-o</span> Output type. If none provided, a summary is printed. <span class="s2">"csv"</span> is the only supported alternative. Dumps the response metrics <span class="k">in </span>comma-separated values format. <span class="nt">-m</span> HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS. <span class="nt">-H</span> Custom HTTP header. You can specify as many as needed by repeating the flag. For example, <span class="nt">-H</span> <span class="s2">"Accept: text/html"</span> <span class="nt">-H</span> <span class="s2">"Content-Type: application/xml"</span> <span class="nb">.</span> <span class="nt">-t</span> Timeout <span class="k">for </span>each request <span class="k">in </span>seconds. Default is 20, use 0 <span class="k">for </span>infinite. <span class="nt">-A</span> HTTP Accept header. <span class="nt">-d</span> HTTP request body. <span class="nt">-D</span> HTTP request body from file. For example, /home/user/file.txt or ./file.txt. <span class="nt">-T</span> Content-type, defaults to <span class="s2">"text/html"</span><span class="nb">.</span> <span class="nt">-a</span> Basic authentication, username:password. <span class="nt">-x</span> HTTP Proxy address as host:port. <span class="nt">-h2</span> Enable HTTP/2. <span class="nt">-host</span> HTTP Host header. <span class="nt">-disable-compression</span> Disable compression. <span class="nt">-disable-keepalive</span> Disable keep-alive, prevents re-use of TCP connections between different HTTP requests. <span class="nt">-disable-redirects</span> Disable following of HTTP redirects <span class="nt">-cpus</span> Number of used cpu cores. <span class="o">(</span>default <span class="k">for </span>current machine is 8 cores<span class="o">)</span> </code></pre></div></div> <h3 id="vegeta">vegeta</h3> <p>http 负载测试工具,功能强大 👉 <a href="http://github.com/tsenart/vegeta">Github</a></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Usage: vegeta <span class="o">[</span>global flags] &lt;<span class="nb">command</span><span class="o">&gt;</span> <span class="o">[</span><span class="nb">command </span>flags] global flags: <span class="nt">-cpus</span> int Number of CPUs to use <span class="o">(</span>default 8<span class="o">)</span> <span class="nt">-profile</span> string Enable profiling of <span class="o">[</span>cpu, heap] <span class="nt">-version</span> Print version and <span class="nb">exit </span>attack <span class="nb">command</span>: <span class="nt">-body</span> string Requests body file <span class="nt">-cert</span> string TLS client PEM encoded certificate file <span class="nt">-connections</span> int Max open idle connections per target host <span class="o">(</span>default 10000<span class="o">)</span> <span class="nt">-duration</span> duration Duration of the <span class="nb">test</span> <span class="o">[</span>0 <span class="o">=</span> forever] <span class="nt">-format</span> string Targets format <span class="o">[</span>http, json] <span class="o">(</span>default <span class="s2">"http"</span><span class="o">)</span> <span class="nt">-h2c</span> Send HTTP/2 requests without TLS encryption <span class="nt">-header</span> value Request header <span class="nt">-http2</span> Send HTTP/2 requests when supported by the server <span class="o">(</span>default <span class="nb">true</span><span class="o">)</span> <span class="nt">-insecure</span> Ignore invalid server TLS certificates <span class="nt">-keepalive</span> Use persistent connections <span class="o">(</span>default <span class="nb">true</span><span class="o">)</span> <span class="nt">-key</span> string TLS client PEM encoded private key file <span class="nt">-laddr</span> value Local IP address <span class="o">(</span>default 0.0.0.0<span class="o">)</span> <span class="nt">-lazy</span> Read targets lazily <span class="nt">-max-body</span> value Maximum number of bytes to capture from response bodies. <span class="o">[</span><span class="nt">-1</span> <span class="o">=</span> no limit] <span class="o">(</span>default <span class="nt">-1</span><span class="o">)</span> <span class="nt">-name</span> string Attack name <span class="nt">-output</span> string Output file <span class="o">(</span>default <span class="s2">"stdout"</span><span class="o">)</span> <span class="nt">-rate</span> value Number of requests per <span class="nb">time </span>unit <span class="o">(</span>default 50/1s<span class="o">)</span> <span class="nt">-redirects</span> int Number of redirects to follow. <span class="nt">-1</span> will not follow but marks as success <span class="o">(</span>default 10<span class="o">)</span> <span class="nt">-resolvers</span> value List of addresses <span class="o">(</span>ip:port<span class="o">)</span> to use <span class="k">for </span>DNS resolution. Disables use of <span class="nb">local </span>system DNS. <span class="o">(</span>comma separated list<span class="o">)</span> <span class="nt">-root-certs</span> value TLS root certificate files <span class="o">(</span>comma separated list<span class="o">)</span> <span class="nt">-targets</span> string Targets file <span class="o">(</span>default <span class="s2">"stdin"</span><span class="o">)</span> <span class="nt">-timeout</span> duration Requests <span class="nb">timeout</span> <span class="o">(</span>default 30s<span class="o">)</span> <span class="nt">-workers</span> uint Initial number of workers <span class="o">(</span>default 10<span class="o">)</span> encode <span class="nb">command</span>: <span class="nt">-output</span> string Output file <span class="o">(</span>default <span class="s2">"stdout"</span><span class="o">)</span> <span class="nt">-to</span> string Output encoding <span class="o">[</span>csv, gob, json] <span class="o">(</span>default <span class="s2">"json"</span><span class="o">)</span> plot <span class="nb">command</span>: <span class="nt">-output</span> string Output file <span class="o">(</span>default <span class="s2">"stdout"</span><span class="o">)</span> <span class="nt">-threshold</span> int Threshold of data points above which series are downsampled. <span class="o">(</span>default 4000<span class="o">)</span> <span class="nt">-title</span> string Title and header of the resulting HTML page <span class="o">(</span>default <span class="s2">"Vegeta Plot"</span><span class="o">)</span> report <span class="nb">command</span>: <span class="nt">-every</span> duration Report interval <span class="nt">-output</span> string Output file <span class="o">(</span>default <span class="s2">"stdout"</span><span class="o">)</span> <span class="nt">-type</span> string Report <span class="nb">type </span>to generate <span class="o">[</span>text, json, hist[buckets]] <span class="o">(</span>default <span class="s2">"text"</span><span class="o">)</span> examples: <span class="nb">echo</span> <span class="s2">"GET http://localhost/"</span> | vegeta attack <span class="nt">-duration</span><span class="o">=</span>5s | <span class="nb">tee </span>results.bin | vegeta report vegeta report <span class="nt">-type</span><span class="o">=</span>json results.bin <span class="o">&gt;</span> metrics.json <span class="nb">cat </span>results.bin | vegeta plot <span class="o">&gt;</span> plot.html <span class="nb">cat </span>results.bin | vegeta report <span class="nt">-type</span><span class="o">=</span><span class="s2">"hist[0,100ms,200ms,300ms]"</span> </code></pre></div></div> <h3 id="dive">dive</h3> <p>功能强大的 Docker 镜像分析工具,可以查看每层镜像的具体差异等 👉 <a href="http://github.com/wagoodman/dive">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/ik3ng.gif" alt="dive" /></p> <h3 id="ctop">ctop</h3> <p>容器运行时资源分析,如 CPU、内存消耗等 👉 <a href="http://github.com/bcicen/ctop">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/mr3x3.gif" alt="ctop" /></p> <h3 id="container-diff">container-diff</h3> <p>Google 推出的工具,功能就顾名思义了 👉 <a href="http://github.com/GoogleContainerTools/container-diff">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/dtapx.png" alt="container-diff" /></p> <h3 id="transfersh">transfer.sh</h3> <p>快捷的终端文件分享工具 👉 <a href="http://github.com/dutchcoders/transfer.sh">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/76vh0.png" alt="transfer.sh" /></p> <h3 id="vuls">vuls</h3> <p>Linux/FreeBSD 漏洞扫描工具 👉 <a href="http://github.com/future-architect/vuls">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/bpsps.jpg" alt="vuls" /></p> <h3 id="restic">restic</h3> <p>高性能安全的文件备份工具 👉 <a href="http://github.com/restic/restic">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/g51z4.png" alt="restic" /></p> <h3 id="gitql">gitql</h3> <p>使用 sql 的方式查询 git 提交 👉 <a href="http://github.com/cloudson/gitql">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/4h095.gif" alt="gitql" /></p> <h3 id="gitflow-toolkit">gitflow-toolkit</h3> <p>帮助生成满足 Gitflow 格式 commit message 的小工具(自己写的) 👉 <a href="http://github.com/mritd/gitflow-toolkit">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/1e2v1.gif" alt="gitflow-toolkit" /></p> <h3 id="git-chglog">git-chglog</h3> <p>对主流的 Gitflow 格式的 commit message 生成 CHANGELOG 👉 <a href="http://github.com/git-chglog/git-chglog">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/zphxd.gif" alt="git-chglog" /></p> <h3 id="grv">grv</h3> <p>一个 git 终端图形化浏览工具 👉 <a href="http://github.com/rgburke/grv">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/k1vh2.jpg" alt="grv" /></p> <h3 id="jid">jid</h3> <p>命令行 json 格式化处理工具,类似 jq,不过感觉更加强大 👉 <a href="http://github.com/simeji/jid">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/3k4ue.gif" alt="jid" /></p> <h3 id="annie">annie</h3> <p>类似 youget 的一个视频下载工具,可以解析大部分视频网站直接下载 👉 <a href="http://github.com/iawia002/annie">Github</a></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>annie <span class="nt">-i</span> http://www.youtube.com/watch?v<span class="o">=</span>dQw4w9WgXcQ Site: YouTube youtube.com Title: Rick Astley - Never Gonna Give You Up <span class="o">(</span>Video<span class="o">)</span> Type: video Streams: <span class="c"># All available quality</span> <span class="o">[</span>248] <span class="nt">-------------------</span> Quality: 1080p video/webm<span class="p">;</span> <span class="nv">codecs</span><span class="o">=</span><span class="s2">"vp9"</span> Size: 49.29 MiB <span class="o">(</span>51687554 Bytes<span class="o">)</span> <span class="c"># download with: annie -f 248 ...</span> <span class="o">[</span>137] <span class="nt">-------------------</span> Quality: 1080p video/mp4<span class="p">;</span> <span class="nv">codecs</span><span class="o">=</span><span class="s2">"avc1.640028"</span> Size: 43.45 MiB <span class="o">(</span>45564306 Bytes<span class="o">)</span> <span class="c"># download with: annie -f 137 ...</span> <span class="o">[</span>398] <span class="nt">-------------------</span> Quality: 720p video/mp4<span class="p">;</span> <span class="nv">codecs</span><span class="o">=</span><span class="s2">"av01.0.05M.08"</span> Size: 37.12 MiB <span class="o">(</span>38926432 Bytes<span class="o">)</span> <span class="c"># download with: annie -f 398 ...</span> <span class="o">[</span>136] <span class="nt">-------------------</span> Quality: 720p video/mp4<span class="p">;</span> <span class="nv">codecs</span><span class="o">=</span><span class="s2">"avc1.4d401f"</span> Size: 31.34 MiB <span class="o">(</span>32867324 Bytes<span class="o">)</span> <span class="c"># download with: annie -f 136 ...</span> <span class="o">[</span>247] <span class="nt">-------------------</span> Quality: 720p video/webm<span class="p">;</span> <span class="nv">codecs</span><span class="o">=</span><span class="s2">"vp9"</span> Size: 31.03 MiB <span class="o">(</span>32536181 Bytes<span class="o">)</span> <span class="c"># download with: annie -f 247 ...</span> </code></pre></div></div> <h3 id="up">up</h3> <p>Linux 下管道式终端搜索工具 👉 <a href="http://github.com/akavel/up">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/n8zdj.gif" alt="up" /></p> <h3 id="lego">lego</h3> <p>Let’s Encrypt 证书申请工具 👉 <a href="http://github.com/xenolf/lego">Github</a></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME: lego - Let<span class="s1">'s Encrypt client written in Go USAGE: lego [global options] command [command options] [arguments...] COMMANDS: run Register an account, then create and install a certificate revoke Revoke a certificate renew Renew a certificate dnshelp Shows additional help for the --dns global option help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --domains value, -d value Add a domain to the process. Can be specified multiple times. --csr value, -c value Certificate signing request filename, if an external CSR is to be used --server value, -s value CA hostname (and optionally :port). The server certificate must be trusted in order to avoid further modifications to the client. (default: "http://acme-v02.api.letsencrypt.org/directory") --email value, -m value Email used for registration and recovery contact. --filename value Filename of the generated certificate --accept-tos, -a By setting this flag to true you indicate that you accept the current Let'</span>s Encrypt terms of service. <span class="nt">--eab</span> Use External Account Binding <span class="k">for </span>account registration. Requires <span class="nt">--kid</span> and <span class="nt">--hmac</span><span class="nb">.</span> <span class="nt">--kid</span> value Key identifier from External CA. Used <span class="k">for </span>External Account Binding. <span class="nt">--hmac</span> value MAC key from External CA. Should be <span class="k">in </span>Base64 URL Encoding without padding format. Used <span class="k">for </span>External Account Binding. <span class="nt">--key-type</span> value, <span class="nt">-k</span> value Key <span class="nb">type </span>to use <span class="k">for </span>private keys. Supported: rsa2048, rsa4096, rsa8192, ec256, ec384 <span class="o">(</span>default: <span class="s2">"rsa2048"</span><span class="o">)</span> <span class="nt">--path</span> value Directory to use <span class="k">for </span>storing the data <span class="o">(</span>default: <span class="s2">"./.lego"</span><span class="o">)</span> <span class="nt">--exclude</span> value, <span class="nt">-x</span> value Explicitly disallow solvers by name from being used. Solvers: <span class="s2">"http-01"</span>, <span class="s2">"dns-01"</span>, <span class="s2">"tls-alpn-01"</span><span class="nb">.</span> <span class="nt">--webroot</span> value Set the webroot folder to use <span class="k">for </span>HTTP based challenges to write directly <span class="k">in </span>a file <span class="k">in</span> .well-known/acme-challenge <span class="nt">--memcached-host</span> value Set the memcached host<span class="o">(</span>s<span class="o">)</span> to use <span class="k">for </span>HTTP based challenges. Challenges will be written to all specified hosts. <span class="nt">--http</span> value Set the port and interface to use <span class="k">for </span>HTTP based challenges to listen on. Supported: interface:port or :port <span class="nt">--tls</span> value Set the port and interface to use <span class="k">for </span>TLS based challenges to listen on. Supported: interface:port or :port <span class="nt">--dns</span> value Solve a DNS challenge using the specified provider. Disables all other challenges. Run <span class="s1">'lego dnshelp'</span> <span class="k">for </span><span class="nb">help </span>on usage. <span class="nt">--http-timeout</span> value Set the HTTP <span class="nb">timeout </span>value to a specific value <span class="k">in </span>seconds. The default is 10 seconds. <span class="o">(</span>default: 0<span class="o">)</span> <span class="nt">--dns-timeout</span> value Set the DNS <span class="nb">timeout </span>value to a specific value <span class="k">in </span>seconds. The default is 10 seconds. <span class="o">(</span>default: 0<span class="o">)</span> <span class="nt">--dns-resolvers</span> value Set the resolvers to use <span class="k">for </span>performing recursive DNS queries. Supported: host:port. The default is to use the system resolvers, or Google<span class="s1">'s DNS resolvers if the system'</span>s cannot be determined. <span class="nt">--pem</span> Generate a .pem file by concatenating the .key and .crt files together. <span class="nt">--help</span>, <span class="nt">-h</span> show <span class="nb">help</span> <span class="nt">--version</span>, <span class="nt">-v</span> print the version </code></pre></div></div> <h3 id="noti">noti</h3> <p>贼好用的终端命令异步执行通知工具 👉 <a href="http://github.com/variadico/noti">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/m2r1e.jpg" alt="noti" /></p> <h3 id="gosu">gosu</h3> <p>临时切换到指定用户运行特定命令,方便测试权限问题 👉 <a href="http://github.com/tianon/gosu">Github</a></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gosu Usage: ./gosu user-spec <span class="nb">command</span> <span class="o">[</span>args] eg: ./gosu tianon bash ./gosu nobody:root bash <span class="nt">-c</span> <span class="s1">'whoami &amp;&amp; id'</span> ./gosu 1000:1 <span class="nb">id</span> </code></pre></div></div> <h3 id="sup">sup</h3> <p>类似 Ansible 的一个批量执行工具,暂且称之为低配版 Ansible 👉 <a href="http://github.com/pressly/sup">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/x0eaz.gif" alt="sup" /></p> <h3 id="aptly">aptly</h3> <p>Debian 仓库管理工具 👉 <a href="http://github.com/aptly-dev/aptly">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/8e0ml.jpg" alt="aptly" /></p> <h3 id="mmh">mmh</h3> <p>支持无限跳板机登录的 ssh 小工具(自己写的) 👉 <a href="http://github.com/mritd/mmh">Github</a></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/37638.gif" alt="mmh" /></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 27 Nov 2018 12:45:46 +0800 /2018/11/27/simple-tool-written-in-golang/ /2018/11/27/simple-tool-written-in-golang/ Golang Linux Golang Linux 远程 Debug kubeadm <blockquote> <p>最近在看 kubeadm 的源码,不过有些东西光看代码还是没法太清楚,还是需要实际运行才能看到具体代码怎么跑的,还得打断点 debug;无奈的是本机是 mac,debug 得在 Linux 下,so 研究了一下 remote debug</p> </blockquote> <h2 id="一环境准备">一、环境准备</h2> <ul> <li>GoLand 2018.2.4</li> <li>Golang 1.11.2</li> <li>delve v1.1.0</li> <li>Kubernetest master</li> <li>Ubuntu 18.04</li> <li>能够高速访问外网(自行理解)</li> </ul> <p><strong>这里不会详细写如何安装 Go 开发环境以及 GoLand 安装,本文默认读者已经至少已经对 Go 开发环境以及代码有一定了解;顺便提一下 GoLand,这玩意属于 jetbrains 系列 IDE,在大约 2018.1 版本后在线激活服务器已经全部失效,不过网上还有其他本地离线激活工具,具体请自行 Google,如果后续工资能支撑得起,请补票支持正版(感恩节全家桶半价真香😂)</strong></p> <h3 id="11获取源码">1.1、获取源码</h3> <p>需要注意的是 Kubernetes 源码虽然托管在 Github,但是在使用 <code class="highlighter-rouge">go get</code> 的时候要使用 <code class="highlighter-rouge">k8s.io</code> 域名</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>go get <span class="nt">-d</span> k8s.io/kubernetes </code></pre></div></div> <p><code class="highlighter-rouge">go get</code> 命令是接受标准的 http 代理的,这个源码下载会非常慢,源码大约 1G 左右,所以最好使用加速工具下载</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ which proxy /usr/local/bin/proxy ➜ ~ <span class="nb">cat</span> /usr/local/bin/proxy <span class="c">#!/bin/bash</span> <span class="nv">http_proxy</span><span class="o">=</span>http://127.0.0.1:8123 <span class="nv">http_proxy</span><span class="o">=</span>http://127.0.0.1:8123 <span class="nv">$*</span> ➜ ~ proxy go get <span class="nt">-d</span> k8s.io/kubernetes </code></pre></div></div> <h3 id="12安装-delve">1.2、安装 delve</h3> <p>delve 是一个 Golang 的 debug 工具,有点类似 gdb,不过是专门针对 Golang 的,GoLand 的 debug 实际上就是使用的这个开源工具;为了进行远程 debug,运行 kubeadm 的机器必须安装 delve,从而进行远程连接</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 同样这里省略在 Linux 安装 go 环境操作</span> go get <span class="nt">-u</span> github.com/derekparker/delve/cmd/dlv </code></pre></div></div> <h2 id="二远程-debug">二、远程 Debug</h2> <h3 id="21重新编译-kubeadm">2.1、重新编译 kubeadm</h3> <p>默认情况下直接编译出的 kubeadm 是无法进行 debug 的,因为 Golang 的编译器会进行编译优化,比如进行内联等;所以要关闭编译优化和内联,方便 debug</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> <span class="k">${</span><span class="nv">GOPATH</span><span class="k">}</span>/src/k8s.io/kubernetes/cmd/kubeadm <span class="nv">GOOS</span><span class="o">=</span><span class="s2">"linux"</span> <span class="nv">GOARCH</span><span class="o">=</span><span class="s2">"amd64"</span> go build <span class="nt">-gcflags</span> <span class="s2">"all=-N -l"</span> </code></pre></div></div> <h3 id="22远程运行-kubeadm">2.2、远程运行 kubeadm</h3> <p>将编译好的 kubeadm 复制到远程,并且使用 delve 启动它,此时 delve 会监听 api 端口,GoLand 就可以远程连接过来了</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dlv <span class="nt">--listen</span><span class="o">=</span>192.168.1.61:2345 <span class="nt">--headless</span><span class="o">=</span><span class="nb">true</span> <span class="nt">--api-version</span><span class="o">=</span>2 <span class="nb">exec</span> ./kubeadm init </code></pre></div></div> <p><strong>注意: 要指定需要 debug 的 kubeadm 的子命令,否则可能出现连接上以后 GoLand 无反应的情况</strong></p> <h3 id="23运行-goland">2.3、运行 GoLand</h3> <p>在 GoLand 中打开 kubernetes 源码,在需要 debug 的代码中打上断点,这里以 init 子命令为例</p> <p>首先新建一个远程 debug configuration</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/i6oed.png" alt="create configuration" /></p> <p>名字可以随便写,主要是地址和端口</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/rmczj.png" alt="conifg delve" /></p> <p>接下来在目标源码位置打断点,以下为 init 子命令的源码位置</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/ylf97.png" alt="create breakpoint" /></p> <p>最后只需要点击 debug 按钮即可</p> <p><img src="http://mritd.b0.upaiyun.com/markdown/ns2yw.png" alt="debug" /></p> <p><strong>在没有运行 GoLand debug 之前,目标机器的实际指令是不会运行的,也就是说在 GoLand 没有连接到远程 delve 启动的 <code class="highlighter-rouge">kubeadm init</code> 命令之前,<code class="highlighter-rouge">kubeadm init</code> 并不会真正运行;当点击 GoLand 的终止 debug 按钮后,远程的 delve 也会随之退出</strong></p> <p><img src="http://mritd.b0.upaiyun.com/markdown/lmdke.png" alt="stop" /></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Sun, 25 Nov 2018 11:11:28 +0800 /2018/11/25/kubeadm-remote-debug/ /2018/11/25/kubeadm-remote-debug/ Golang Kubernetes Golang Kubenretes Mac: Extract JDK to folder, without running installer <blockquote> <p>重装了 mac 系统,由于一些公司项目必须使用 Oracle JDK(验证码等组件用了一些 Oracle 独有的 API) 所以又得重新安装;但是 Oracle 只提供了 pkg 的安装方式,研究半天找到了一个解包 pkg 的安装方式,这里记录一下</p> </blockquote> <p>不使用 pkg 的原因是每次更新版本都要各种安装,最烦人的是 IDEA 选择 JDK 时候弹出的文件浏览器没法进入到这种方式安装的 JDK 的系统目录…mmp,后来从国外网站找到了一篇文章,基本套路如下</p> <ul> <li>下载 Oracle JDK,从 dmg 中拷贝 pkg 到任意位置</li> <li>解压 pkg 到任意位置 <code class="highlighter-rouge">pkgutil --expand your_jdk.pkg jdkdir</code></li> <li>进入到目录中,解压主文件 <code class="highlighter-rouge">cd jdkdir/jdk_version.pkg &amp;&amp; cpio -idv &lt; Payload</code></li> <li>移动 jdk 到任意位置 <code class="highlighter-rouge">mv Contents/Home ~/myjdk</code></li> </ul> <p>原文地址: <a href="http://augustl.com/blog/2014/extracting_java_to_folder_no_installer_osx/">OS X: Extract JDK to folder, without running installer</a></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Fri, 23 Nov 2018 12:33:20 +0800 /2018/11/23/extract-jdk-to-folder-on-mac/ /2018/11/23/extract-jdk-to-folder-on-mac/ Java Java Go ssh 交互式执行命令 <blockquote> <p>最近在写一个跳板机登录的小工具,其中涉及到了用 Go 来进行交互式执行命令,简单地说就是弄个终端出来;一开始随便 Google 了一下,copy 下来基本上就是能跑了…但是后来发现了一些各种各样的小问题,强迫症的我实在受不了,最后翻了一下 Teleport 的源码,从中学到了不少有用的知识,这里记录一下</p> </blockquote> <h2 id="一原始版本">一、原始版本</h2> <blockquote> <p>不想看太多可以直接跳转到 <a href="#三完整代码">第三部分</a> 拿代码</p> </blockquote> <h3 id="11样例代码">1.1、样例代码</h3> <p>一开始随便 Google 出来的代码,copy 上就直接跑;代码基本如下:</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// 创建 ssh 配置</span><span class="x"> </span><span class="n">sshConfig</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">ssh</span><span class="o">.</span><span class="n">ClientConfig</span><span class="p">{</span><span class="x"> </span><span class="n">User</span><span class="o">:</span><span class="x"> </span><span class="s">"root"</span><span class="p">,</span><span class="x"> </span><span class="n">Auth</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="n">ssh</span><span class="o">.</span><span class="n">AuthMethod</span><span class="p">{</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">Password</span><span class="p">(</span><span class="s">"password"</span><span class="p">),</span><span class="x"> </span><span class="p">},</span><span class="x"> </span><span class="n">HostKeyCallback</span><span class="o">:</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">InsecureIgnoreHostKey</span><span class="p">(),</span><span class="x"> </span><span class="n">Timeout</span><span class="o">:</span><span class="x"> </span><span class="m">5</span><span class="x"> </span><span class="o">*</span><span class="x"> </span><span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">,</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="c">// 创建 client</span><span class="x"> </span><span class="n">client</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="x"> </span><span class="s">"192.168.1.20:22"</span><span class="p">,</span><span class="x"> </span><span class="n">sshConfig</span><span class="p">)</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">client</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x"> </span><span class="c">// 获取 session</span><span class="x"> </span><span class="n">session</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">client</span><span class="o">.</span><span class="n">NewSession</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x"> </span><span class="c">// 拿到当前终端文件描述符</span><span class="x"> </span><span class="n">fd</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="kt">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Fd</span><span class="p">())</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="c">// request pty</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">RequestPty</span><span class="p">(</span><span class="s">"xterm-256color"</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">TerminalModes</span><span class="p">{})</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="c">// 对接 std</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Stdout</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Stderr</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Stderr</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Stdin</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Shell</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="x"> </span><span class="kt">error</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Exit</span><span class="p">(</span><span class="m">1</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span></code></pre></div></div> <h3 id="12遇到的问题">1.2、遇到的问题</h3> <p>以上代码跑起来后,基本上遇到了以下问题:</p> <ul> <li>执行命令有回显,表现为敲一个 <code class="highlighter-rouge">ls</code> 出现两行</li> <li>本地终端大小调整,远端完全无反应,导致显示不全</li> <li>Tmux 下终端连接后窗口标题显示的是原始命令,而不是目标机器 shell 环境的目录位置</li> <li>首次连接一些刚装完系统的机器可能出现执行命令后回显不换行</li> </ul> <h2 id="二改进代码">二、改进代码</h2> <h3 id="21回显问题">2.1、回显问题</h3> <p>关于回显问题,实际上解决方案很简单,设置当前终端进入 <code class="highlighter-rouge">raw</code> 模式即可;代码如下:</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// 拿到当前终端文件描述符</span><span class="x"> </span><span class="n">fd</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="kt">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Fd</span><span class="p">())</span><span class="x"> </span><span class="c">// make raw</span><span class="x"> </span><span class="n">state</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">MakeRaw</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">Restore</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span><span class="x"> </span><span class="n">state</span><span class="p">)</span><span class="x"> </span></code></pre></div></div> <p>代码很简单,网上一大堆,But…基本没有文章详细说这个 <code class="highlighter-rouge">raw</code> 模式到底是个啥玩意;好在万能的 StackOverflow 对于不熟悉 Linux 的人给出了一个很清晰的解释: <a href="http://unix.stackexchange.com/questions/21752/what-s-the-difference-between-a-raw-and-a-cooked-device-driver">What’s the difference between a “raw” and a “cooked” device driver?</a></p> <p>大致意思就是说 <strong>在终端处于 <code class="highlighter-rouge">Cooked</code> 模式时,当你输入一些字符后,默认是被当前终端 cache 住的,在你敲了回车之前这些文本都在 cache 中,这样允许应用程序做一些处理,比如捕获 <code class="highlighter-rouge">Cntl-D</code> 等按键,这时候就会出现敲回车后本地终端帮你打印了一下,导致出现类似回显的效果;当设置终端为 <code class="highlighter-rouge">raw</code> 模式后,所有的输入将不被 cache,而是发送到应用程序,在我们的代码中表现为通过 <code class="highlighter-rouge">io.Copy</code> 直接发送到了远端 shell 程序</strong></p> <h3 id="22终端大小问题">2.2、终端大小问题</h3> <p>当本地调整了终端大小后,远程终端毫无反应;后来发现在 <code class="highlighter-rouge">*ssh.Session</code> 上有一个 <code class="highlighter-rouge">WindowChange</code> 方法,用于向远端发送窗口调整事件;解决方案就是启动一个 <code class="highlighter-rouge">goroutine</code> 在后台不断监听窗口改变事件,然后调用 <code class="highlighter-rouge">WindowChange</code> 即可;代码如下:</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// 监听窗口变更事件</span><span class="x"> </span><span class="n">sigwinchCh</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">chan</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Signal</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">)</span><span class="x"> </span><span class="n">signal</span><span class="o">.</span><span class="n">Notify</span><span class="p">(</span><span class="n">sigwinchCh</span><span class="p">,</span><span class="x"> </span><span class="n">syscall</span><span class="o">.</span><span class="n">SIGWINCH</span><span class="p">)</span><span class="x"> </span><span class="n">fd</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="kt">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Fd</span><span class="p">())</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">select</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// 阻塞读取</span><span class="x"> </span><span class="k">case</span><span class="x"> </span><span class="n">sigwinch</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&lt;-</span><span class="n">sigwinchCh</span><span class="o">:</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">sigwinch</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">,</span><span class="x"> </span><span class="n">currTermHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="c">// 判断一下窗口尺寸是否有改变</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">currTermHeight</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="n">termHeight</span><span class="x"> </span><span class="o">&amp;&amp;</span><span class="x"> </span><span class="n">currTermWidth</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="n">termWidth</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">continue</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="c">// 更新远端大小</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">WindowChange</span><span class="p">(</span><span class="n">currTermHeight</span><span class="p">,</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Unable to send window-change reqest: %s."</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">continue</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">,</span><span class="x"> </span><span class="n">currTermHeight</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}()</span><span class="x"> </span></code></pre></div></div> <h3 id="23tmux-标题以及回显不换行">2.3、Tmux 标题以及回显不换行</h3> <p>这两个问题实际上都是由于我们直接对接了 <code class="highlighter-rouge">stderr</code>、<code class="highlighter-rouge">stdout</code> 和 <code class="highlighter-rouge">stdin</code> 造成的,实际上我们应当启动一个异步的管道式复制行为,并且最好带有 buf 的发送;代码如下:</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">stdin</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">StdinPipe</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="n">stdout</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">StdoutPipe</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="n">stderr</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">StderrPipe</span><span class="p">()</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stderr</span><span class="p">,</span><span class="x"> </span><span class="n">stderr</span><span class="p">)</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="p">,</span><span class="x"> </span><span class="n">stdout</span><span class="p">)</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">buf</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="x"> </span><span class="m">128</span><span class="p">)</span><span class="x"> </span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">n</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">n</span><span class="x"> </span><span class="o">&gt;</span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">stdin</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="o">:</span><span class="n">n</span><span class="p">])</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">checkErr</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}()</span><span class="x"> </span></code></pre></div></div> <h2 id="三完整代码">三、完整代码</h2> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span><span class="x"> </span><span class="n">SSHTerminal</span><span class="x"> </span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">Session</span><span class="x"> </span><span class="o">*</span><span class="n">ssh</span><span class="o">.</span><span class="n">Session</span><span class="x"> </span><span class="n">exitMsg</span><span class="x"> </span><span class="kt">string</span><span class="x"> </span><span class="n">stdout</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Reader</span><span class="x"> </span><span class="n">stdin</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Writer</span><span class="x"> </span><span class="n">stderr</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Reader</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">sshConfig</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">ssh</span><span class="o">.</span><span class="n">ClientConfig</span><span class="p">{</span><span class="x"> </span><span class="n">User</span><span class="o">:</span><span class="x"> </span><span class="s">"root"</span><span class="p">,</span><span class="x"> </span><span class="n">Auth</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="n">ssh</span><span class="o">.</span><span class="n">AuthMethod</span><span class="p">{</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">Password</span><span class="p">(</span><span class="s">"password"</span><span class="p">),</span><span class="x"> </span><span class="p">},</span><span class="x"> </span><span class="n">HostKeyCallback</span><span class="o">:</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">InsecureIgnoreHostKey</span><span class="p">(),</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">client</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">Dial</span><span class="p">(</span><span class="s">"tcp"</span><span class="p">,</span><span class="x"> </span><span class="s">"192.168.1.20:22"</span><span class="p">,</span><span class="x"> </span><span class="n">sshConfig</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">client</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">New</span><span class="p">(</span><span class="n">client</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">SSHTerminal</span><span class="p">)</span><span class="x"> </span><span class="n">updateTerminalSize</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// SIGWINCH is sent to the process when the window size of the terminal has</span><span class="x"> </span><span class="c">// changed.</span><span class="x"> </span><span class="n">sigwinchCh</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">chan</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Signal</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">)</span><span class="x"> </span><span class="n">signal</span><span class="o">.</span><span class="n">Notify</span><span class="p">(</span><span class="n">sigwinchCh</span><span class="p">,</span><span class="x"> </span><span class="n">syscall</span><span class="o">.</span><span class="n">SIGWINCH</span><span class="p">)</span><span class="x"> </span><span class="n">fd</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="kt">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Fd</span><span class="p">())</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">select</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// The client updated the size of the local PTY. This change needs to occur</span><span class="x"> </span><span class="c">// on the server side PTY as well.</span><span class="x"> </span><span class="k">case</span><span class="x"> </span><span class="n">sigwinch</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&lt;-</span><span class="n">sigwinchCh</span><span class="o">:</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">sigwinch</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">,</span><span class="x"> </span><span class="n">currTermHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="c">// Terminal size has not changed, don't do anything.</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">currTermHeight</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="n">termHeight</span><span class="x"> </span><span class="o">&amp;&amp;</span><span class="x"> </span><span class="n">currTermWidth</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="n">termWidth</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">continue</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">WindowChange</span><span class="p">(</span><span class="n">currTermHeight</span><span class="p">,</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Unable to send window-change reqest: %s."</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">continue</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">currTermWidth</span><span class="p">,</span><span class="x"> </span><span class="n">currTermHeight</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}()</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">SSHTerminal</span><span class="p">)</span><span class="x"> </span><span class="n">interactiveSession</span><span class="p">()</span><span class="x"> </span><span class="kt">error</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">exitMsg</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Fprintln</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="p">,</span><span class="x"> </span><span class="s">"the connection was closed on the remote side on "</span><span class="p">,</span><span class="x"> </span><span class="n">time</span><span class="o">.</span><span class="n">Now</span><span class="p">()</span><span class="o">.</span><span class="n">Format</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">RFC822</span><span class="p">))</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">else</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Fprintln</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="p">,</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">exitMsg</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}()</span><span class="x"> </span><span class="n">fd</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="kt">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Fd</span><span class="p">())</span><span class="x"> </span><span class="n">state</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">MakeRaw</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">Restore</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span><span class="x"> </span><span class="n">state</span><span class="p">)</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">terminal</span><span class="o">.</span><span class="n">GetSize</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">termType</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Getenv</span><span class="p">(</span><span class="s">"TERM"</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">termType</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">termType</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="s">"xterm-256color"</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">RequestPty</span><span class="p">(</span><span class="n">termType</span><span class="p">,</span><span class="x"> </span><span class="n">termHeight</span><span class="p">,</span><span class="x"> </span><span class="n">termWidth</span><span class="p">,</span><span class="x"> </span><span class="n">ssh</span><span class="o">.</span><span class="n">TerminalModes</span><span class="p">{})</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">updateTerminalSize</span><span class="p">()</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stdin</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">StdinPipe</span><span class="p">()</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">StdoutPipe</span><span class="p">()</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">StderrPipe</span><span class="p">()</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stderr</span><span class="p">,</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="n">io</span><span class="o">.</span><span class="n">Copy</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">Stdout</span><span class="p">,</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span><span class="x"> </span><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">buf</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">([]</span><span class="kt">byte</span><span class="p">,</span><span class="x"> </span><span class="m">128</span><span class="p">)</span><span class="x"> </span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">n</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Stdin</span><span class="o">.</span><span class="n">Read</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">n</span><span class="x"> </span><span class="o">&gt;</span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">buf</span><span class="p">[</span><span class="o">:</span><span class="n">n</span><span class="p">])</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">exitMsg</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">()</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="p">}()</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">Shell</span><span class="p">()</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">t</span><span class="o">.</span><span class="n">Session</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="n">New</span><span class="p">(</span><span class="n">client</span><span class="x"> </span><span class="o">*</span><span class="n">ssh</span><span class="o">.</span><span class="n">Client</span><span class="p">)</span><span class="x"> </span><span class="kt">error</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">session</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">client</span><span class="o">.</span><span class="n">NewSession</span><span class="p">()</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">defer</span><span class="x"> </span><span class="n">session</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">SSHTerminal</span><span class="p">{</span><span class="x"> </span><span class="n">Session</span><span class="o">:</span><span class="x"> </span><span class="n">session</span><span class="p">,</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">return</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">interactiveSession</span><span class="p">()</span><span class="x"> </span><span class="p">}</span><span class="x"> </span></code></pre></div></div> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Fri, 09 Nov 2018 23:13:44 +0800 /2018/11/09/go-interactive-shell/ /2018/11/09/go-interactive-shell/ Golang Golang Go 代码的扩展套路 <blockquote> <p>折腾 Go 已经有一段时间了,最近在用 Go 写点 web 的东西;在搭建脚手架的过程中总是有点不适应,尤其对可扩展性上总是感觉没有 Java 那么顺手;索性看了下 coredns 的源码,最后追踪到 caddy 源码;突然发现他们对代码内的 plugin 机制有一些骚套路,这里索性记录一下</p> </blockquote> <h3 id="一问题由来">一、问题由来</h3> <p>纵观现在所有的 Go web 框架,在文档上可以看到使用方式很简明;非常符合我对 Go 的一贯感受: “所写即所得”;就拿 Gin 这个来说,在 README.md 上可以很轻松的看到 <code class="highlighter-rouge">engine</code> 或者说 <code class="highlighter-rouge">router</code> 这玩意的使用,比如下面这样:</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="c">// Disable Console Color</span><span class="x"> </span><span class="c">// gin.DisableConsoleColor()</span><span class="x"> </span><span class="c">// Creates a gin router with default middleware:</span><span class="x"> </span><span class="c">// logger and recovery (crash-free) middleware</span><span class="x"> </span><span class="n">router</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">gin</span><span class="o">.</span><span class="n">Default</span><span class="p">()</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">GET</span><span class="p">(</span><span class="s">"/someGet"</span><span class="p">,</span><span class="x"> </span><span class="n">getting</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">POST</span><span class="p">(</span><span class="s">"/somePost"</span><span class="p">,</span><span class="x"> </span><span class="n">posting</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">PUT</span><span class="p">(</span><span class="s">"/somePut"</span><span class="p">,</span><span class="x"> </span><span class="n">putting</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">DELETE</span><span class="p">(</span><span class="s">"/someDelete"</span><span class="p">,</span><span class="x"> </span><span class="n">deleting</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">PATCH</span><span class="p">(</span><span class="s">"/somePatch"</span><span class="p">,</span><span class="x"> </span><span class="n">patching</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">HEAD</span><span class="p">(</span><span class="s">"/someHead"</span><span class="p">,</span><span class="x"> </span><span class="n">head</span><span class="p">)</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">OPTIONS</span><span class="p">(</span><span class="s">"/someOptions"</span><span class="p">,</span><span class="x"> </span><span class="n">options</span><span class="p">)</span><span class="x"> </span><span class="c">// By default it serves on :8080 unless a</span><span class="x"> </span><span class="c">// PORT environment variable was defined.</span><span class="x"> </span><span class="n">router</span><span class="o">.</span><span class="n">Run</span><span class="p">()</span><span class="x"> </span><span class="c">// router.Run(":3000") for a hard coded port</span><span class="x"> </span><span class="p">}</span><span class="x"> </span></code></pre></div></div> <p>乍一看简单到爆,但实际使用中,在脚手架搭建上,我们需要规划好 <strong>包结构、配置文件、命令行参数、数据库连接、cache</strong> 等等;直到目前为止,至少我没有找到一种非常规范的后端 MVC 的标准架子结构;这点目前确实不如 Java 的生态;作为最初的脚手架搭建者,站在这个角度,我想我们更应当考虑如何做好适当的抽象、隔离;以防止后面开发者对系统基础功能可能造成的破坏。</p> <p>综上所述,再配合 Gin 或者说 Go 的代码风格,这就形成了一种强烈的冲突;在 Java 中,由于有注解(<code class="highlighter-rouge">Annotation</code>)的存在,事实上你是可以有这种操作的: <strong>新建一个 Class,创建 func,在上面加上合适的注解,最终框架会通过注解扫描的方式以适当的形式进行初始化</strong>;而 Go 中并没有 <code class="highlighter-rouge">Annotation</code> 这玩意,我们很难实现在 <strong>代码运行时扫描自身做出一种策略性调整</strong>;从而下面这个需求很难实现: <strong>作为脚手架搭建者,我希望我的基础代码安全的放在一个特定位置,后续开发者开发应当以一种类似可热插拔的形式注入进来</strong>,比如 Gin 的 router 路由设置,我不希望每次有修改都会有人动我的 router 核心配置文件。</p> <h3 id="二caddy-的套路">二、Caddy 的套路</h3> <p>在翻了 coredns 的源码后,我发现他是依赖于 Caddy 这框架运行的,coredns 的代码内的插件机制也是直接调用的 Caddy;所以接着我就翻到了 Caddy 源码,其中的代码如下(完整代码<a href="http://github.com/mholt/caddy/blob/master/plugins.go">点击这里</a>):</p> <div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// RegisterPlugin plugs in plugin. All plugins should register</span><span class="x"> </span><span class="c">// themselves, even if they do not perform an action associated</span><span class="x"> </span><span class="c">// with a directive. It is important for the process to know</span><span class="x"> </span><span class="c">// which plugins are available.</span><span class="x"> </span><span class="c">//</span><span class="x"> </span><span class="c">// The plugin MUST have a name: lower case and one word.</span><span class="x"> </span><span class="c">// If this plugin has an action, it must be the name of</span><span class="x"> </span><span class="c">// the directive that invokes it. A name is always required</span><span class="x"> </span><span class="c">// and must be unique for the server type.</span><span class="x"> </span><span class="k">func</span><span class="x"> </span><span class="n">RegisterPlugin</span><span class="p">(</span><span class="n">name</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">plugin</span><span class="x"> </span><span class="n">Plugin</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">name</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="nb">panic</span><span class="p">(</span><span class="s">"plugin must have a name"</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">ok</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">plugins</span><span class="p">[</span><span class="n">plugin</span><span class="o">.</span><span class="n">ServerType</span><span class="p">];</span><span class="x"> </span><span class="o">!</span><span class="n">ok</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">plugins</span><span class="p">[</span><span class="n">plugin</span><span class="o">.</span><span class="n">ServerType</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="n">Plugin</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="k">if</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">dup</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">plugins</span><span class="p">[</span><span class="n">plugin</span><span class="o">.</span><span class="n">ServerType</span><span class="p">][</span><span class="n">name</span><span class="p">];</span><span class="x"> </span><span class="n">dup</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="nb">panic</span><span class="p">(</span><span class="s">"plugin named "</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="n">name</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="s">" already registered for server type "</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="n">plugin</span><span class="o">.</span><span class="n">ServerType</span><span class="p">)</span><span class="x"> </span><span class="p">}</span><span class="x"> </span><span class="n">plugins</span><span class="p">[</span><span class="n">plugin</span><span class="o">.</span><span class="n">ServerType</span><span class="p">][</span><span class="n">name</span><span class="p">]</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">plugin</span><span class="x"> </span><span class="p">}</span><span class="x"> </span></code></pre></div></div> <p>套路很清奇,为了实现我上面说的那个需求: “后面开发不需要动我核心代码,我还能允许他们动态添加”,Caddy 套路就是<strong>定义一个 map,map 里用于存放一种特定形式的 func,并且暴露出一个方法用于向 map 内添加指定 func,然后在合适的时机遍历这个 map,并执行其中的 func。</strong>这种套路利用了 Go 函数式编程的特性,将行为先存储在容器中,然后后续再去调用这些行为。</p> <h3 id="三总结">三、总结</h3> <p>长篇大论这么久,实际上我也是在一边折腾 Go 的过程中一边总结和对比跟 Java 的差异;在 Java 中扫描自己注解的套路 Go 中没法实现,但是 Go 利用其函数式编程的优势也可以利用一些延迟加载方式实现对应的功能;总结来说,不同语言有其自己的特性,当有对比的时候,可能更加深刻。</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 23 Oct 2018 21:32:13 +0800 /2018/10/23/golang-code-plugin/ /2018/10/23/golang-code-plugin/ Golang Golang Google container registry 同步 <h2 id="一起因">一、起因</h2> <p>玩 Kubenretes 的基本都很清楚,Kubernetes 很多组件的镜像全部托管在 <code class="highlighter-rouge">gcr.io</code> 这个域名下(现在换成了 <code class="highlighter-rouge">k8s.gcr.io</code>);由于众所周知的原因,这个网站在国内是不可达的;当时由于 Docker Hub 提供了 <code class="highlighter-rouge">Auto Build</code> 功能,机智的想到一个解决办法;就是利用 Docker Hub 的 <code class="highlighter-rouge">Auto Build</code>,创建只有一行的 Dockerfile,里面就一句 <code class="highlighter-rouge">FROM gcr.io/xxxx</code>,然后让 Docker Hub 帮你构建完成后拉取即可</p> <p>这种套路的基本方案就是利用一个第三方公共仓库,这个仓库可以访问不可达的 <code class="highlighter-rouge">gcr.io</code>,然后生成镜像,我们再从这个仓库 pull 即可;为此我创建了一个 Github 仓库(<a href="http://github.com/mritd/docker-library">docker-library</a>);时隔这么久以后,我猜想大家都已经有了这种自己的仓库…不过最近发现这个仓库仍然在有人 fork…</p> <p>为了一劳永逸的解决这个问题,只能撸点代码解决这个问题了</p> <h2 id="二仓库使用">二、仓库使用</h2> <p>为了解决上述问题,我写了一个 <a href="http://github.com/mritd/gcrsync">gcrsync</a> 工具,并且借助 <a href="http://travis-ci.org/mritd/gcrsync">Travis CI</a> 让其每天自动运行,将所有用得到的 <code class="highlighter-rouge">gcr.io</code> 下的镜像同步到了 Docker Hub</p> <p><strong>目前对于一个 <code class="highlighter-rouge">gcr.io</code> 下的镜像,可以直接替换为 <code class="highlighter-rouge">gcrxio</code> 用户名,然后从 Docker Hub 直接拉取</strong>,以下为一个示例:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 原始命令</span> docker pull k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0 <span class="c"># 使用同步仓库</span> docker pull gcrxio/kubernetes-dashboard-amd64:v1.10.0 </code></pre></div></div> <h2 id="三同步细节说明">三、同步细节说明</h2> <p>为了保证同步镜像的安全性,同步工具已经开源在 <a href="http://github.com/mritd/gcrsync">gcrsync</a> 仓库,同步细节如下:</p> <ul> <li>工具每天由 <a href="http://travis-ci.org/mritd/gcrsync">Travis CI</a> 自动进行一次 build,然后进行推送</li> <li>工具每次推送前首先 clone 元数据仓库 <a href="http://github.com/mritd/gcr">gcr</a></li> <li>工具每次推送首先获取 <code class="highlighter-rouge">gcr.io</code> 指定 <code class="highlighter-rouge">namespace</code> 下的所有镜像(<code class="highlighter-rouge">namesapce</code> 由 <a href="http://github.com/mritd/gcrsync/blob/master/.travis.yml">.travis.yml</a> <code class="highlighter-rouge">script</code> 段定义)</li> <li>获取 <code class="highlighter-rouge">gcr.io</code> 镜像后,再读取元数据仓库(<a href="http://github.com/mritd/gcr">gcr</a>) 中与 <code class="highlighter-rouge">namesapce</code> 同名文件(实际是个 json)</li> <li>接着对比双方差异,得出需要同步的镜像</li> <li>最后通过 API 调用本地的 docker 进行 <code class="highlighter-rouge">pull</code>、<code class="highlighter-rouge">tag</code>、<code class="highlighter-rouge">push</code> 操作,完成镜像推送</li> <li>所有镜像推送成功后,更新元数据仓库内 <code class="highlighter-rouge">namespace</code> 对应的 json 文件,最后在生成 <a href="http://github.com/mritd/gcr/blob/master/CHANGELOG.md">CHANGELOG</a>,执行 <code class="highlighter-rouge">git push</code> 到远程元数据仓库</li> </ul> <p>综上所述,如果想得知<strong>具体 <code class="highlighter-rouge">gcrxio</code> 用户下都有那些镜像,可直接访问 <a href="http://github.com/mritd/gcr">gcr</a> 元数据仓库,查看对应 <code class="highlighter-rouge">namesapce</code> 同名的 json 文件即可;每天增量同步的信息会追加到 <a href="http://github.com/mritd/gcr">gcr</a> 仓库的 <code class="highlighter-rouge">CHANGELOG.md</code> 文件中</strong></p> <h2 id="四gcrsync">四、gcrsync</h2> <p>为方便审查镜像安全性,以下为 <a href="http://github.com/mritd/gcrsync">gcrsync</a> 工具的代码简介,代码仓库文件如下:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ gcrsync git:<span class="o">(</span>master<span class="o">)</span> tree <span class="nt">-I</span> vendor <span class="nb">.</span> ├── CHANGELOG.md ├── Gopkg.lock ├── Gopkg.toml ├── LICENSE ├── README.md ├── cmd │   ├── compare.go │   ├── monitor.go │   ├── root.go │   ├── sync.go │   └── test.go ├── dist │   ├── gcrsync_darwin_amd64 │   ├── gcrsync_linux_386 │   └── gcrsync_linux_amd64 ├── main.go └── pkg ├── gcrsync │   ├── docker.go │   ├── gcr.go │   ├── git.go │   ├── registry.go │   └── sync.go └── utils └── common.go </code></pre></div></div> <p>cmd 目录下为标准的 <code class="highlighter-rouge">cobra</code> 框架生成的子命令文件,其中每个命令包含了对应的 flag 设置,如 <code class="highlighter-rouge">namesapce</code>、<code class="highlighter-rouge">proxy</code> 等;<code class="highlighter-rouge">pkg/gcrsync</code> 目录下的文件为核心代码:</p> <ul> <li><code class="highlighter-rouge">docker.go</code> 包含了对本地 docker daemon API 调用,包括 <code class="highlighter-rouge">pull</code>、<code class="highlighter-rouge">tag</code>、<code class="highlighter-rouge">push</code> 操作</li> <li><code class="highlighter-rouge">gcr.go</code> 包含了对 <code class="highlighter-rouge">gcr.io</code> 指定 <code class="highlighter-rouge">namespace</code> 下镜像列表获取操作</li> <li><code class="highlighter-rouge">registry.go</code> 包含了对 Docker Hub 下指定用户(默认 <code class="highlighter-rouge">gcrxio</code>)的镜像列表获取操作(其主要用于首次执行 <code class="highlighter-rouge">compare</code> 命令生成 json 文件)</li> <li><code class="highlighter-rouge">sync.go</code> 为主要的程序入口,其中包含了对其他文件内方法的调用,设置并发池等</li> </ul> <h2 id="五其他说明">五、其他说明</h2> <p>该仓库不保证镜像实时同步,默认每天同步一次(由 <a href="http://travis-ci.org/mritd/gcrsync">Travis CI</a> 执行),如有特殊需求,如增加 <code class="highlighter-rouge">namesapce</code> 等请开启 issue;最后,请不要再 fork <a href="http://github.com/mritd/docker-library">docker-library</a> 这个仓库了</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Mon, 17 Sep 2018 21:19:40 +0800 /2018/09/17/google-container-registry-sync/ /2018/09/17/google-container-registry-sync/ Kubernetes Docker Kubernetes 使用 Bootstrap Token 完成 TLS Bootstrapping <blockquote> <p>最近在测试 Kubernetes 1.11.2 新版本的相关东西,发现新版本的 Bootstrap Token 功能已经进入 Beta 阶段,索性便尝试了一下;虽说目前是为 kubeadm 设计的,不过手动挡用起来也不错,这里记录一下使用方式</p> </blockquote> <h2 id="一环境准备">一、环境准备</h2> <p>首先需要有一个运行状态正常的 Master 节点,目前我测试的是版本是 1.11.2,低版本我没测试;其次本文默认 Node 节点 Docker、kubelet 二进制文件、systemd service 配置等都已经处理好,更具体的环境如下:</p> <p><strong>Master 节点 IP 为 <code class="highlighter-rouge">192.168.1.61</code>,Node 节点 IP 为 <code class="highlighter-rouge">192.168.1.64</code></strong></p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker1.node ➜ ~ kubectl version Client Version: version.Info<span class="o">{</span>Major:<span class="s2">"1"</span>, Minor:<span class="s2">"11"</span>, GitVersion:<span class="s2">"v1.11.2"</span>, GitCommit:<span class="s2">"bb9ffb1654d4a729bb4cec18ff088eacc153c239"</span>, GitTreeState:<span class="s2">"clean"</span>, BuildDate:<span class="s2">"2018-08-07T23:08:19Z"</span>, GoVersion:<span class="s2">"go1.10.3"</span>, Compiler:<span class="s2">"gc"</span>, Platform:<span class="s2">"linux/amd64"</span><span class="o">}</span> Server Version: version.Info<span class="o">{</span>Major:<span class="s2">"1"</span>, Minor:<span class="s2">"11"</span>, GitVersion:<span class="s2">"v1.11.2"</span>, GitCommit:<span class="s2">"bb9ffb1654d4a729bb4cec18ff088eacc153c239"</span>, GitTreeState:<span class="s2">"clean"</span>, BuildDate:<span class="s2">"2018-08-07T23:08:19Z"</span>, GoVersion:<span class="s2">"go1.10.3"</span>, Compiler:<span class="s2">"gc"</span>, Platform:<span class="s2">"linux/amd64"</span><span class="o">}</span> docker1.node ➜ ~ docker info Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 18.06.1-ce Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: <span class="nb">true </span>Native Overlay Diff: <span class="nb">true </span>Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: <span class="nb">local </span>Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e runc version: 69663f0bd4b60df09991c08812a60108003fa340 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.15.0-33-generic Operating System: Ubuntu 18.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.847GiB Name: docker1.node ID: AJOD:RBJZ:YP3G:HCGV:KT4R:D4AF:SBDN:5B76:JM4M:OCJA:YJMJ:OCYQ Docker Root Dir: /data/docker Debug Mode <span class="o">(</span>client<span class="o">)</span>: <span class="nb">false </span>Debug Mode <span class="o">(</span>server<span class="o">)</span>: <span class="nb">false </span>Registry: http://index.docker.io/v1/ Labels: Experimental: <span class="nb">false </span>Insecure Registries: 127.0.0.0/8 Live Restore Enabled: <span class="nb">false</span> </code></pre></div></div> <h2 id="二tls-bootstrapping-回顾">二、TLS Bootstrapping 回顾</h2> <p>在正式进行 TLS Bootstrapping 操作之前,<strong>如果对 TLS Bootstrapping 完全没接触过的请先阅读 <a href="/2018/01/07/kubernetes-tls-bootstrapping-note">Kubernetes TLS bootstrapping 那点事</a></strong>;我想这里有必要简单说明下使用 Token 时整个启动引导过程:</p> <ul> <li>在集群内创建特定的 <code class="highlighter-rouge">Bootstrap Token Secret</code>,该 Secret 将替代以前的 <code class="highlighter-rouge">token.csv</code> 内置用户声明文件</li> <li>在集群内创建首次 TLS Bootstrap 申请证书的 ClusterRole、后续 renew Kubelet client/server 的 ClusterRole,以及其相关对应的 ClusterRoleBinding;并绑定到对应的组或用户</li> <li>调整 Controller Manager 配置,以使其能自动签署相关证书和自动清理过期的 TLS Bootstrapping Token</li> <li>生成特定的包含 TLS Bootstrapping Token 的 <code class="highlighter-rouge">bootstrap.kubeconfig</code> 以供 kubelet 启动时使用</li> <li>调整 Kubelet 配置,使其首次启动加载 <code class="highlighter-rouge">bootstrap.kubeconfig</code> 并使用其中的 TLS Bootstrapping Token 完成首次证书申请</li> <li>证书被 Controller Manager 签署,成功下发,Kubelet 自动重载完成引导流程</li> <li>后续 Kubelet 自动 renew 相关证书</li> <li>可选的: 集群搭建成功后立即清除 <code class="highlighter-rouge">Bootstrap Token Secret</code>,或等待 Controller Manager 待其过期后删除,以防止被恶意利用</li> </ul> <h2 id="三使用-bootstrap-token">三、使用 Bootstrap Token</h2> <p>第二部分算作大纲了,这部分将会按照第二部分的总体流程来走,同时会对一些细节进行详细说明</p> <h3 id="31创建-bootstrap-token">3.1、创建 Bootstrap Token</h3> <p>既然整个功能都时刻强调这个 Token,那么第一步肯定是生成一个 token,生成方式如下:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>➜ ~ <span class="nb">echo</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">head</span> <span class="nt">-c</span> 6 /dev/urandom | <span class="nb">md5sum</span> | <span class="nb">head</span> <span class="nt">-c</span> 6<span class="k">)</span><span class="s2">"</span>.<span class="s2">"</span><span class="k">$(</span><span class="nb">head</span> <span class="nt">-c</span> 16 /dev/urandom | <span class="nb">md5sum</span> | <span class="nb">head</span> <span class="nt">-c</span> 16<span class="k">)</span><span class="s2">"</span> 47f392.d22d04e89a65eb22 </code></pre></div></div> <p>这个 <code class="highlighter-rouge">47f392.d22d04e89a65eb22</code> 就是生成的 Bootstrap Token,保存好 token,因为后续要用;关于这个 token 解释如下:</p> <p>Token 必须满足 <code class="highlighter-rouge">[a-z0-9]{6}\.[a-z0-9]{16}</code> 格式;以 <code class="highlighter-rouge">.</code> 分割,前面的部分被称作 <code class="highlighter-rouge">Token ID</code>,<code class="highlighter-rouge">Token ID</code> 并不是 “机密信息”,它可以暴露出去;相对的后面的部分称为 <code class="highlighter-rouge">Token Secret</code>,它应该是保密的</p> <p>本部分官方文档地址 <a href="http://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/#token-format">Token Format</a></p> <h3 id="32创建-bootstrap-token-secret">3.2、创建 Bootstrap Token Secret</h3> <p>对于 Kubernetes 来说 <code class="highlighter-rouge">Bootstrap Token Secret</code> 也仅仅是一个特殊的 <code class="highlighter-rouge">Secret</code> 而已;对于这个特殊的 <code class="highlighter-rouge">Secret</code> 样例 yaml 配置如下:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span> <span class="na">metadata</span><span class="pi">:</span> <span class="c1"># Name MUST be of form "bootstrap-token-&lt;token id&gt;"</span> <span class="na">name</span><span class="pi">:</span> <span class="s">bootstrap-token-07401b</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">kube-system</span> <span class="c1"># Type MUST be 'bootstrap.kubernetes.io/token'</span> <span class="na">type</span><span class="pi">:</span> <span class="s">bootstrap.kubernetes.io/token</span> <span class="na">stringData</span><span class="pi">:</span> <span class="c1"># Human readable description. Optional.</span> <span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">The</span><span class="nv"> </span><span class="s">default</span><span class="nv"> </span><span class="s">bootstrap</span><span class="nv"> </span><span class="s">token</span><span class="nv"> </span><span class="s">generated</span><span class="nv"> </span><span class="s">by</span><span class="nv"> </span><span class="s">'kubeadm</span><span class="nv"> </span><span class="s">init'."</span> <span class="c1"># Token ID and secret. Required.</span> <span class="na">token-id</span><span class="pi">:</span> <span class="s">47f392</span> <span class="na">token-secret</span><span class="pi">:</span> <span class="s">d22d04e89a65eb22</span> <span class="c1"># Expiration. Optional.</span> <span class="na">expiration</span><span class="pi">:</span> <span class="s">2018-09-10T00:00:11Z</span> <span class="c1"># Allowed usages.</span> <span class="na">usage-bootstrap-authentication</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="na">usage-bootstrap-signing</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span> <span class="c1"># Extra groups to authenticate the token as. Must start with "system:bootstrappers:"</span> <span class="na">auth-extra-groups</span><span class="pi">:</span> <span class="s">system:bootstrappers:worker,system:bootstrappers:ingress</span> </code></pre></div></div> <p>需要注意几点:</p> <ul> <li>作为 <code class="highlighter-rouge">Bootstrap Token Secret</code> 的 type 必须为 <code class="highlighter-rouge">bootstrap.kubernetes.io/token</code>,name 必须为 <code class="highlighter-rouge">bootstrap-token-&lt;token id&gt;</code> (Token ID 就是上一步创建的 Token 前一部分)</li> <li><code class="highlighter-rouge">usage-bootstrap-authentication</code>、<code class="highlighter-rouge">usage-bootstrap-signing</code> 必须存才且设置为 <code class="highlighter-rouge">true</code> (我个人感觉 <code class="highlighter-rouge">usage-bootstrap-signing</code> 可以没有,具体见文章最后部分)</li> <li><code class="highlighter-rouge">expiration</code> 字段是可选的,如果设置则 <code class="highlighter-rouge">Secret</code> 到期后将由 Controller Manager 中的 <code class="highlighter-rouge">tokencleaner</code> 自动清理</li> <li><code class="highlighter-rouge">auth-extra-groups</code> 也是可选的,令牌的扩展认证组,组必须以 <code class="highlighter-rouge">system:bootstrappers:</code> 开头</li> </ul> <p>最后使用 <code class="highlighter-rouge">kubectl create -f bootstrap.secret.yaml</code> 创建即可</p> <p>本部分官方文档地址 <a href="http://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/#bootstrap-token-secret-format">Bootstrap Token Secret Format</a></p> <h3 id="33创建-clusterrole-和-clusterrolebinding">3.3、创建 ClusterRole 和 ClusterRoleBinding</h3> <p>具体都有哪些 <code class="highlighter-rouge">ClusterRole</code> 和 <code class="highlighter-rouge">ClusterRoleBinding</code>,以及其作用请参考上一篇的 <a href="/2018/01/07/kubernetes-tls-bootstrapping-note">Kubernetes TLS bootstrapping 那点事</a>,不想在这里重复了</p> <p>在 1.8 以后三个 <code class="highlighter-rouge">ClusterRole</code> 中有两个已经有了,我们只需要创建剩下的一个即可:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># A ClusterRole which instructs the CSR approver to approve a node requesting a</span> <span class="c1"># serving cert matching its client cert.</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">system:certificates.k8s.io:certificatesigningrequests:selfnodeserver</span> <span class="na">rules</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">apiGroups</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">certificates.k8s.io"</span><span class="pi">]</span> <span class="na">resources</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">certificatesigningrequests/selfnodeserver"</span><span class="pi">]</span> <span class="na">verbs</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">create"</span><span class="pi">]</span> </code></pre></div></div> <p>然后是三个 <code class="highlighter-rouge">ClusterRole</code> 对应的 <code class="highlighter-rouge">ClusterRoleBinding</code>;需要注意的是 <strong>在使用 <code class="highlighter-rouge">Bootstrap Token</code> 进行引导时,Kubelet 组件使用 Token 发起的请求其用户名为 <code class="highlighter-rouge">system:bootstrap:&lt;token id&gt;</code>,用户组为 <code class="highlighter-rouge">system:bootstrappers</code>;so 我们在创建 <code class="highlighter-rouge">ClusterRoleBinding</code> 时要绑定到这个用户或者组上</strong>;当然我选择懒一点,全部绑定到组上</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 允许 system:bootstrappers 组用户创建 CSR 请求</span> kubectl create clusterrolebinding kubelet-bootstrap <span class="nt">--clusterrole</span><span class="o">=</span>system:node-bootstrapper <span class="nt">--group</span><span class="o">=</span>system:bootstrappers <span class="c"># 自动批准 system:bootstrappers 组用户 TLS bootstrapping 首次申请证书的 CSR 请求</span> kubectl create clusterrolebinding node-client-auto-approve-csr <span class="nt">--clusterrole</span><span class="o">=</span>system:certificates.k8s.io:certificatesigningrequests:nodeclient <span class="nt">--group</span><span class="o">=</span>system:bootstrappers <span class="c"># 自动批准 system:nodes 组用户更新 kubelet 自身与 apiserver 通讯证书的 CSR 请求</span> kubectl create clusterrolebinding node-client-auto-renew-crt <span class="nt">--clusterrole</span><span class="o">=</span>system:certificates.k8s.io:certificatesigningrequests:selfnodeclient <span class="nt">--group</span><span class="o">=</span>system:nodes <span class="c"># 自动批准 system:nodes 组用户更新 kubelet 10250 api 端口证书的 CSR 请求</span> kubectl create clusterrolebinding node-server-auto-renew-crt <span class="nt">--clusterrole</span><span class="o">=</span>system:certificates.k8s.io:certificatesigningrequests:selfnodeserver <span class="nt">--group</span><span class="o">=</span>system:nodes </code></pre></div></div> <p>关于本部分首次请求用户名变为 <code class="highlighter-rouge">system:bootstrap:&lt;token id&gt;</code> 官方文档原文如下:</p> <blockquote> <p>Tokens authenticate as the username system:bootstrap:<token> and are members of the group system:bootstrappers. Additional groups may be specified in the token’s Secret.</token></p> </blockquote> <h3 id="34调整-controller-manager">3.4、调整 Controller Manager</h3> <p>根据官方文档描述,Controller Manager 需要启用 <code class="highlighter-rouge">tokencleaner</code> 和 <code class="highlighter-rouge">bootstrapsigner</code> (目测这个 <code class="highlighter-rouge">bootstrapsigner</code> 实际上并不需要,顺便加着吧),完整配置如下(为什么贴完整配置? 文章凑数啊…):</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">KUBE_CONTROLLER_MANAGER_ARGS</span><span class="o">=</span><span class="s2">" --address=127.0.0.1 </span><span class="se">\</span><span class="s2"> --bind-address=192.168.1.61 </span><span class="se">\</span><span class="s2"> --port=10252 </span><span class="se">\</span><span class="s2"> --secure-port=10258 </span><span class="se">\</span><span class="s2"> --cluster-name=kubernetes </span><span class="se">\</span><span class="s2"> --cluster-signing-cert-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --cluster-signing-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem </span><span class="se">\</span><span class="s2"> --controllers=*,bootstrapsigner,tokencleaner </span><span class="se">\</span><span class="s2"> --deployment-controller-sync-period=10s </span><span class="se">\</span><span class="s2"> --experimental-cluster-signing-duration=86700h0m0s </span><span class="se">\</span><span class="s2"> --enable-garbage-collector=true </span><span class="se">\</span><span class="s2"> --leader-elect=true </span><span class="se">\</span><span class="s2"> --master=http://127.0.0.1:8080 </span><span class="se">\</span><span class="s2"> --node-monitor-grace-period=40s </span><span class="se">\</span><span class="s2"> --node-monitor-period=5s </span><span class="se">\</span><span class="s2"> --pod-eviction-timeout=5m0s </span><span class="se">\</span><span class="s2"> --terminated-pod-gc-threshold=50 </span><span class="se">\</span><span class="s2"> --root-ca-file=/etc/kubernetes/ssl/k8s-root-ca.pem </span><span class="se">\</span><span class="s2"> --service-account-private-key-file=/etc/kubernetes/ssl/k8s-root-ca-key.pem </span><span class="se">\</span><span class="s2"> --feature-gates=RotateKubeletServerCertificate=true"</span> </code></pre></div></div> <h3 id="35生成-bootstrapkubeconfig">3.5、生成 bootstrap.kubeconfig</h3> <p>前面所有步骤实际上都是在处理 Api Server、Controller Manager 这一块,为的就是 “老子启动后 TLS Bootstarpping 发证书申请你两个要立马允许,不能拒绝老子”;接下来就是比较重要的 <code class="highlighter-rouge">bootstrap.kubeconfig</code> 配置生成,这个 <code class="highlighter-rouge">bootstrap.kubeconfig</code> 是最终被 Kubelet 使用的,里面包含了相关的 Token,以帮助 Kubelet 在第一次通讯时能成功沟通 Api Server;生成方式如下:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># 设置集群参数</span> kubectl config set-cluster kubernetes <span class="se">\</span> <span class="nt">--certificate-authority</span><span class="o">=</span>/etc/kubernetes/ssl/k8s-root-ca.pem <span class="se">\</span> <span class="nt">--embed-certs</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span> <span class="nt">--server</span><span class="o">=</span>http://127.0.0.1:6443 <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig <span class="c"># 设置客户端认证参数</span> kubectl config set-credentials system:bootstrap:47f392 <span class="se">\</span> <span class="nt">--token</span><span class="o">=</span>47f392.d22d04e89a65eb22 <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig <span class="c"># 设置上下文参数</span> kubectl config set-context default <span class="se">\</span> <span class="nt">--cluster</span><span class="o">=</span>kubernetes <span class="se">\</span> <span class="nt">--user</span><span class="o">=</span>system:bootstrap:47f392 <span class="se">\</span> <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig <span class="c"># 设置默认上下文</span> kubectl config use-context default <span class="nt">--kubeconfig</span><span class="o">=</span>bootstrap.kubeconfig </code></pre></div></div> <h3 id="36调整-kubelet">3.6、调整 Kubelet</h3> <p>Kubelet 启动参数需要做一些相应调整,以使其能正确的使用 <code class="highlighter-rouge">Bootstartp Token</code>,完整配置如下(与使用 token.csv 配置没什么变化,因为主要变更在 bootstrap.kubeconfig 中):</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">KUBELET_ARGS</span><span class="o">=</span><span class="s2">" --address=192.168.1.64 </span><span class="se">\</span><span class="s2"> --allow-privileged=true </span><span class="se">\</span><span class="s2"> --alsologtostderr </span><span class="se">\</span><span class="s2"> --anonymous-auth=true </span><span class="se">\</span><span class="s2"> --bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig </span><span class="se">\</span><span class="s2"> --cert-dir=/etc/kubernetes/ssl </span><span class="se">\</span><span class="s2"> --cgroup-driver=cgroupfs </span><span class="se">\</span><span class="s2"> --cluster-dns=10.254.0.2 </span><span class="se">\</span><span class="s2"> --cluster-domain=cluster.local. </span><span class="se">\</span><span class="s2"> --fail-swap-on=false </span><span class="se">\</span><span class="s2"> --healthz-port=10248 </span><span class="se">\</span><span class="s2"> --healthz-bind-address=192.168.1.64 </span><span class="se">\</span><span class="s2"> --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true </span><span class="se">\</span><span class="s2"> --node-labels=node-role.kubernetes.io/k8s-master=true </span><span class="se">\</span><span class="s2"> --image-gc-high-threshold=70 </span><span class="se">\</span><span class="s2"> --image-gc-low-threshold=50 </span><span class="se">\</span><span class="s2"> --kube-reserved=cpu=500m,memory=512Mi,ephemeral-storage=1Gi </span><span class="se">\</span><span class="s2"> --kubeconfig=/etc/kubernetes/kubelet.kubeconfig </span><span class="se">\</span><span class="s2"> --system-reserved=cpu=1000m,memory=1024Mi,ephemeral-storage=1Gi </span><span class="se">\</span><span class="s2"> --serialize-image-pulls=false </span><span class="se">\</span><span class="s2"> --sync-frequency=30s </span><span class="se">\</span><span class="s2"> --pod-infra-container-image=k8s.gcr.io/pause:3.1 </span><span class="se">\</span><span class="s2"> --resolv-conf=/etc/resolv.conf </span><span class="se">\</span><span class="s2"> --rotate-certificates"</span> </code></pre></div></div> <p><strong>一切准备就绪后,执行 <code class="highlighter-rouge">systemctl daemon-reload &amp;&amp; systemctl start kubelet</code> 启动即可</strong></p> <h2 id="四其他说明">四、其他说明</h2> <p>可能有人已经注意到,在官方文档中最后部分有关于 <a href="http://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/#configmap-signing">ConfigMap Signing</a> 的相关描述,同时要求了启用 <code class="highlighter-rouge">bootstrapsigner</code> 这个 controller,而且在上文创建 <code class="highlighter-rouge">Bootstrap Token Secret</code> 中我也说 <code class="highlighter-rouge">usage-bootstrap-signing</code> 这个可以不设置;其中官方文档上的描述我们能看到的大致只说了这么两段稍微有点用的话:</p> <blockquote> <p>In addition to authentication, the tokens can be used to sign a ConfigMap. This is used early in a cluster bootstrap process before the client trusts the API server. The signed ConfigMap can be authenticated by the shared token.</p> </blockquote> <blockquote> <p>The ConfigMap that is signed is cluster-info in the kube-public namespace. The typical flow is that a client reads this ConfigMap while unauthenticated and ignoring TLS errors. It then validates the payload of the ConfigMap by looking at a signature embedded in the ConfigMap.</p> </blockquote> <p>从这两段话中我们只能得出两个结论:</p> <ul> <li>Bootstrap Token 能对 ConfigMap 签名</li> <li>可以签名一个 <code class="highlighter-rouge">kube-public</code> NameSpace 下的名字叫 <code class="highlighter-rouge">cluster-info</code> 的 ConfigMap,并且这个 ConfigMap 可以在没进行引导之前强行读取</li> </ul> <p>说实话这两段话搞得我百思不得<del>骑姐</del>其解,最终我在 kubeadm 的相关文档中找到了真正的说明及作用:</p> <ul> <li>在使用 <code class="highlighter-rouge">kubeadm init</code> 时创建 <code class="highlighter-rouge">cluster-info</code> 这个 ConfigMap,ConfigMap 中包含了集群基本信息</li> <li>在使用 <code class="highlighter-rouge">kubeadm join</code> 时目标节点强行读取 ConfigMap 以得知集群基本信息,然后进行 <code class="highlighter-rouge">join</code></li> </ul> <p><strong>综上所述,我个人认为手动部署下,在仅仅使用 Bootstrap Token 进行 TLS Bootstrapping 时,<code class="highlighter-rouge">bootstrapsigner</code> 这个 controller 和 <code class="highlighter-rouge">Bootstrap Token Secret</code> 中的 <code class="highlighter-rouge">usage-bootstrap-signing</code> 选项是没有必要的,当然我还没测试(胡吹谁不会)…</strong></p> <p>最后附上 <code class="highlighter-rouge">kubeadm</code> 的文档说明: <a href="http://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#create-the-public-cluster-info-configmap">Create the public cluster-info ConfigMap</a>、<a href="http://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#discovery-cluster-info">Discovery cluster-info</a></p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Tue, 28 Aug 2018 16:54:43 +0800 /2018/08/28/kubernetes-tls-bootstrapping-with-bootstrap-token/ /2018/08/28/kubernetes-tls-bootstrapping-with-bootstrap-token/ Kubernetes Kubernetes Kubernetes 证书配置 <blockquote> <p>一直以来自己的 Kubernetes 集群大部分证书配置全部都在使用一个 CA,而事实上很多教程也没有具体的解释过这些证书代表的作用以及含义;今天索性仔细的翻了翻,顺便看到了一篇老外的文章,感觉写的不错,这里顺带着自己的理解总结一下。</p> </blockquote> <h2 id="一kubernetes-证书分类">一、Kubernetes 证书分类</h2> <p>这里的证书分类只是我自己定义的一种 “并不 ok” 的概念;从整体的作用上 Kubernetes 证书大致上应当分为两类:</p> <ul> <li>API Server 用于校验请求合法性证书</li> <li>对其他敏感信息进行签名的证书(如 Service Account)</li> </ul> <p>对于 API Server 用于检验请求合法性的证书配置一般会在 API Server 中配置好,而对其他敏感信息签名加密的证书一般会可能放在 Controller Manager 中配置,也可能还在 API Server,具体不同版本需要撸文档</p> <p>另外需要明确的是: <strong>Kubernetes 中 CA 证书并不一定只有一个,很多证书配置实际上是不相干的,只是大家为了方便普遍选择了使用一个 CA 进行签发;同时有一些证书如果不设置也会自动默认一个,就目前我所知的大约有 5 个可以完全不同的证书签发体系(或者说由不同的 CA 签发)</strong></p> <h2 id="二api-server-中的证书配置">二、API Server 中的证书配置</h2> <h3 id="21api-server-证书">2.1、API Server 证书</h3> <p>API Server 证书配置中最应当明确的两个选项应该是以下两个:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">--tls-cert-file</span> string File containing the default x509 Certificate <span class="k">for </span>http. <span class="o">(</span>CA cert, <span class="k">if </span>any, concatenated after server cert<span class="o">)</span><span class="nb">.</span> If http serving is enabled, and <span class="nt">--tls-cert-file</span> and <span class="nt">--tls-private-key-file</span> are not provided, a self-signed certificate and key are generated <span class="k">for </span>the public address and saved to the directory specified by <span class="nt">--cert-dir</span><span class="nb">.</span> <span class="nt">--tls-private-key-file</span> string File containing the default x509 private key matching <span class="nt">--tls-cert-file</span><span class="nb">.</span> </code></pre></div></div> <p>从描述上就可以看出,这两个选项配置的就是 API Server http 端点应当使用的证书</p> <h3 id="22client-ca-证书">2.2、Client CA 证书</h3> <p>接下来就是我们常见的 CA 配置:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">--client-ca-file</span> string If <span class="nb">set</span>, any request presenting a client certificate signed by one of the authorities <span class="k">in </span>the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. </code></pre></div></div> <p>该配置明确了 Clent 连接 API Server 时,API Server 应当确保其证书源自哪个 CA 签发;如果其证书不是由该 CA 签发,则拒绝请求;事实上,这个 CA 不必与 http 端点所使用的证书 CA 相同;同时这里的 Client 是一个泛指的,可以是 kubectl,也可能是你自己开发的应用</p> <h3 id="23请求头证书">2.3、请求头证书</h3> <p>由于 API Server 是支持多种认证方式的,其中一种就是使用 HTTP 头中的指定字段来进行认证,相关配置如下:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">--requestheader-allowed-names</span> stringSlice List of client certificate common names to allow to provide usernames <span class="k">in </span>headers specified by <span class="nt">--requestheader-username-headers</span><span class="nb">.</span> If empty, any client certificate validated by the authorities <span class="k">in</span> <span class="nt">--requestheader-client-ca-file</span> is allowed. <span class="nt">--requestheader-client-ca-file</span> string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames <span class="k">in </span>headers specified by <span class="nt">--requestheader-username-headers</span><span class="nb">.</span> WARNING: generally <span class="k">do </span>not depend on authorization being already <span class="k">done for </span>incoming requests. </code></pre></div></div> <p>当指定这个 CA 证书后,则 API Server 使用 HTTP 头进行认证时会检测其 HTTP 头中发送的证书是否由这个 CA 签发;同样它也可独立于其他 CA(可以是个独立的 CA);具体可以参考 <a href="http://kubernetes.io/docs/reference/access-authn-authz/authentication/#authenticating-proxy">Authenticating Proxy</a></p> <h3 id="24kubelet-证书">2.4、Kubelet 证书</h3> <p>对于 Kubelet 组件,API Server 单独提供了证书配置选项,同时 Kubelet 组件也提供了反向设置的相关选项:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># API Server</span> <span class="nt">--kubelet-certificate-authority</span> string Path to a cert file <span class="k">for </span>the certificate authority. <span class="nt">--kubelet-client-certificate</span> string Path to a client cert file <span class="k">for </span>TLS. <span class="nt">--kubelet-client-key</span> string Path to a client key file <span class="k">for </span>TLS. <span class="c"># Kubelet</span> <span class="nt">--client-ca-file</span> string If <span class="nb">set</span>, any request presenting a client certificate signed by one of the authorities <span class="k">in </span>the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. <span class="nt">--tls-cert-file</span> string File containing x509 Certificate used <span class="k">for </span>serving http <span class="o">(</span>with intermediate certs, <span class="k">if </span>any, concatenated after server cert<span class="o">)</span><span class="nb">.</span> If <span class="nt">--tls-cert-file</span> and <span class="nt">--tls-private-key-file</span> are not provided, a self-signed certificate and key are generated <span class="k">for </span>the public address and saved to the directory passed to <span class="nt">--cert-dir</span><span class="nb">.</span> <span class="nt">--tls-private-key-file</span> string File containing x509 private key matching <span class="nt">--tls-cert-file</span><span class="nb">.</span> </code></pre></div></div> <p>相信这个配置不用多说就能猜到,这个就是用于指定 API Server 与 Kubelet 通讯所使用的证书以及其签署的 CA;同样这个 CA 可以完全独立与上述其他CA</p> <h2 id="三service-account-证书">三、Service Account 证书</h2> <p>在 API Server 配置中,对于 Service Account 同样有两个证书配置:</p> <div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">--service-account-key-file</span> stringArray File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple <span class="nb">times </span>with different files. If unspecified, <span class="nt">--tls-private-key-file</span> is used. Must be specified when <span class="nt">--service-account-signing-key</span> is provided <span class="nt">--service-account-signing-key-file</span> string Path to the file that contains the current private key of the service account token issuer. The issuer will sign issued ID tokens with this private key. <span class="o">(</span>Requires the <span class="s1">'TokenRequest'</span> feature gate.<span class="o">)</span> </code></pre></div></div> <p>这两个配置描述了对 Service Account 进行签名验证时所使用的证书;不过需要注意的是这里并没有明确要求证书 CA,所以这两个证书的 CA 理论上也是可以完全独立的;至于未要求 CA 问题,可能是由于 jwt 库并不支持 CA 验证</p> <h2 id="四总结">四、总结</h2> <p>Kubernetes 中大部分证书都是用于 API Server 各种鉴权使用的;在不同鉴权方案或者对象上实际证书体系可以完全不同;具体是使用多个 CA 好还是都用一个,取决于集群规模、安全性要求等等因素,至少目前来说没有明确的那个好与不好</p> <p>最后,嗯…吹牛逼就吹到这,有点晚了,得睡觉了…</p> <p>转载请注明出处,本文采用 <a href="http://creativecommons.org/licenses/by-nc-nd/4.0/">CC4.0</a> 协议授权</p> Sun, 26 Aug 2018 22:54:16 +0800 /2018/08/26/kubernetes-certificate-configuration/ /2018/08/26/kubernetes-certificate-configuration/ Kubernetes Kubernetes