7 Star 56 Fork 15

DiDi-opensource / sharingan

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

Sharingan

Build Status Gitter License GoDoc

Sharingan

Sharingan,中文名:写轮眼,是漫画《火影忍者》中的一种瞳术,具有复制、幻术等能力,在幻术世界里,一切因素包括时间,地点,质量等,都由施术者掌控。

一、简介

Sharingan是一个基于golang的流量录制回放工具,录制线上真实请求流量进行回放测试,适合项目重构、回归测试等。

1.1、背景

随着微服务架构的兴起,服务之间的依赖关系变的越来越复杂,软件测试也面临新的挑战:系统升级频繁、服务依赖众多等等。

  • 常见的测试方案(如:单元测试、系统测试等)构造和维护测试用例成本高,特别是业务复杂的场景。「构造测试数据
  • 依赖第三方服务众多,线下测试环境不太稳定,经常出现下游服务不可用导致测试失败的情况发生。「维护测试环境成本

为此,我们需要开发一套工具来缓解上述问题。

1.2、方案

  • 录制线上服务真实请求流量(包括下游调用流量),在线下进行回放,解决构造测试数据难的问题。「复制能力」
  • 回放的时候匹配Mock下游调用,不再依赖具体的下游服务,解决维护测试环境成本高的问题。「幻术能力」

1.3、特性

  • 支持下游流量录制。相比tcpcopygoreplay等方案,回放不依赖下游环境。
  • 支持并发流量录制和回放。录制对服务影响小,回放速度更快。
  • 支持时间重置、噪音去除、批量回放、覆盖率报告、常见协议解析等等。
  • 支持写流量回放,不会污染应用数据。
  • 不依赖业务框架,低应用浸入。

二、快速开始

2.1、使用示例

# Step1: 下载sharingan项目
$ git clone https://github.com/didi/sharingan.git
$ cd sharingan

# Step2: 使用定制版golang,以go1.13为例「慢?科学上网试试」
$ sudo sh install.sh go1.13 # 支持go1.10 ~ go1.15,限mac、linux、amd64「执行出错?请查看底部常见问题」
$ export GOROOT=/tmp/recorder-go1.13
$ export PATH=$GOROOT/bin:$PATH

# Step3: 编译、后台启动replayer-agent「默认会占用3515、8998端口,可修改」
# [回放接入文档]内有直接下载bin文件的链接,无需build
$ cd replayer-agent
$ go build
$ nohup ./replayer-agent >> run.log 2>&1 &

# Step4: 编译、后台启动example示例「默认会占用9999端口,可修改」
$ cd ../example
$ go build -tags="replayer" -gcflags="all=-N -l"
$ nohup ./example >> run.log 2>&1 &

# Step5: 打开回放页面
$ 浏览器打开,http://127.0.0.1:8998 # 非本机替换IP即可
$ 页面选择要回放的流量点执行          # 内置提前录制好的3条example示例流量

2.2、接入文档

三、技术方案

3.1、模块划分

  • recorder: 流量录制模块,录制流量本地文件存储、发送流量到录制agent等。
  • recorder-agent:流量录制agent,单独进程启动,控制录制比例、流量存储等。
  • replayer: 流量回放模块,重定向连接到Mock Server、Mock时间、添加流量标识等。
  • replayer-agent:流量回放agent,单独进程启动,查询流量、查询/上报噪音、流量diff、批量回放、生成覆盖率报告等。

3.2、整体架构图

架构图

3.3、录制方案

  • 修改golang源码,对外暴露Hook接口。「所有改动通过官方测试用例」
  • 提供API串联不同goroutine之间的网络调用。「常见的http、mysql、redis流量都不需要特别设置」
  • 提供单独的agent筛选流量、控制比例。
  • 更多参考:流量录制实现原理

3.4、回放方案

  • 连接重定向:将服务所有Connect网络调用重定向到Mock Server。「安装replayer-agent时候自带」
  • 流量匹配:Mock Server会根据服务真实的下游请求匹配一个返回。「mock下游调用」
  • 时间重置:将程序执行时间回退到录制执行时刻,尽量避免时间因素带来的干扰。
  • 噪音去除:提供API可以将已知的噪音流量去掉,如:traceID,每次请求本来就不一样。
  • 常见协议解析:会解析http、mysql、redis、thrift等协议,方便diff对比。
  • 更多参考:流量回放实现原理

四、演进之路

关于流量录制和回放,在内部进行过多次探索,主要经历下面三个阶段:

4.1、月光宝盒(串行录制、串行回放)

录制:利用tcpdump录制流量,改造router层将请求串行化,利用时间来分割请求。

回放:利用iptables转发流量到mock服务,匹配请求并mock返回。支持时间重置、流量对比等。

不足:录制流量覆盖率低,一次只能录制一个请求。iptables转发,噪音干扰严重。

4.2、Fastdev(并行录制、串行回放)

录制:改造golang源码,利用goroutine+工作委托技术串联区分请求。链路追踪原理

回放:利用mock库重定向connect系统调用,转发流量到mock服务。支持Dashboard、噪音去除等。monkey mock原理

不足:录制接口和实现混合,golang多版本支持困难。不支持并发回放,启动阶段流量无法代理会失败,定时任务流量干扰严重。

4.3、Sharingan(并行录制、并行回放)

录制:接口和实现分离,golang源码改造部分只暴露接口,具体录制实现单独提供包支持。确保golang源码改动通过官方测试,支持1.10~1.14所有版本;优化录制服务性能。

回放:添加流量标识,支持并发回放;支持启动阶段流量代理;利用定制版golang,消除定时任务流量干扰;时间重置不再依赖本地文件,支持replayer-agent单独部署;支持常见协议解析。

五、效果展示

5.1、流量回放

5.1.1、单个回放

单个回放

对于下游请求很多且复杂的情况,支持对下游协议进行筛选 单个回放-协议刷选

5.1.2、批量回放

批量回放的并发度默认是10,可通过增加-parallel参数修改。 批量回放

5.2、覆盖率报告

5.2.1、整体报告

覆盖率报告支持覆盖率结果累计,即支持 多次 单个回放和批量回放后,统一生成覆盖率结果。 整体报告

5.2.1、覆盖详情

覆盖详情

六、更多

6.1、如何贡献

欢迎大家参与进来,更多参考Contribute

6.2、许可

基于Apache-2.0协议进行分发和使用,更多参考LICENSE

6.3、成员

huengyj20060714qiaodandedidibikong0411plpanfzl-yty

6.4、感谢

特别感谢TaoWen ,流量录制和回放初版设计者,为后续开源奠定了很好的基础。

6.5、联系我们

  • 微信交流群 【加管理员微信,拉进交流群】

WEIXIN

七、常见问题

  • 执行 $ sudo sh install.sh go1.13 提示 wget: command not found...

    # 首先安装 brew
    $ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
    # 然后用 `brew` 安装 `wget`:
    $ brew install wget
    # 下载并运行上述连接中安装包后
    $ sudo port install wget
 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright (C) 2017 Beijing Didi Infinity Technology and Development Co.,Ltd. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

Sharingan(写轮眼)是一个基于golang的流量录制回放工具,适合项目重构、回归测试等。 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
1
https://gitee.com/didiopensource/sharingan.git
git@gitee.com:didiopensource/sharingan.git
didiopensource
sharingan
sharingan
master

搜索帮助