4 Star 5 Fork 0

Gitee 极速下载 / nydus

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
此仓库是为了提升国内下载速度的镜像仓库,每日同步一次。 原始仓库: https://github.com/dragonflyoss/image-service
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

[⬇️ Download] [📖 Website] [☸ Quick Start (Kubernetes)] [🤓 Quick Start (nerdctl)] [❓ FAQs & Troubleshooting]

Nydus: Dragonfly Container Image Service

Release Version License Twitter Nydus Stars

Smoke Test Image Conversion Release Test Daily Benchmark Coverage

Introduction

Nydus implements a content-addressable file system on the RAFS format, which enhances the current OCI image specification by improving container launch speed, image space and network bandwidth efficiency, and data integrity.

The following Benchmarking results demonstrate that Nydus images significantly outperform OCI images in terms of container cold startup elapsed time on Containerd, particularly as the OCI image size increases.

Container Cold Startup

Principles

Provide Fast, Secure And Easy Access to Data Distribution

  • Performance: Second-level container startup speed, millisecond-level function computation code package loading speed.
  • Low Cost: Written in memory-safed language Rust, numerous optimizations help improve memory, CPU, and network consumption.
  • Flexible: Supports container runtimes such as runC and Kata, and provides Confidential Containers and vulnerability scanning capabilities
  • Security: End to end data integrity check, Supply Chain Attack can be detected and avoided at runtime.

Key features

  • On-demand Load: Container images/packages are downloaded on-demand in chunk unit to boost startup.
  • Chunk Deduplication: Chunk level data de-duplication cross-layer or cross-image to reduce storage, transport, and memory cost.
  • Compatible with Ecosystem: Storage backend support with Registry, OSS, NAS, Shared Disk, and P2P service. Compatible with the OCI images, and provide native eStargz images support.
  • Data Analyzability: Record accesses, data layout optimization, prefetch, IO amplification, abnormal behavior detection.
  • POSIX Compatibility: In-Kernel EROFS or FUSE filesystems together with overlayfs provide full POSIX compatibility
  • I/O optimization: Use merged filesystem tree, data prefetching and User I/O amplification to reduce read latency and improve user I/O performance.

Ecosystem

Nydus tools

Tool Description
nydusd Nydus user-space daemon, it processes all fscache/FUSE messages from the kernel and parses Nydus images to fullfil those requests
nydus-image Convert a single layer of OCI format container image into a nydus format container image generating meta part file and data part file respectively
nydusify It pulls OCI image down and unpack it, invokes nydus-image create to convert image and then pushes the converted image back to registry and data storage
nydusctl Nydusd CLI client (nydus-image inspect), query daemon's working status/metrics and configure it
ctr-remote An enhanced containerd CLI tool enable nydus support with containerd ctr
nydus-docker-graphdriver [Experimental] Works as a docker remote graph driver to control how images and containers are stored and managed
nydus-overlayfs Containerd mount helper to invoke overlayfs mount with tweaking mount options a bit. So nydus prerequisites can be passed to vm-based runtime
nydus-backend-proxy A simple HTTP server to serve local directory as a blob backend for nydusd

Supported platforms

Type Platform Description Status
Storage Registry/OSS/S3/NAS Support for OCI-compatible distribution implementations such as Docker Hub, Harbor, Github GHCR, Aliyun ACR, NAS, and Aliyun OSS-like object storage service
Storage/Build Harbor Provides a general service for Harbor to support acceleration image conversion based on kinds of accelerator like Nydus and eStargz etc
Distribution Dragonfly Improve the runtime performance of Nydus image even further with the Dragonfly P2P data distribution system
Build Buildkit Provides the ability to build and export Nydus images directly from Dockerfile
Build/Runtime Nerdctl The containerd client to build or run (requires nydus snapshotter) Nydus image
Runtime Docker / Moby Run Nydus image in Docker container with containerd and nydus-snapshotter
Runtime Kubernetes Run Nydus image using CRI interface
Runtime Containerd Nydus Snapshotter, a containerd remote plugin to run Nydus image
Runtime CRI-O / Podman Run Nydus image with CRI-O or Podman 🚧
Runtime KataContainers Run Nydus image in KataContainers as a native solution
Runtime EROFS Run Nydus image directly in-kernel EROFS for even greater performance improvement

Build

Build Binary

# build debug binary
make
# build release binary
make release
# build static binary with docker
make docker-static

Build Nydus Image

Convert OCIv1 image to Nydus image: Nydusify, Acceld or Nerdctl.

Build Nydus image from Dockerfile directly: Buildkit.

Build Nydus layer from various sources: Nydus Image Builder.

Image prefetch optimization

To further reduce container startup time, a nydus image with a prefetch list can be built using the NRI plugin (containerd >=1.7): Container Image Optimizer

Run

Quick Start

For more details on how to lazily start a container with nydus-snapshotter and nydus image on Kubernetes nodes or locally use nerdctl rather than CRI, please refer to Nydus Setup

Run Nydus Snapshotter

Nydus-snapshotter is a non-core sub-project of containerd.

Check out its code and tutorial from Nydus-snapshotter repository. It works as a containerd remote snapshotter to help setup container rootfs with nydus images, which handles nydus image format when necessary. When running without nydus images, it is identical to the containerd's builtin overlayfs snapshotter.

Run Nydusd Daemon

Normally, users do not need to start nydusd by hand. It is started by nydus-snapshotter when a container rootfs is prepared.

Run Nydusd Daemon to serve Nydus image: Nydusd.

Run Nydus with in-kernel EROFS filesystem

In-kernel EROFS has been fully compatible with RAFS v6 image format since Linux 5.16. In other words, uncompressed RAFS v6 images can be mounted over block devices since then.

Since Linux 5.19, EROFS has added a new file-based caching (fscache) backend. In this way, compressed RAFS v6 images can be mounted directly with fscache subsystem, even such images are partially available. estargz can be converted on the fly and mounted in this way too.

Guide to running Nydus with fscache: Nydus-fscache

Run Nydus with Dragonfly P2P system

Nydus is deeply integrated with Dragonfly P2P system, which can greatly reduce the network latency and the single point pressure of the registry server. Benchmarking results in the production environment demonstrate that using Dragonfly can reduce network latency by more than 80%, to understand the performance results and integration steps, please refer to the nydus integration.

If you want to deploy Dragonfly and Nydus at the same time through Helm, please refer to the Quick Start.

Run OCI image directly with Nydus

Nydus is able to generate a tiny artifact called a nydus zran from an existing OCI image in the short time. This artifact can be used to accelerate the container boot time without the need for a full image conversion. For more information, please see the documentation.

Run with Docker(Moby)

Nydus provides a variety of methods to support running on docker(Moby), please refer to Nydus Setup for Docker(Moby) Environment

Run with macOS

Nydus can also run with macfuse(a.k.a osxfuse). For more details please read nydus with macOS.

Run eStargz image (with lazy pulling)

The containerd remote snapshotter plugin nydus-snapshotter can be used to run nydus images, or to run eStargz images directly by appending --enable-stargz command line option.

In the future, zstd::chunked can work in this way as well.

Run Nydus Service

Using the key features of nydus as native in your project without preparing and invoking nydusd deliberately, nydus-service helps to reuse the core services of nyuds.

Documentation

Please visit Wiki, or docs

Community

Nydus aims to form a vendor-neutral opensource image distribution solution to all communities. Questions, bug reports, technical discussion, feature requests and contribution are always welcomed!

We're very pleased to hear your use cases any time. Feel free to reach us via Slack or Dingtalk.

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

Nydus 是 Dragonfly 的容器镜像服务 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Rust
1
https://gitee.com/mirrors/nydus.git
git@gitee.com:mirrors/nydus.git
mirrors
nydus
nydus
master

搜索帮助