# conda-mirror **Repository Path**: shineboy/conda-mirror ## Basic Information - **Project Name**: conda-mirror - **Description**: No description available - **Primary Language**: Unknown - **License**: BSD-3-Clause - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2022-03-15 - **Last Updated**: 2024-10-13 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # 本项目的部分改动和优化 - 1.添加下载进度条； - 2.添加多线程下载； - 3.添加main、free的channel支持； - 4.取消缓存目录在部分操作删除已下载包 ## 安装 ``` git clone https://github.com/lixiang0/conda-mirror cd conda-mirror python setup.py install ``` ## 使用示例 1.下载目录下创建conf.yaml，内容为： ``` blacklist: - build: 'py2*' whitelist: - build: 'py3*' ``` 释义：屏蔽python2的版本，只下载python3的版本包 2.执行命令： ``` conda-mirror --upstream-channel main --platform linux-64 -vvv --num-threads 12 --temp-directory ./temp/ -k --config conf.yaml conda index ~/files/anaconda-bare/pkgs/main #这里注意要换成自己的目录 ``` 释义： - platform：下载对应平台下的包 - vvv:显示warnning信息 - num-threads下载和验证线程数目 - temp-directory：缓存目录 - k：不使用https - config：包过滤配置 上述命令先是镜像，然后是建立索引。 执行完毕后，新建一个test环境示例： ``` conda create -n test Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /home/ruben/anaconda3/envs/test Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate test # # To deactivate an active environment, use # # $ conda deactivate ``` 运行截图： ![](conda-mirror1.png) ------ 以下为原项目的README.MD # conda-mirror [![Project Status: Inactive – The project has reached a stable, usable state but is no longer being actively developed; support/maintenance will be provided as time allows.](https://www.repostatus.org/badges/latest/inactive.svg)](https://www.repostatus.org/#inactive) [![Build Status](https://travis-ci.org/Valassis-Digital-Media/conda-mirror.svg?branch=master)](https://travis-ci.org/Valassis-Digital-Media/conda-mirror) [![PyPI version](https://badge.fury.io/py/conda-mirror.svg)](https://badge.fury.io/py/conda-mirror) [![codecov](https://codecov.io/gh/Valassis-Digital-Media/conda-mirror/branch/master/graph/badge.svg)](https://codecov.io/gh/Valassis-Digital-Media/conda-mirror) Mirrors an upstream conda channel to a local directory. ## Install `conda-mirror` is available on PyPI and conda-forge. Install with: `pip install conda-mirror` or: `conda install conda-mirror -c conda-forge` ## Compatibility `conda-mirror` is intentionally a py3 only package ## CLI CLI interface for `conda-mirror.py` ``` usage: conda-mirror [-h] [--upstream-channel UPSTREAM_CHANNEL] [--target-directory TARGET_DIRECTORY] [--temp-directory TEMP_DIRECTORY] [--platform PLATFORM] [-v] [--config CONFIG] [--pdb] [--num-threads NUM_THREADS] [--version] [--dry-run] [--no-validate-target] [--minimum-free-space MINIMUM_FREE_SPACE] CLI interface for conda-mirror.py optional arguments: -h, --help show this help message and exit --upstream-channel UPSTREAM_CHANNEL The target channel to mirror. Can be a channel on anaconda.org like "conda-forge" or a full qualified channel like "https://repo.continuum.io/pkgs/free/" --target-directory TARGET_DIRECTORY The place where packages should be mirrored to --temp-directory TEMP_DIRECTORY Temporary download location for the packages. Defaults to a randomly selected temporary directory. Note that you might need to specify a different location if your default temp directory has less available space than your mirroring target --platform PLATFORM The OS platform(s) to mirror. one of: {'linux-64', 'linux-32','osx-64', 'win-32', 'win-64'} -v, --verbose logging defaults to error/exception only. Takes up to three '-v' flags. '-v': warning. '-vv': info. '-vvv': debug. --config CONFIG Path to the yaml config file --pdb Enable PDB debugging on exception --num-threads NUM_THREADS Num of threads for validation. 1: Serial mode. 0: All available. --version Print version and quit --dry-run Show what will be downloaded and what will be removed. Will not validate existing packages --no-validate-target Skip validation of files already present in target- directory --minimum-free-space MINIMUM_FREE_SPACE Threshold for free diskspace. Given in megabytes. ``` ## Example Usage WARNING: Invoking this command will pull ~10GB and take at least an hour `conda-mirror --upstream-channel conda-forge --target-directory local_mirror --platform linux-64` ## More Details ### blacklist/whitelist configuration example-conf.yaml: ```yaml blacklist: - license: "*agpl*" - license: None - license: "" whitelist: - name: system ``` `blacklist` removes package(s) that match the condition(s) listed from the upstream repodata. `whitelist` re-includes any package(s) from blacklist that match the whitelist conditions. blacklist and whitelist both take lists of dictionaries. The keys in the dictionary need to be values in the `repodata.json` metadata. The values are (unix) globs to match on. Go here for the full repodata of the upstream "defaults" channel: http://conda.anaconda.org/anaconda/linux-64/repodata.json Here are the contents of one of the entries in repodata['packages'] ```python {'botocore-1.4.10-py34_0.tar.bz2': {'arch': 'x86_64', 'binstar': {'channel': 'main', 'owner_id': '55fc8527d3234d09d4951c71', 'package_id': '56b88ea1be1cc95a362b218e'}, 'build': 'py34_0', 'build_number': 0, 'date': '2016-04-11', 'depends': ['docutils >=0.10', 'jmespath >=0.7.1,<1.0.0', 'python 3.4*', 'python-dateutil >=2.1,<3.0.0'], 'license': 'Apache', 'md5': 'b35a5c1240ba672e0d9d1296141e383c', 'name': 'botocore', 'platform': 'linux', 'requires': [], 'size': 1831799, 'version': '1.4.10'}} ``` See implementation details in the `conda_mirror:match` function for more information. #### Common usage patterns ##### Mirror **only** one specific package If you wanted to match exactly the botocore package listed above with your config, then you could use the following configuration to first blacklist **all** packages and then include just the botocore packages: ```yaml blacklist: - name: "*" whitelist: - name: botocore version: 1.4.10 build: py34_0 ``` ##### Mirror everything but agpl licenses ```yaml blacklist: - license: "*agpl*" ``` ##### Mirror only python 3 packages ```yaml blacklist: - name: "*" whitelist: - build: "*py3*" ``` ## Testing ### Install test requirements Note: Will install packages from pip ``` $ pip install -r test-requirements.txt Requirement already satisfied: pytest in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 1)) Requirement already satisfied: coverage in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 2)) Requirement already satisfied: pytest-ordering in /home/edill/miniconda/lib/python3.5/site-packages (from -r test-requirements.txt (line 3)) Requirement already satisfied: py>=1.4.29 in /home/edill/miniconda/lib/python3.5/site-packages (from pytest->-r test-requirements.txt (line 1)) ``` ### Run the tests, invoking with the `coverage` tool. ``` $ coverage run run_tests.py sys.argv=['run_tests.py'] ========================================= test session starts ========================================== platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.31, pluggy-0.4.0 -- /home/edill/miniconda/bin/python cachedir: .cache rootdir: /home/edill/dev/maxpoint/github/conda-mirror, inifile: plugins: xonsh-0.5.2, ordering-0.4 collected 4 items test/test_conda_mirror.py::test_match PASSED test/test_conda_mirror.py::test_cli[https://repo.continuum.io/pkgs/free-linux-64] PASSED test/test_conda_mirror.py::test_cli[conda-forge-linux-64] PASSED test/test_conda_mirror.py::test_handling_bad_package PASSED ======================================= 4 passed in 4.41 seconds ======================================= ``` ### Show the coverage statistics ``` $ coverage report -m Name Stmts Miss Cover Missing ------------------------------------------------------------ conda_mirror/__init__.py 3 0 100% conda_mirror/conda_mirror.py 236 20 92% 203-205, 209-210, 214, 240, 249-254, 262-264, 303, 366, 497, 542-543, 629 ------------------------------------------------------------ TOTAL 239 20 92% ``` ## Other After a new contributor makes a pull-request that is approved, we will reach out and invite you to be a maintainer of the project. ## Releasing To release you need three things 1. Commit rights to conda-mirror 2. A github token 3. The version number that you want to use for the new tag After you have all three of these things, run the release.sh script (on a unix machine) and pass it the tag that you want to use and your github token: ```bash GITHUB_TOKEN= ./release.sh ```