diff --git a/docs/mac-user-manual.md b/docs/mac-user-manual.md index f7d5771451beea723b5a921b66090329ebfb7b05..2e7b82e4a9802d930f5cbdc3d15cbcd51426bc88 100644 --- a/docs/mac-user-manual.md +++ b/docs/mac-user-manual.md @@ -58,15 +58,17 @@ brew install wget 2. 配置**EulerLauncher**: - - 查看`qemu`及`wget`所处位置,`qemu`二进制文件在不同架构下名称不同,请根据自身情况选择正确的名称(Apple Silicon: qemu-system-aarch64; Intel: qemu-system-x86_64): + - 查看`qemu`,`qemu-img`及`wget`所处位置,`qemu`二进制文件在不同架构下名称不同,请根据自身情况选择正确的名称(Apple Silicon: qemu-system-aarch64; Intel: qemu-system-x86_64): ``` Shell which wget which qemu-system-{host_arch} + which qemu-img ``` 参考输出: ``` /opt/homebrew/bin/wget /opt/homebrew/bin/qemu-system-aarch64 + /opt/homebrew/bin/qemu-img ``` 查看完成后,记录路径结果,在接下来的步骤中将会使用到。 @@ -82,6 +84,7 @@ brew install wget work_dir = # eulerlauncher工作目录,用于存储虚拟机镜像、虚拟机文件等 wget_dir = # wget的可执行文件路径,请参考上一步的内容进行配置 qemu_dir = # qemu的可执行文件路径,请参考上一步的内容进行配置 + qemu_img_dir = # qemu-img的可执行文件路径,请参考上一步的内容进行配置 debug = True [vm] @@ -130,18 +133,18 @@ eulerlauncher download-image 22.03-LTS Downloading: 22.03-LTS, this might take a while, please check image status with "images" command. ``` -镜像下载请求是一个异步请求,具体的下载动作将在后台完成,具体耗时与你的网络情况相关,整体的镜像下载流程包括下载、解压缩、格式转换等相关子流程,在下载过程中可以通过 `image` 命令随时查看下载进展与镜像状态: +镜像下载请求是一个异步请求,具体的下载动作将在后台完成,具体耗时与你的网络情况相关,整体的镜像下载流程包括下载、解压缩、格式转换等相关子流程,在下载过程中可以通过 `image` 命令随时查看下载进展与镜像状态,进度条格式为`([downloaded_bytes] [percentage] [download_speed] [remaining_download_time])`: ```Shell eulerlauncher images -+-----------+----------+--------------+ -| Images | Location | Status | -+-----------+----------+--------------+ -| 22.03-LTS | Remote | Downloadable | -| 21.09 | Remote | Downloadable | -| 22.03-LTS | Local | Downloading | -+-----------+----------+--------------+ ++-----------+----------+------------------------------------+ +| Images | Location | Status | ++-----------+----------+------------------------------------+ +| 22.03-LTS | Remote | Downloadable | +| 21.09 | Remote | Downloadable | +| 22.03-LTS | Local | Downloading: 33792K 8% 4.88M 55s | ++-----------+----------+------------------------------------+ ``` @@ -167,7 +170,7 @@ eulerlauncher images eulerlauncher load-image --path {image_file_path} IMAGE_NAME ``` -当前支持加载的镜像格式有 `xxx.qcow2.xz`,`xxx.qcow2` +当前支持加载的镜像格式有 `xxx.{qcow2, raw, vmdk, vhd, vhdx, qcow, vdi}.[xz]` 例如: @@ -182,13 +185,13 @@ Loading: 2203-load, this might take a while, please check image status with "ima ```Shell eulerlauncher images -+-----------+----------+--------------+ -| Images | Location | Status | -+-----------+----------+--------------+ -| 22.03-LTS | Remote | Downloadable | -| 21.09 | Remote | Downloadable | -| 2203-load | Local | Loading | -+-----------+----------+--------------+ ++-----------+----------+----------------------------+ +| Images | Location | Status | ++-----------+----------+----------------------------+ +| 22.03-LTS | Remote | Downloadable | +| 21.09 | Remote | Downloadable | +| 2203-load | Local | Loading: (24.00/100%) | ++-----------+----------+----------------------------+ eulerlauncher images @@ -252,6 +255,17 @@ eulerlauncher delete-instance {instance_name} ``` 根据虚拟机名称删除指定的虚拟机。 +5. 为虚拟机打快照,并导出为镜像 +```Shell +eulerlauncher take-snapshot --snapshot_name snap --export_path path vm_name +``` +通过`--snapshot_name`指定快照名称,`--export_path`指定导出镜像存放位置,`vm_name`为虚拟机名。 + +6. 将虚拟机导出为主流编程框架开发镜像 +```Shell +eulerlauncher export-development-image --image_name image --export_path path vm_name +``` +通过`--image_name`指定导出镜像名称,`--export_path`指定导出镜像存放位置,`vm_name`为虚拟机名,默认导出为Python/Go/Java主流编程框架开发镜像。 [1]: https://developer.apple.com/documentation/vmnet [2]: https://gitee.com/openeuler/eulerlauncher/releases \ No newline at end of file diff --git a/docs/win-user-manual.md b/docs/win-user-manual.md index 831f360150423c817c67318ea95d5e56d11e80df..7fafc279029b98e34b9770d1671694af5e8d4a57 100644 --- a/docs/win-user-manual.md +++ b/docs/win-user-manual.md @@ -65,18 +65,18 @@ eulerlauncher download-image 22.03-LTS Downloading: 22.03-LTS, this might take a while, please check image status with "images" command. ``` -镜像下载请求是一个异步请求,具体的下载动作将在后台完成,具体耗时与你的网络情况相关,整体的镜像下载流程包括下载、解压缩、格式转换等相关子流程,在下载过程中可以通过 `image` 命令随时查看下载进展与镜像状态: +镜像下载请求是一个异步请求,具体的下载动作将在后台完成,具体耗时与你的网络情况相关,整体的镜像下载流程包括下载、解压缩、格式转换等相关子流程,在下载过程中可以通过 `image` 命令随时查看下载进展与镜像状态,进度条格式为`[downloaded_bytes]/[total_bytes] (percentage)`: ```PowerShell eulerlauncher images -+-----------+----------+--------------+ -| Images | Location | Status | -+-----------+----------+--------------+ -| 22.03-LTS | Remote | Downloadable | -| 21.09 | Remote | Downloadable | -| 22.03-LTS | Local | Downloading | -+-----------+----------+--------------+ ++-----------+----------+------------------------------------+ +| Images | Location | Status | ++-----------+----------+------------------------------------+ +| 22.03-LTS | Remote | Downloadable | +| 21.09 | Remote | Downloadable | +| 22.03-LTS | Local | Downloading: 97.76/ 386.74MB (25%) | ++-----------+----------+------------------------------------+ ``` @@ -102,7 +102,7 @@ eulerlauncher images eulerlauncher load-image --path {image_file_path} IMAGE_NAME ``` -当前支持加载的镜像格式有 `xxx.qcow2.xz`,`xxx.qcow2` +当前支持加载的镜像格式有 `xxx.{qcow2, raw, vmdk, vhd, vhdx, qcow, vdi}.[xz]` 例如: @@ -117,13 +117,13 @@ Loading: 2203-load, this might take a while, please check image status with "ima ```PowerShell eulerlauncher images -+-----------+----------+--------------+ -| Images | Location | Status | -+-----------+----------+--------------+ -| 22.03-LTS | Remote | Downloadable | -| 21.09 | Remote | Downloadable | -| 2203-load | Local | Loading | -+-----------+----------+--------------+ ++-----------+----------+----------------------------+ +| Images | Location | Status | ++-----------+----------+----------------------------+ +| 22.03-LTS | Remote | Downloadable | +| 21.09 | Remote | Downloadable | +| 2203-load | Local | Loading: (24.00/100%) | ++-----------+----------+----------------------------+ eulerlauncher images @@ -187,5 +187,17 @@ eulerlauncher delete-instance {instance_name} ``` 根据虚拟机名称删除指定的虚拟机。 +5. 为虚拟机打快照,并导出为镜像 +```Shell +eulerlauncher take-snapshot --snapshot_name snap --export_path path vm_name +``` +通过`--snapshot_name`指定快照名称,`--export_path`指定导出镜像存放位置,`vm_name`为虚拟机名。 + +6. 将虚拟机导出为主流编程框架开发镜像 +```Shell +eulerlauncher export-development-image --image_name image --export_path path vm_name +``` +通过`--image_name`指定导出镜像名称,`--export_path`指定导出镜像存放位置,`vm_name`为虚拟机名,默认导出为Python/Go/Java主流编程框架开发镜像。 + [1]: https://gitee.com/openeuler/omnivirt/releases [2]: https://learn.microsoft.com/zh-cn/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v \ No newline at end of file diff --git a/etc/eulerlauncher.conf b/etc/eulerlauncher.conf index a101a510a7e4508fa09797c0573d0299cf152f04..dcac7da0b1b2b715834e34f8cc7d8e341fe07642 100644 --- a/etc/eulerlauncher.conf +++ b/etc/eulerlauncher.conf @@ -3,6 +3,7 @@ log_dir = work_dir = wget_dir = qemu_dir = +qemu_img_dir = debug = True [vm] diff --git a/eulerlauncher/backends/mac/image_handler.py b/eulerlauncher/backends/mac/image_handler.py index f9c8ec6753466f9183fc2d0a6a2fde0e08ac2f1e..122ac766a76348fd30d806c1017c3ef6355af89d 100644 --- a/eulerlauncher/backends/mac/image_handler.py +++ b/eulerlauncher/backends/mac/image_handler.py @@ -24,6 +24,7 @@ class MacImageHandler(object): self.image_record_file = image_record_file self.base_dir = base_dir self.wget_bin = conf.conf.get('default', 'wget_dir') + self.qemu_img_bin = conf.conf.get('default', 'qemu_img_dir') self.LOG = logger @@ -40,8 +41,10 @@ class MacImageHandler(object): images['local'][img_to_download] = img_dict omni_utils.save_json_data(self.image_record_file, images) + download_progress_bar_path = os.path.join(self.image_dir, 'download_progress_bar_' + img_to_download) download_cmd = [self.wget_bin, images['remote'][img_to_download]['path'], - '-O', os.path.join(self.image_dir, img_name), '--no-check-certificate'] + '-O', os.path.join(self.image_dir, img_name), '--no-check-certificate', + '--progress=dot:mega', '-q', '--show-progress', f'-o {download_progress_bar_path}'] self.LOG.debug(' '.join(download_cmd)) subprocess.call(' '.join(download_cmd), shell=True) #wget.download(url=images['remote'][img_to_download]['path'], out=os.path.join(self.image_dir, img_name), bar=None) @@ -59,12 +62,24 @@ class MacImageHandler(object): os.remove(os.path.join(self.image_dir, img_name)) # Record local image + if os.path.exists(os.path.join(self.image_dir, "download_progress_bar_" + img_to_download)): + os.remove(os.path.join(self.image_dir, "download_progress_bar_" + img_to_download)) img_dict['status'] = constants.IMAGE_STATUS_READY img_dict['path'] = os.path.join(self.image_dir, qcow2_name) images['local'][img_to_download] = img_dict omni_utils.save_json_data(self.image_record_file, images) self.LOG.debug(f'Image: {img_to_download} is ready ...') + def download_progress_bar(self, img_name): + progress_bar = [] + progress_bar_path = os.path.join(self.image_dir, 'download_progress_bar_' + img_name) + if os.path.exists(progress_bar_path): + with open(progress_bar_path, "r", encoding=omni_utils.detect_encoding(progress_bar_path), errors='ignore') as progress_bar_file: + progress_bar = progress_bar_file.readlines() + if len(progress_bar) > 1 and progress_bar[-2].strip() != "": + return constants.IMAGE_STATUS_DOWNLOADING + ": " + progress_bar[-2].strip() + else: + return constants.IMAGE_STATUS_DOWNLOADING def delete_image(self, images, img_to_delete): if img_to_delete not in images['local'].keys(): @@ -98,21 +113,44 @@ class MacImageHandler(object): images['local'][image.name] = image.to_dict() omni_utils.save_json_data(self.image_record_file, images) - if fmt == 'qcow2': - qcow2_name = f'{img_to_load}.qcow2' - shutil.copyfile(path, os.path.join(self.image_dir, qcow2_name)) + if fmt not in constants.IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED: + img_name = f'{img_to_load}.{fmt}' + shutil.copyfile(path, os.path.join(self.image_dir, img_name)) else: # Decompress the image self.LOG.debug(f'Decompressing image file: {path} ...') - qcow2_name = f'{img_to_load}.qcow2' - with open(path, 'rb') as pr, open(os.path.join(self.image_dir, qcow2_name), 'wb') as pw: + fmt = fmt.split('.')[0] + img_name = f'{img_to_load}.{fmt}' + with open(path, 'rb') as pr, open(os.path.join(self.image_dir, img_name), 'wb') as pw: data = pr.read() data_dec = lzma.decompress(data) pw.write(data_dec) + # Convert the img to qcow2 + qcow2_name = img_to_load + '.qcow2' + if fmt != "qcow2": + self.LOG.debug(f'Converting image file: {img_name} to {qcow2_name} ...') + load_progress_bar_path = os.path.join(self.image_dir, "load_progress_bar_" + img_to_load) + cmd = 'sudo {0} convert -p -O qcow2 {1} {2} > {3}' + subprocess.call(cmd.format(self.qemu_img_bin, os.path.join(self.image_dir, img_name), os.path.join(self.image_dir, qcow2_name), load_progress_bar_path), shell=True) + self.LOG.debug(f'Cleanup temp files ...') + os.remove(os.path.join(self.image_dir, img_name)) + os.remove(load_progress_bar_path) + # Record local image image.path = os.path.join(self.image_dir, qcow2_name) image.status = constants.IMAGE_STATUS_READY images['local'][image.name] = image.to_dict() omni_utils.save_json_data(self.image_record_file, images) self.LOG.debug(f'Image: {qcow2_name} is ready ...') + + def load_progress_bar(self, img_name): + progress_bar_lines = [] + progress_bar_path = os.path.join(self.image_dir, "load_progress_bar_" + img_name) + if os.path.exists(progress_bar_path): + with open(progress_bar_path, 'r', encoding=omni_utils.detect_encoding(progress_bar_path), errors='ignore') as progress_bar_file: + progress_bar_lines = progress_bar_file.readlines() + if len(progress_bar_lines) > 1 and progress_bar_lines[-2].strip() != "": + return constants.IMAGE_STATUS_LOADING + ": " + progress_bar_lines[-2].strip() + else: + return constants.IMAGE_STATUS_LOADING \ No newline at end of file diff --git a/eulerlauncher/backends/mac/instance_handler.py b/eulerlauncher/backends/mac/instance_handler.py index d9f5ba97ff3624ad12cd572043230343bfdb8ebe..53152dce787a833672a880448cde410961445174 100644 --- a/eulerlauncher/backends/mac/instance_handler.py +++ b/eulerlauncher/backends/mac/instance_handler.py @@ -5,6 +5,7 @@ import signal import subprocess import sys import time +import paramiko from oslo_utils import uuidutils @@ -100,16 +101,21 @@ class MacInstanceHandler(object): except KeyError: return 0 - def create_instance(self, name, image_id, instance_record, all_instances, all_images, arch='x86'): + def create_instance(self, name, image_id, instance_record, all_instances, all_images, is_same, mac_address, uuid, arch='x86'): # Create dir for the instance - vm_uuid = uuidutils.generate_uuid() + if not is_same: + vm_uuid = uuidutils.generate_uuid() + vm_mac_address = omni_utils.generate_mac() + else: + vm_uuid = uuid + vm_mac_address = mac_address vm_dict = { 'name': name, 'uuid': vm_uuid, 'image': image_id, 'vm_state': constants.VM_STATE_MAP[99], 'ip_address': 'N/A', - 'mac_address': omni_utils.generate_mac(), + 'mac_address': vm_mac_address, 'identification': { 'type': 'pid', 'id': None @@ -175,3 +181,53 @@ class MacInstanceHandler(object): return 0 + def _get_vm_img_id_by_name(self, name): + instance = omni_utils.load_json_data(self.instance_record_file)['instances'][name] + return instance['image'] + + def take_snapshot(self, name, snapshot_name, dest_path, all_instances, all_images, instance_record): + vm_img_id = self._get_vm_img_id_by_name(name) + vm_root_disk_src_path = os.path.join(self.instance_dir, name, vm_img_id + '.qcow2') + vm_root_disk_dst_path = os.path.join(self.instance_dir, vm_img_id + '.qcow2') + mac_address = all_instances['instances'][name]['mac_address'] + vm_uuid = all_instances['instances'][name]['uuid'] + # shutdown the vm first for taking snapshot + shutil.copyfile(vm_root_disk_src_path, vm_root_disk_dst_path) + self.delete_instance(name, instance_record, all_instances) + self.driver.take_and_export_snapshot(snapshot_name, vm_root_disk_dst_path, snapshot_name, dest_path) + # a little hack here, since the running vm's image is already deleted, change the local image path to the copyed image path + all_images['local'][vm_img_id]['path'] = vm_root_disk_dst_path + self.create_instance(name, vm_img_id, instance_record, all_instances, all_images, True, mac_address, vm_uuid, 'x86') + os.remove(vm_root_disk_dst_path) + return os.path.join(dest_path, f'{snapshot_name}.qcow2') + + def _get_vm_ip_by_name(self, name): + instance = omni_utils.load_json_data(self.instance_record_file)['instances'][name] + if not instance['ip_address']: + ip_address = self._parse_ip_addr(instance['mac_address']) + else: + ip_address = instance['ip_address'] + return ip_address + + def make_development_image(self, name, pwd): + ssh_client = paramiko.SSHClient() + try: + ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh_client.connect(self._get_vm_ip_by_name(name), 22, "root", pwd) + bash_command = """ + if which apt >/dev/null 2>&1; + then apt install python3-dev golang openjdk-11-jdk + elif which yum >/dev/null 2>&1; + then yum install -y python3-devel golang java-11-openjdk-devel + elif which dnf >/dev/null 2>&1; + then dnf install -y python3-devel golang java-11-openjdk-devel + fi + """ + stdin, stdout, stderr = ssh_client.exec_command(bash_command) + + self.LOG.debug(stdout.read().decode()) + ssh_client.close() + return 0 + except Exception as e: + self.LOG.debug(f"install development environment failed: {str(e)}") + return 1 \ No newline at end of file diff --git a/eulerlauncher/backends/mac/qemu.py b/eulerlauncher/backends/mac/qemu.py index 2bdd900d0db4bd6a6f408db32a47875c535b12b2..1824187704167b2ceaba509ee1e1bb1288b5decb 100644 --- a/eulerlauncher/backends/mac/qemu.py +++ b/eulerlauncher/backends/mac/qemu.py @@ -11,6 +11,7 @@ class QemuDriver(object): host_arch_raw = platform.uname().machine host_arch = constants.ARCH_MAP[host_arch_raw] self.qemu_bin = conf.conf.get('default', 'qemu_dir') + self.qemu_img_bin = conf.conf.get('default', 'qemu_img_dir') self.uefi_file = os.path.join('/Library/Application\ Support/org.openeuler.eulerlauncher/','edk2-' + host_arch + '-code.fd') self.uefi_params = ',if=pflash,format=raw,readonly=on' self.vm_cpu = conf.conf.get('vm', 'cpu_num') @@ -28,3 +29,16 @@ class QemuDriver(object): self.LOG.debug(' '.join(qemu_cmd)) instance_process = subprocess.Popen(' '.join(qemu_cmd), shell=True) return instance_process + + def take_and_export_snapshot(self, snapshot_name, vm_root_disk, export_image_name, dest_path): + take_snapshot_cmd = [ + 'sudo', self.qemu_img_bin, 'snapshot', '-c', f'{snapshot_name}', f'{vm_root_disk}' + ] + self.LOG.debug(' '.join(take_snapshot_cmd)) + subprocess.call(' '.join(take_snapshot_cmd), shell=True) + export_image_cmd = [ + 'sudo', self.qemu_img_bin, 'convert', '-l', f'snapshot.name={snapshot_name}', '-O', + 'qcow2', f'{vm_root_disk}', os.path.join(dest_path, f'{export_image_name}.qcow2') + ] + self.LOG.debug(' '.join(export_image_cmd)) + subprocess.call(' '.join(export_image_cmd), shell=True) \ No newline at end of file diff --git a/eulerlauncher/backends/win/image_handler.py b/eulerlauncher/backends/win/image_handler.py index 00530fdaf2d1eebb96d5984764c2dfd7bc79a9b4..f0398a57ec5187b96035c5ad7f4f6a04487f6e3c 100644 --- a/eulerlauncher/backends/win/image_handler.py +++ b/eulerlauncher/backends/win/image_handler.py @@ -29,6 +29,15 @@ class WinImageHandler(object): # Download the image img_name = wget.filename_from_url(images['remote'][img_to_download]['path']) img_dict = copy.deepcopy(images['remote'][img_to_download]) + downloaded_bytes = 0 + + def progress_bar(current, total, width): + nonlocal downloaded_bytes + if current == 0 or current - downloaded_bytes > 1024*1024 or current == total: + progress_percent = int((current / total) * 100) + downloaded_bytes = current + with open(os.path.join(self.image_dir, 'download_progress_bar_' + img_to_download), 'w') as progress_file: + progress_file.write(f"{current/1024/1024: .2f}/{total/1024/1024: .2f}MB ({progress_percent}%)") if not os.path.exists(os.path.join(self.image_dir, img_name)): self.LOG.debug(f'Downloading image: {img_to_download} from remote repo ...') @@ -36,7 +45,7 @@ class WinImageHandler(object): img_dict['status'] = constants.IMAGE_STATUS_DOWNLOADING images['local'][img_to_download] = img_dict omni_utils.save_json_data(self.image_record_file, images) - wget.download(url=images['remote'][img_to_download]['path'], out=os.path.join(self.image_dir, img_name), bar=None) + wget.download(url=images['remote'][img_to_download]['path'], out=os.path.join(self.image_dir, img_name), bar=progress_bar) self.LOG.debug(f'Image: {img_to_download} succesfully downloaded from remote repo ...') # Decompress the image @@ -49,8 +58,7 @@ class WinImageHandler(object): image_name = "" if self.pattern == "hyper-v": - # Convert the qcow2 img to vhdx - + # Convert the img to vhdx vhdx_name = img_to_download + '.vhdx' image_name = vhdx_name self.LOG.debug(f'Converting image file: {img_name} to {vhdx_name} ...') @@ -60,6 +68,8 @@ class WinImageHandler(object): self.LOG.debug(f'Cleanup temp files ...') os.remove(os.path.join(self.image_dir, qcow2_name)) + if os.path.exists(os.path.join(self.image_dir, "download_progress_bar_" + img_to_download)): + os.remove(os.path.join(self.image_dir, "download_progress_bar_" + img_to_download)) elif self.pattern == 'qemu': image_name = qcow2_name @@ -70,6 +80,17 @@ class WinImageHandler(object): omni_utils.save_json_data(self.image_record_file, images) self.LOG.debug(f'Image: {img_to_download} is ready ...') + def download_progress_bar(self, img_name): + progress_bar = "" + progress_bar_path = os.path.join(self.image_dir, 'download_progress_bar_' + img_name) + if os.path.exists(progress_bar_path): + with open(progress_bar_path, "r") as progress_bar_file: + progress_bar = progress_bar_file.read() + if progress_bar != "": + return constants.IMAGE_STATUS_DOWNLOADING + ": " + progress_bar + else: + return constants.IMAGE_STATUS_DOWNLOADING + def delete_image(self, images, img_to_delete): if img_to_delete not in images['local'].keys(): return 1 @@ -100,26 +121,33 @@ class WinImageHandler(object): image.status = constants.IMAGE_STATUS_LOADING images['local'][image.name] = image.to_dict() omni_utils.save_json_data(self.image_record_file, images) - qcow2_name = f'{img_to_load}.qcow2' - if fmt == 'qcow2': - shutil.copyfile(path, os.path.join(self.image_dir, qcow2_name)) + + if fmt not in constants.IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED: + img_name = f'{img_to_load}.{fmt}' + shutil.copyfile(path, os.path.join(self.image_dir, img_name)) else: # Decompress the image self.LOG.debug(f'Decompressing image file: {path} ...') - with open(path, 'rb') as pr, open(os.path.join(self.image_dir, qcow2_name), 'wb') as pw: + fmt = fmt.split('.')[0] + img_name = f'{img_to_load}.{fmt}' + with open(path, 'rb') as pr, open(os.path.join(self.image_dir, img_name), 'wb') as pw: data = pr.read() data_dec = lzma.decompress(data) pw.write(data_dec) + qcow2_name = img_name image_name = '' if self.pattern == 'hyper-v': # Convert the qcow2 img to vhdx vhdx_name = img_to_load + '.vhdx' - self.LOG.debug(f'Converting image file: {qcow2_name} to {vhdx_name} ...') - with powershell.PowerShell('GBK') as ps: - cmd = 'qemu-img convert -O vhdx {0} {1}' - outs, errs = ps.run(cmd.format(os.path.join(self.image_dir, qcow2_name), os.path.join(self.image_dir, vhdx_name))) - self.LOG.debug(f'Cleanup temp files ...') - os.remove(os.path.join(self.image_dir, qcow2_name)) + if fmt != 'vhdx': + self.LOG.debug(f'Converting image file: {qcow2_name} to {vhdx_name} ...') + load_progress_bar_path = os.path.join(self.image_dir, "load_progress_bar_" + img_to_load) + with powershell.PowerShell('GBK') as ps: + cmd = 'qemu-img convert -p -O vhdx {0} {1} | Out-File -FilePath {2}' + outs, errs = ps.run(cmd.format(os.path.join(self.image_dir, img_name), os.path.join(self.image_dir, vhdx_name), load_progress_bar_path)) + self.LOG.debug(f'Cleanup temp files ...') + os.remove(os.path.join(self.image_dir, qcow2_name)) + os.remove(load_progress_bar_path) image_name = vhdx_name elif self.pattern == 'qemu': image_name = qcow2_name @@ -130,3 +158,14 @@ class WinImageHandler(object): images['local'][image.name] = image.to_dict() omni_utils.save_json_data(self.image_record_file, images) self.LOG.debug(f'Image: {image_name} is ready ...') + + def load_progress_bar(self, img_name): + progress_bar_lines = [] + progress_bar_path = os.path.join(self.image_dir, "load_progress_bar_" + img_name) + if os.path.exists(progress_bar_path): + with open(progress_bar_path, 'r', encoding=omni_utils.detect_encoding(progress_bar_path), errors='ignore') as progress_bar_file: + progress_bar_lines = progress_bar_file.readlines() + if len(progress_bar_lines) > 1 and progress_bar_lines[-2].strip() != "": + return constants.IMAGE_STATUS_LOADING + ": " + progress_bar_lines[-2].strip() + else: + return constants.IMAGE_STATUS_LOADING \ No newline at end of file diff --git a/eulerlauncher/backends/win/instance_handler.py b/eulerlauncher/backends/win/instance_handler.py index 689139ea4794f3171aef61c08104d9b8741a39f2..2d07ec8b156d13f8da1afad7484ebfd05dc41ca4 100644 --- a/eulerlauncher/backends/win/instance_handler.py +++ b/eulerlauncher/backends/win/instance_handler.py @@ -1,6 +1,8 @@ import os import shutil import time +import paramiko +from glob import glob import psutil from oslo_utils import uuidutils @@ -66,7 +68,7 @@ class WinInstanceHandler(object): vm_list = _vmops.list_instances() return vm_list - def create_instance(self, name, image_id, instance_record, all_instances, all_images, arch='x86'): + def create_instance(self, name, image_id, instance_record, all_instances, all_images, is_same, mac_address, uuid, arch='x86'): # Create dir for the instance instance_path = os.path.join(self.instance_dir, name) os.makedirs(instance_path) @@ -257,3 +259,34 @@ class WinInstanceHandler(object): self.LOG.debug("Power on instance", instance=instance) self._set_vm_state(instance, os_win_const.HYPERV_VM_STATE_ENABLED) + def take_snapshot(self, name, snapshot_name, dest_path, all_instances, all_images, instance_record): + _vmops.take_snapshot(name, snapshot_name) + _vmops.export_vm(name, os.path.join(dest_path)) + os.rename(glob(os.path.join(dest_path, name, 'Virtual Hard Disks', '*.vhdx'))[0], os.path.join(dest_path, name, 'Virtual Hard Disks', f'{snapshot_name}.vhdx')) + shutil.move(os.path.join(dest_path, name, 'Virtual Hard Disks', f'{snapshot_name}.vhdx'), os.path.join(dest_path)) + # remove the exported file dir + shutil.rmtree(os.path.join(dest_path, name)) + return os.path.join(dest_path, f'{snapshot_name}.vhdx') + + def make_development_image(self, name, pwd): + ssh_client = paramiko.SSHClient() + try: + ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh_client.connect(_vmops.get_instance_ip_addr(name), 22, "root", pwd) + bash_command = """ + if which apt >/dev/null 2>&1; + then apt install python3-dev golang openjdk-11-jdk + elif which yum >/dev/null 2>&1; + then yum install -y python3-devel golang java-11-openjdk-devel + elif which dnf >/dev/null 2>&1; + then dnf install -y python3-devel golang java-11-openjdk-devel + fi + """ + stdin, stdout, stderr = ssh_client.exec_command(bash_command) + + self.LOG.debug(stdout.read().decode()) + ssh_client.close() + return 0 + except Exception as e: + self.LOG.debug(f"install development environment failed: {str(e)}") + return 1 \ No newline at end of file diff --git a/eulerlauncher/backends/win/vmops.py b/eulerlauncher/backends/win/vmops.py index 2f1340c6ade5b6ccb8e3b3930ec88f5b75363e3f..0ccf874116b30457536891859c290b3fc81f3565 100644 --- a/eulerlauncher/backends/win/vmops.py +++ b/eulerlauncher/backends/win/vmops.py @@ -37,6 +37,7 @@ class VMOps(object): def __init__(self, virtapi=None): self._virtapi = virtapi self._vmutils = VMUtils_omni() + self._migrationutils = utilsfactory.get_migrationutils() self._netutils = utilsfactory.get_networkutils() self._hostutils = utilsfactory.get_hostutils() @@ -186,6 +187,12 @@ class VMOps(object): def get_host_ips(self): return self._hostutils.get_local_ips() + def take_snapshot(self, vm_name, snapshot_name): + self._vmutils.take_vm_snapshot(vm_name, snapshot_name) + + def export_vm(self, vm_name, dest_path): + self._migrationutils.export_vm(vm_name, dest_path, copy_snapshots_config=os_win_const.EXPORT_CONFIG_NO_SNAPSHOTS, copy_vm_storage=True, create_export_subdir=True) + def parse_ip_addr(self, mac_addr): ip = '' macs = mac_addr.split(':') @@ -224,4 +231,4 @@ class VMOps(object): return constants.VM_STATE_MAP[2] return constants.VM_STATE_MAP[3] else: - return constants.VM_STATE_MAP[99] + return constants.VM_STATE_MAP[99] \ No newline at end of file diff --git a/eulerlauncher/cli.py b/eulerlauncher/cli.py index 0dcb03f05640f9967df2fbdee123928c742c1177..630eaa9a693ea12e587628fb3290602b280afe43 100644 --- a/eulerlauncher/cli.py +++ b/eulerlauncher/cli.py @@ -1,5 +1,6 @@ import click import prettytable as pt +import getpass from eulerlauncher.grpcs import client @@ -125,6 +126,32 @@ def launch(vm_name, image, arch): else: print(ret['msg']) +@click.command() +@click.argument('vm_name') +@click.option('--snapshot_name', help='name for the snapshot image') +@click.option('--export_path', help='path for the exported snapshot image') +def take_snapshot(vm_name, snapshot_name, export_path): + + try: + ret = launcher_client.take_snapshot(vm_name, snapshot_name, export_path) + except Exception: + print('Calling to EulerLauncherd daemon failed, please check EulerLauncherd daemon status ...') + else: + print(ret['msg']) + +@click.command() +@click.argument('vm_name') +@click.option('--image_name', help='name for the Python/Go/Java development image') +@click.option('--export_path', help='path for the exported Python/Go/Java development image') +def export_development_image(vm_name, image_name, export_path): + + try: + pwd = getpass.getpass(f"Password for vm[{vm_name}] as root: ") + ret = launcher_client.export_development_image(vm_name, image_name, export_path, pwd) + except Exception: + print('Calling to EulerLauncherd daemon failed, please check EulerLauncherd daemon status ...') + else: + print(ret['msg']) # @click.command() # @click.option('--count', default=1, help='Number of greetings.') @@ -148,5 +175,6 @@ if __name__ == '__main__': cli.add_command(launch) cli.add_command(delete_image) cli.add_command(delete_instance) - cli() - + cli.add_command(take_snapshot) + cli.add_command(export_development_image) + cli() \ No newline at end of file diff --git a/eulerlauncher/grpcs/client.py b/eulerlauncher/grpcs/client.py index dd21b2fbecb2cd3bf84717587902972308f0f54d..978f77e588760852764494c70fbcb8ab561d3e5e 100644 --- a/eulerlauncher/grpcs/client.py +++ b/eulerlauncher/grpcs/client.py @@ -50,7 +50,7 @@ class Client(object): return err_msg supported = False - for tp in constants.IMAGE_LOAD_SUPPORTED_TYPES: + for tp in constants.IMAGE_LOAD_SUPPORTED_TYPES + constants.IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED: if path.endswith(tp): supported = True break @@ -94,3 +94,17 @@ class Client(object): """ return self._instances.delete(name) + + @omnivirt_utils.response2dict + def take_snapshot(self, vm_name, snapshot_name, export_path): + """ Take snapshot + """ + + return self._instances.take_snapshot(vm_name, snapshot_name, export_path) + + @omnivirt_utils.response2dict + def export_development_image(self, vm_name, image_name, export_path, pwd): + """ Export Python/Go/Java development image + """ + + return self._instances.export_development_image(vm_name, image_name, export_path, pwd) \ No newline at end of file diff --git a/eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto b/eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto index 7d160f65444547c2b18e38196601e5086f4f7c3f..6446ac319b725b1c73905048c2843a8f00ae346b 100644 --- a/eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto +++ b/eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto @@ -8,6 +8,8 @@ service InstanceGrpcService { rpc list_instances (ListInstancesRequest) returns (ListInstancesResponse) {} rpc create_instance (CreateInstanceRequest) returns (CreateInstanceResponse) {} rpc delete_instance (DeleteInstanceRequest) returns (DeleteInstanceResponse) {} + rpc take_snapshot (TakeSnapshotRequest) returns (TakeSnapshotResponse) {} + rpc export_development_image (ExportDevelopmentImageRequest) returns (ExportDevelopmentImageResponse) {} } @@ -47,3 +49,24 @@ message DeleteInstanceResponse { uint32 ret = 1; string msg = 2; } + +message TakeSnapshotRequest { + string name = 1; + string snapshot = 2; + string dest_path = 3; +} + +message TakeSnapshotResponse { + string msg = 1; +} + +message ExportDevelopmentImageRequest { + string name = 1; + string image = 2; + string dest_path = 3; + string pwd = 4; +} + +message ExportDevelopmentImageResponse { + string msg = 1; +} \ No newline at end of file diff --git a/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2.py b/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2.py index 49a9e3350b252c998067ce6db4b749e9f0efcb6b..257a0c665189cfdc610fd746ff7577a48ea85305 100644 --- a/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2.py +++ b/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2.py @@ -13,7 +13,7 @@ _sym_db = _symbol_database.Default() -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n6eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto\x12\x08omnivirt\"M\n\x08Instance\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05image\x18\x02 \x01(\t\x12\x10\n\x08vm_state\x18\x03 \x01(\t\x12\x12\n\nip_address\x18\x04 \x01(\t\"\x16\n\x14ListInstancesRequest\">\n\x15ListInstancesResponse\x12%\n\tinstances\x18\x01 \x03(\x0b\x32\x12.omnivirt.Instance\"B\n\x15\x43reateInstanceRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05image\x18\x02 \x01(\t\x12\x0c\n\x04\x61rch\x18\x03 \x01(\t\"j\n\x16\x43reateInstanceResponse\x12\x0b\n\x03ret\x18\x01 \x01(\r\x12\x0b\n\x03msg\x18\x02 \x01(\t\x12)\n\x08instance\x18\x03 \x01(\x0b\x32\x12.omnivirt.InstanceH\x00\x88\x01\x01\x42\x0b\n\t_instance\"%\n\x15\x44\x65leteInstanceRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"2\n\x16\x44\x65leteInstanceResponse\x12\x0b\n\x03ret\x18\x01 \x01(\r\x12\x0b\n\x03msg\x18\x02 \x01(\t2\x9a\x02\n\x13InstanceGrpcService\x12S\n\x0elist_instances\x12\x1e.omnivirt.ListInstancesRequest\x1a\x1f.omnivirt.ListInstancesResponse\"\x00\x12V\n\x0f\x63reate_instance\x12\x1f.omnivirt.CreateInstanceRequest\x1a .omnivirt.CreateInstanceResponse\"\x00\x12V\n\x0f\x64\x65lete_instance\x12\x1f.omnivirt.DeleteInstanceRequest\x1a .omnivirt.DeleteInstanceResponse\"\x00\x42\x03\x80\x01\x01\x62\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n6eulerlauncher/grpcs/eulerlauncher_grpc/instances.proto\x12\x08omnivirt\"M\n\x08Instance\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05image\x18\x02 \x01(\t\x12\x10\n\x08vm_state\x18\x03 \x01(\t\x12\x12\n\nip_address\x18\x04 \x01(\t\"\x16\n\x14ListInstancesRequest\">\n\x15ListInstancesResponse\x12%\n\tinstances\x18\x01 \x03(\x0b\x32\x12.omnivirt.Instance\"B\n\x15\x43reateInstanceRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05image\x18\x02 \x01(\t\x12\x0c\n\x04\x61rch\x18\x03 \x01(\t\"j\n\x16\x43reateInstanceResponse\x12\x0b\n\x03ret\x18\x01 \x01(\r\x12\x0b\n\x03msg\x18\x02 \x01(\t\x12)\n\x08instance\x18\x03 \x01(\x0b\x32\x12.omnivirt.InstanceH\x00\x88\x01\x01\x42\x0b\n\t_instance\"%\n\x15\x44\x65leteInstanceRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"2\n\x16\x44\x65leteInstanceResponse\x12\x0b\n\x03ret\x18\x01 \x01(\r\x12\x0b\n\x03msg\x18\x02 \x01(\t\"H\n\x13TakeSnapshotRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x10\n\x08snapshot\x18\x02 \x01(\t\x12\x11\n\tdest_path\x18\x03 \x01(\t\"#\n\x14TakeSnapshotResponse\x12\x0b\n\x03msg\x18\x01 \x01(\t\"\\\n\x1d\x45xportDevelopmentImageRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05image\x18\x02 \x01(\t\x12\x11\n\tdest_path\x18\x03 \x01(\t\x12\x0b\n\x03pwd\x18\x04 \x01(\t\"-\n\x1e\x45xportDevelopmentImageResponse\x12\x0b\n\x03msg\x18\x01 \x01(\t2\xdd\x03\n\x13InstanceGrpcService\x12S\n\x0elist_instances\x12\x1e.omnivirt.ListInstancesRequest\x1a\x1f.omnivirt.ListInstancesResponse\"\x00\x12V\n\x0f\x63reate_instance\x12\x1f.omnivirt.CreateInstanceRequest\x1a .omnivirt.CreateInstanceResponse\"\x00\x12V\n\x0f\x64\x65lete_instance\x12\x1f.omnivirt.DeleteInstanceRequest\x1a .omnivirt.DeleteInstanceResponse\"\x00\x12P\n\rtake_snapshot\x12\x1d.omnivirt.TakeSnapshotRequest\x1a\x1e.omnivirt.TakeSnapshotResponse\"\x00\x12o\n\x18\x65xport_development_image\x12\'.omnivirt.ExportDevelopmentImageRequest\x1a(.omnivirt.ExportDevelopmentImageResponse\"\x00\x42\x03\x80\x01\x01\x62\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -36,6 +36,14 @@ if _descriptor._USE_C_DESCRIPTORS == False: _globals['_DELETEINSTANCEREQUEST']._serialized_end=448 _globals['_DELETEINSTANCERESPONSE']._serialized_start=450 _globals['_DELETEINSTANCERESPONSE']._serialized_end=500 - _globals['_INSTANCEGRPCSERVICE']._serialized_start=503 - _globals['_INSTANCEGRPCSERVICE']._serialized_end=785 + _globals['_TAKESNAPSHOTREQUEST']._serialized_start=502 + _globals['_TAKESNAPSHOTREQUEST']._serialized_end=574 + _globals['_TAKESNAPSHOTRESPONSE']._serialized_start=576 + _globals['_TAKESNAPSHOTRESPONSE']._serialized_end=611 + _globals['_EXPORTDEVELOPMENTIMAGEREQUEST']._serialized_start=613 + _globals['_EXPORTDEVELOPMENTIMAGEREQUEST']._serialized_end=705 + _globals['_EXPORTDEVELOPMENTIMAGERESPONSE']._serialized_start=707 + _globals['_EXPORTDEVELOPMENTIMAGERESPONSE']._serialized_end=752 + _globals['_INSTANCEGRPCSERVICE']._serialized_start=755 + _globals['_INSTANCEGRPCSERVICE']._serialized_end=1232 # @@protoc_insertion_point(module_scope) diff --git a/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2_grpc.py b/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2_grpc.py index f2cbb38fa634b797392c6bb303036580c9e9be30..7e796789ded0b15b33b043c081925a4d4a43f730 100644 --- a/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2_grpc.py +++ b/eulerlauncher/grpcs/eulerlauncher_grpc/instances_pb2_grpc.py @@ -29,6 +29,16 @@ class InstanceGrpcServiceStub(object): request_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.DeleteInstanceRequest.SerializeToString, response_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.DeleteInstanceResponse.FromString, ) + self.take_snapshot = channel.unary_unary( + '/omnivirt.InstanceGrpcService/take_snapshot', + request_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotRequest.SerializeToString, + response_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotResponse.FromString, + ) + self.export_development_image = channel.unary_unary( + '/omnivirt.InstanceGrpcService/export_development_image', + request_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageRequest.SerializeToString, + response_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageResponse.FromString, + ) class InstanceGrpcServiceServicer(object): @@ -52,6 +62,18 @@ class InstanceGrpcServiceServicer(object): context.set_details('Method not implemented!') raise NotImplementedError('Method not implemented!') + def take_snapshot(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def export_development_image(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + def add_InstanceGrpcServiceServicer_to_server(servicer, server): rpc_method_handlers = { @@ -70,6 +92,16 @@ def add_InstanceGrpcServiceServicer_to_server(servicer, server): request_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.DeleteInstanceRequest.FromString, response_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.DeleteInstanceResponse.SerializeToString, ), + 'take_snapshot': grpc.unary_unary_rpc_method_handler( + servicer.take_snapshot, + request_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotRequest.FromString, + response_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotResponse.SerializeToString, + ), + 'export_development_image': grpc.unary_unary_rpc_method_handler( + servicer.export_development_image, + request_deserializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageRequest.FromString, + response_serializer=eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageResponse.SerializeToString, + ), } generic_handler = grpc.method_handlers_generic_handler( 'omnivirt.InstanceGrpcService', rpc_method_handlers) @@ -130,3 +162,37 @@ class InstanceGrpcService(object): eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.DeleteInstanceResponse.FromString, options, channel_credentials, insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + @staticmethod + def take_snapshot(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/omnivirt.InstanceGrpcService/take_snapshot', + eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotRequest.SerializeToString, + eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.TakeSnapshotResponse.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + @staticmethod + def export_development_image(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/omnivirt.InstanceGrpcService/export_development_image', + eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageRequest.SerializeToString, + eulerlauncher_dot_grpcs_dot_eulerlauncher__grpc_dot_instances__pb2.ExportDevelopmentImageResponse.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) diff --git a/eulerlauncher/grpcs/instances.py b/eulerlauncher/grpcs/instances.py index 0e4c7f072f076babf51138c7e836eccf573ef04e..5d4bf5e941917afde590b27b9af2f76a239bc7c7 100644 --- a/eulerlauncher/grpcs/instances.py +++ b/eulerlauncher/grpcs/instances.py @@ -21,3 +21,15 @@ class Instance(object): request = instances_pb2.DeleteInstanceRequest(name=name) response = self.client.delete_instance(request) return response + + def take_snapshot(self, vm_name, snapshot_name, export_path): + """Take snapshot""" + request = instances_pb2.TakeSnapshotRequest(name=vm_name, snapshot=snapshot_name, dest_path=export_path) + response = self.client.take_snapshot(request) + return response + + def export_development_image(self, vm_name, image_name, export_path, pwd): + """Export Python/Go/Java development image""" + request = instances_pb2.ExportDevelopmentImageRequest(name=vm_name, image=image_name, dest_path=export_path, pwd=pwd) + response = self.client.export_development_image(request) + return response \ No newline at end of file diff --git a/eulerlauncher/services/imager_service.py b/eulerlauncher/services/imager_service.py index 331e7d89e7e61a635d686b4b681927fe11cf8ec2..d38d65b5db9ae4aa0e0aab96188996fd56fa0ffd 100644 --- a/eulerlauncher/services/imager_service.py +++ b/eulerlauncher/services/imager_service.py @@ -41,6 +41,10 @@ class ImagerService(images_pb2_grpc.ImageGrpcServiceServicer): image.name = img['name'] image.location = img['location'] image.status = img['status'] + if image.status == omni_constants.IMAGE_STATUS_DOWNLOADING: + image.status = self.backend.download_progress_bar(image.name) + elif image.status == omni_constants.IMAGE_STATUS_LOADING: + image.status = self.backend.load_progress_bar(image.name) ret.append(image) LOG.debug(f"Responded: {ret}") return images_pb2.ListImageResponse(images=ret) @@ -51,7 +55,7 @@ class ImagerService(images_pb2_grpc.ImageGrpcServiceServicer): if request.name not in all_images['remote'].keys(): LOG.debug(f'Image: {request.name} not valid for download') - msg = f'Error: Image {request.name} is valid for download, please check image name from REMOTE IMAGE LIST using "images" command ...' + msg = f'Error: Image {request.name} is not valid for download, please check image name from REMOTE IMAGE LIST using "images" command ...' return images_pb2.GeneralImageResponse(ret=1, msg=msg) @omni_utils.asyncwrapper @@ -68,11 +72,11 @@ class ImagerService(images_pb2_grpc.ImageGrpcServiceServicer): LOG.debug(f"Get request to load image: {request.name} from path: {request.path} ...") supported, fmt = omni_utils.check_file_tail( - request.path, omni_constants.IMAGE_LOAD_SUPPORTED_TYPES) + request.path, omni_constants.IMAGE_LOAD_SUPPORTED_TYPES + omni_constants.IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED) if not supported: - supported_fmt = ', '.join(omni_constants.IMAGE_LOAD_SUPPORTED_TYPES) - msg = f'Unsupported image format, the current supported format are: {supported_fmt}.' + supported_fmt = ', '.join(omni_constants.IMAGE_LOAD_SUPPORTED_TYPES + omni_constants.IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED) + msg = f'Unsupported image format, the current supported formats are: {supported_fmt}.' return images_pb2.GeneralImageResponse(ret=1, msg=msg) diff --git a/eulerlauncher/services/instance_service.py b/eulerlauncher/services/instance_service.py index 7d6c85bad7bddfb2d3fd06e514532068eae65b3e..448949130c356dd194b459c3cbcb539fa14ff788 100644 --- a/eulerlauncher/services/instance_service.py +++ b/eulerlauncher/services/instance_service.py @@ -77,7 +77,7 @@ class InstanceService(instances_pb2_grpc.InstanceGrpcServiceServicer): return instances_pb2.CreateInstanceResponse(ret=2, msg=msg) vm = self.backend.create_instance( - request.name, request.image, self.instance_record_file, all_instances, all_img, request.arch) + request.name, request.image, self.instance_record_file, all_instances, all_img, False, 0, 0, request.arch) msg = f'Successfully created {request.name} with image {request.image}' return instances_pb2.CreateInstanceResponse(ret=1, msg=msg, instance=vm) @@ -91,3 +91,31 @@ class InstanceService(instances_pb2_grpc.InstanceGrpcServiceServicer): self.backend.delete_instance(request.name, self.instance_record_file, all_instances) msg = f'Successfully deleted instance: {request.name}.' return instances_pb2.DeleteInstanceResponse(ret=1, msg=msg) + + def take_snapshot(self, request, context): + LOG.debug(f"Get request to take snapshot: {request.name} with snapshot name {request.snapshot}, export to path: {request.dest_path}...") + all_img = utils.load_json_data(self.img_record_file) + all_instances = utils.load_json_data(self.instance_record_file) + if request.name not in all_instances['instances'].keys(): + msg = f'Error: Instance with name {request.name} does not exist.' + return instances_pb2.TakeSnapshotResponse(msg=msg) + export_path = self.backend.take_snapshot(request.name, request.snapshot, request.dest_path, all_instances, all_img, self.instance_record_file) + msg = f'Successfully take snapshot: {request.snapshot} of vm: {request.name} and export snapshot image to {export_path}.' + return instances_pb2.TakeSnapshotResponse(msg=msg) + + def export_development_image(self, request, context): + LOG.debug(f"Get request to export Python/Go/Java development image for: {request.name} with image name {request.image}, export to path: {request.dest_path}...") + all_img = utils.load_json_data(self.img_record_file) + all_instances = utils.load_json_data(self.instance_record_file) + if request.name not in all_instances['instances'].keys(): + msg = f'Error: Instance with name {request.name} does not exist.' + return instances_pb2.ExportDevelopmentImageResponse(msg=msg) + result = self.backend.make_development_image(request.name, request.pwd) + if result != 0: + LOG.debug(f"install Python/Go/Java environment for vm {request.name} failed!") + msg = f"install Python/Go/Java environment for vm {request.name} failed!" + return instances_pb2.ExportDevelopmentImageResponse(msg=msg) + LOG.debug(f"vm: {request.name} install Python/Go/Java development environment finished") + export_path = self.backend.take_snapshot(request.name, request.image, request.dest_path, all_instances, all_img, self.instance_record_file) + msg = f'Successfully export Python/Go/Java development image: {request.image} of vm: {request.name} and export development image to {export_path}.' + return instances_pb2.ExportDevelopmentImageResponse(msg=msg) \ No newline at end of file diff --git a/eulerlauncher/utils/constants.py b/eulerlauncher/utils/constants.py index 9383d2d5f2d0edf75c689631eb017ec0d846b989..3654b79173663b8add0cca5d9c51ac3d3c3179cb 100644 --- a/eulerlauncher/utils/constants.py +++ b/eulerlauncher/utils/constants.py @@ -14,7 +14,8 @@ IMAGE_STATUS_DOWNLOADING = 'Downloading' IMAGE_STATUS_LOADING = 'Loading' IMAGE_STATUS_READY = 'Ready' -IMAGE_LOAD_SUPPORTED_TYPES = ['qcow2.xz', 'qcow2'] +IMAGE_LOAD_SUPPORTED_TYPES = ['qcow2', 'raw', 'vmdk', 'vhd', 'vhdx', 'qcow', 'vdi'] +IMAGE_LOAD_SUPPORTED_TYPES_COMPRESSED = ['qcow2.xz', 'raw.xz', 'vmdk.xz', 'vhd.xz', 'vhdx.xz', 'qcow.xz', 'vdi.xz'] ARCH_MAP = { 'AMD64': 'x86_64', diff --git a/eulerlauncher/utils/utils.py b/eulerlauncher/utils/utils.py index 8381c74aaddaa9d9418fdc9e08dfa1c014ab1e7b..3503fd85bf881b08001c9053a30d40617040bd05 100644 --- a/eulerlauncher/utils/utils.py +++ b/eulerlauncher/utils/utils.py @@ -4,6 +4,7 @@ import os import random from threading import Thread import uuid +import chardet from google.protobuf.json_format import MessageToDict @@ -92,3 +93,9 @@ def check_file_tail(file_name, to_check): break return ret, ret_fmt + +def detect_encoding(file_path): + with open(file_path, 'rb') as file: + raw_data = file.read() + result = chardet.detect(raw_data) + return result['encoding'] \ No newline at end of file diff --git a/requirements-win.txt b/requirements-win.txt index b67b36dc4415cf837515d153a8423b9e6e45fb9a..7ed4143d721f264d574644cdc0c824a682f9cbfa 100644 --- a/requirements-win.txt +++ b/requirements-win.txt @@ -1,6 +1,7 @@ argparse click configparser +chardet dnspython grpcio grpcio-tools @@ -13,6 +14,7 @@ oslo.i18n oslo.log oslo.serialization oslo.utils +paramiko Pillow prettytable protobuf @@ -21,4 +23,4 @@ pystray PyYAML six urllib3 -wget +wget \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index db5e0f68c92003b71fc38050787376e243cb4a4a..e29252cc63e654a0c34ccc63a01cd3c132ec326e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,7 @@ argparse click configparser +chardet dnspython grpcio grpcio-tools @@ -14,6 +15,7 @@ oslo.i18n oslo.log oslo.serialization oslo.utils +paramiko Pillow prettytable protobuf @@ -22,4 +24,4 @@ pystray PyYAML six urllib3 -wget +wget \ No newline at end of file