代码拉取完成，页面将自动刷新
Updated for CUDA 11 and Tensorflow 2!!!
This repository allows you to get started with training a State-of-the-art Deep Learning model with little to no configuration needed! You provide your labeled dataset and you can start the training right away and monitor it with TensorBoard. You can even test your model with our built-in Inference REST API. Training with TensorFlow has never been so easy.
You can also use our BMW-Labeltool-lite to label your dataset. The images and labels can be used directly for training.
This step is recommended to be able to run the solution correctly.
The setup script will check and adjust all the requirements needed based on the user input.
Run the following command
chmod +x setup_solution_parameters.sh && source setup_solution_parameters.sh
The script will check if docker and docker-compose are installed and if not, it will install them.
You will be prompted to choose build architecture GPU/CPU for the training solution.
The script will prompt all the network interfaces that are available so you can select which interface you want to extract the ip address from it
You will be prompted to choose if you want to set up a proxy
You will be prompted to select all the pre-trained weights network that you want to be downloaded during the docker image build (use the up/down arrow to go up and down, space to select/unselect, enter to submit your selection and esc to quit)
To check if you have docker-ce installed:
docker --version
To check if you have docker-compose installed:
docker-compose --version
To check if you have nvidia-docker installed:
dpkg -l | grep nvidia-docker
To check your nvidia drivers version, open your terminal and type the command nvidia-smi
If you don't have neither docker nor docker-compose use the following command
chmod +x install_full.sh && source install_full.sh
If you have docker ce installed and wish only to install docker-compose and perform necessary operations, use the following command
chmod +x install_compose.sh && source install_compose.sh
Install NVIDIA Drivers (418.x or higher) and NVIDIA Docker for GPU training by following the official docs
Make sure that the .gitkeep files in datasets, checkpoints, tensorboards and inference_api/models folder are deleted. (.gitkeep files are placeholder
files used for git)
Make sure that the base_dir field in docker_sdk_api/assets/paths.json is correct (it must match the path of the root of the repo on your machine).
Make sure that the image_name field in docker_sdk_api/assets/paths.json is correct (it must match your chosen architecture for the training
tf2_training_api_cpu or tf2_training_api_gpu).
Go to gui/src/environments/environment.ts and gui/src/environments/environment.prod.ts and change the following:
field url: must match the IP address of your machine
the IP field of the inferenceAPIUrl : must match the IP address of your machine (Use the ifconfig command to check your IP address . Please use your private IP which starts by either 10. or 172.16. or 192.168.)
environment.ts
environment.prod.ts
If you are behind a proxy:
Enter you proxy settings in the <base-dir>/proxy.json file
Enter the following command:
python3 set_proxy_args.py
The following is an example of how a dataset should be structured. Please put all your datasets in the datasets folder.
├──datasets/
├──sample_dataset/
├── images
│ ├── img_1.jpg
│ └── img_2.jpg
├── labels
│ ├── json
│ │ ├── img_1.json
│ │ └── img_2.json
│ └── pascal
│ ├── img_1.xml
│ └── img_2.xml
└── objectclasses.json
PS: you don't need to have both json and pascal folders. Either one is enough
You must include in your dataset an objectclasses.json file with a similar structure to the example below:
Midweight: Downloading specific supported online pre-trained weights during the docker image build.
To do that, open the json file training_api/assets/networks.json and change the values of the networks you wish to download to true.
Heavyweight (default): Downloading all the supported online pre-trained weights during the docker image build.
To do that, open the json file training_api/assets/networks.json and change the value of "select_all" to true.
PS: if you don’t download the weights during the build, you won’t be able to use the network during the training unless you rebuild the solution with the proper network chosen.
All the training are made using the pre-trained network weight based on coco dataset.
If you wish want to deploy the training workflow in GPU mode, please write the following command from the repository's root directory
docker-compose -f build_gpu.yml build
If you wish want to deploy the training workflow in CPU mode, please write the following command from the repository's root directory
docker-compose -f build_cpu.yml build
If you wish want to deploy the training workflow in GPU mode, please write the following command
docker-compose -f run_gpu.yml up
If you wish to deploy the training workflow in CPU mode, please write the following command
docker-compose -f run_cpu.yml up
After a successful run you should see something like the following:
If the app is deployed on your machine: open your web browser and type the following: localhost:4200 or 127.0.0.1:4200
If the app is deployed on a different machine: open your web browser and type the following: <machine_ip>:4200
Prepare your dataset for training
Specify the general parameters for you docker container
Specify the hyperparameters for the training job
Specify the advanced hyperparameters for the training job
Check your training logs to get better insights on the progress of the training
Monitor the training using Tensorboard
Check the status to know when the job is completed successfully
Download your mode and easily test it with the built-in inference API using Swagger
Delete the container's job to stop an ongoing job or to remove the container of a finished job. (Finished jobs are always available to download)
Visualize graphs and metrics of Deleted Jobs with Tensorboard
Check our tips document to have (1) (a better insight on training models based on our expertise) and (2) (a benchmark of the inference speed).
Our tensorboard document helps you find your way more easily while navigating tensorboard
In advanced configuration mode, be careful while making the changes because it can cause errors while training. If that happens, stop the job and try again.
the paths in fine_tune_checkpoint should be entered manually when you choose train from checkpoint
Scroll down to find fine_tune_checkpoint and replace with your network-name and checkpoint name:
fine_tune_checkpoint :
fine_tune_checkpoint: "/checkpoints/<network-name>/<name-of-the-checkpoint>/ckpt-0"
In general settings, choose carefully the container name because choosing a name used by another container will cause errors.
When you try to monitor the job using tensorboard the page may not open, wait for some seconds and refresh the page.
When you leave tensorboard open for a long time, it might freeze. When encountered with such issue simply closing tensorboard tab in the browser and reopening it will solve the problem.
To change the docker-sdk default port 2222 to any other port of your choice:
change the uvicorn port inside docker_sdk_api/docker/Dockerfile to the port of your choice
rebuild the docker-sdk image using the following commend in the root of the repo:
docker-compose -f build.yml build docker_sdkchange the baseEndPoint : <port-of-your-choice inside gui/src/environments/environment.ts and gui/src/environments/environment.prod.ts
rebuild the GUI image using the following comment in the root of the repo:
docker-compose -f build.yml build user_interfaceafter this you can run the solution as following: docker-compose -f run.yml up
You might face some errors in some cases during the training. Most common ones are:
The running container has no RepoTag please kill to proceed: Container ID: <id-container> This issue is caused by some container not having a name, in that case you should rename that container or kill (make sure it is safe to remove this container) it via docker kill <id-container>.Job Not Started: 404 Client Error Not Found("pull access denied for <image-name>, repository does not exists or may require 'docker login' ...) this issue is cause when you are trying to run a training docker image that you don't have. The main reason of this is not properly building the training_api or not setting up project requirements please refer to Setting Up Project Requirements section in the documentation.Dataset Not Valid this error means that you dataset structure is not valid or the images/labels formate are not supported.docker_sdk_api/assets/paths.json field base_dir. You can solve this issue by running ./setup_solution_parameters.sh and choosing the training version you want to use GPU/CPU.If you use this repository in your research, consider citing it using the following Bibtex entry:
@misc{bmwtrainingtool,
author = {BMW TechOffice MUNICH},
title = {TensorFlow Training GUI},
howpublished = {\url{https://github.com/BMW-InnovationLab/BMW-TensorFlow-Training-GUI}},
year = {2022},
}
Hadi Koubeissy, inmind.ai, Beirut, Lebanon
Ismail Shehab, inmind.ai, Beirut, Lebanon
Joe Sleiman, inmind.ai, Beirut, Lebanon
Jimmy Tekli, BMW Innovation Lab, Munich, Germany
Chafic Abou Akar, BMW TechOffice, Munich, Germany
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手
