| deeppavlov.core.commands |
basic training and inferring functions |
| deeppavlov.core.common |
registration and classes initialization functionality, class method decorators |
| deeppavlov.core.data |
basic Dataset, DatasetReader and Vocab classes |
| deeppavlov.core.models |
abstract model classes and interfaces |
| deeppavlov.dataset_readers |
concrete DatasetReader classes |
| deeppavlov.datasets |
concrete Dataset classes |
| deeppavlov.models |
concrete Model classes |
| deeppavlov.skills |
Skill classes. Skills are dialog models. |
| deeppavlov.vocabs |
concrete Vocab classes |
### Config
An NLP pipeline config is a JSON file that contains one required element `chainer`:
```json
{
"chainer": {
"in": ["x"],
"in_y": ["y"],
"pipe": [
...
],
"out": ["y_predicted"]
}
}
```
Chainer is a core concept of DeepPavlov library: chainer builds a pipeline from heterogeneous components
(rule-based/ml/dl) and allows to train and infer pipeline as a whole. Each component in the pipeline specifies
its inputs and outputs as array of names, for example: `"in": ["tokens", "features"]` and `"out": ["token_embeddings", "features_embeddings"]` and you can chain outputs of one components with inputs of other components:
```json
{
"name": "str_lower",
"in": ["x"],
"out": ["x_lower"]
},
{
"name": "nltk_tokenizer",
"in": ["x_lower"],
"out": ["x_tokens"]
},
```
Each [Component](deeppavlov/core/models/component.py) in the pipeline must implement method `__call__` and has `name` parameter, which is its registered codename and can have any other parameters, repeating its `__init__()` method arguments.
Default values of `__init__()` arguments will be overridden with the config values during class instance initialization.
You can reuse components in the pipeline to process different parts of data with help of `id` and `ref` parameters:
```json
{
"name": "nltk_tokenizer",
"id": "tokenizer",
"in": ["x_lower"],
"out": ["x_tokens"]
},
{
"ref": "tokenizer",
"in": ["y"],
"out": ["y_tokens"]
},
```
### Training
There are two abstract classes for trainable components: **Estimator** and **NNModel**.
[**Estimators**](deeppavlov/core/models/estimator.py) are fit once on any data with no batching or validation patience,
so it can be painlessly done at the time of pipeline initialization. [Vocab](deeppavlov/core/data/vocab.py) is a good example of Estimator. `fit` method has to be implemented for each Estimator.
[**NNModel**](deeppavlov/core/models/nn_model.py) requires a more complex training. It trains on the same data
it predicts on and ground truth answers. The process takes multiple epochs with periodic validation and logging.
`train_on_batch` method has to be implemented for each NNModel.
Training is triggered by `deeppavlov.core.commands.train.train_model_from_config()` function.
### Train config
Estimators that are trained should also have `fit_on` parameter with a list of input parameters' names.
A NNModel should have `in_y` parameter with a list of ground truth answers' names. For example:
```json
[
{
"id": "classes_vocab",
"name": "default_vocab",
"fit_on": ["y"],
"level": "token",
"save_path": "vocabs/classes.dict",
"load_path": "vocabs/classes.dict"
},
{
"in": ["x"],
"in_y": ["y"],
"out": ["y_predicted"],
"name": "intent_model",
"save_path": "intents/intent_cnn",
"load_path": "intents/intent_cnn",
"classes_vocab": {
"ref": "classes_vocab"
}
}
]
```
Config for training the pipeline has to have three additional elements: `dataset_reader`, `dataset` and `train`:
```json
{
"dataset_reader": {
"name": ...,
...
}
"dataset": {
"name": ...,
...
},
"chainer": {
...
}
"train": {
...
}
}
```
### Train Parameters
* `epochs` — maximum number of epochs to train NNModel, defaults to `-1`, infinite
* `batch_size`,
* `metrics` — list of names of [registered metrics](deeppavlov/metrics) to evaluate the model on. First one in the list
is used for validation patience
* `metric_optimization` — one of `maximize` or `minimize`, defaults to `maximize`
* `validation_patience` — how many times in a row validation metric has to not improve to stop training, defaults to `5`
* `val_every_n_epochs` — how often to validate the pipe, defaults to `-1`, never
* `log_every_n_batches`, `log_every_n_epochs` — how often to calculate metrics for train data, defaults to `-1`, never
* `validate_best`, `test_best` flags to infer the best saved model on valid and test data, defaults to `true`
### DatasetReader
`DatasetReader` class reads data and returns it in a specified format.
A concrete `DatasetReader` class should be inherited from base
`deeppavlov.data.dataset_reader.DatasetReader` class and registered with a codename:
```python
from deeppavlov.core.common.registry import register
from deeppavlov.core.data.dataset_reader import DatasetReader
@register('dstc2_datasetreader')
class DSTC2DatasetReader(DatasetReader):
```
### Dataset
`Dataset` forms needed sets of data ('train', 'valid', 'test') and forms data batches.
A concrete `Dataset` class should be registered and can be inherited from
`deeppavlov.data.dataset_reader.Dataset` class. `deeppavlov.data.dataset_reader.Dataset`
is not an abstract class and can be used as `Dataset` as well.
### Inferring
All components inherited from `deeppavlov.core.models.component.Componet` abstract class can be inferred. The `__call__()` method should return what a compoent can do. For example, a *tokenizer* should return
*tokens*, a *NER recognizer* should return *recognized entities*, a *bot* should return a *replica*.
A particular format of returned data should be defined in `__call__()`.
Inferring is triggered by `deeppavlov.core.commands.infer.interact_model()` function. There is no need in a separate JSON for inferring.
## License
DeepPavlov is Apache 2.0 - licensed.
## Support and collaboration
If you have any questions, bug reports or feature requests, please feel free to post on our [Github Issues](https://github.com/deepmipt/DeepPavlov/issues) page. Please tag your issue with 'bug', 'feature request', or 'question'. Also we’ll be glad to see your pull-requests to add new datasets, models, embeddings and etc.
## The Team
DeepPavlov is built and maintained by [Neural Networks and Deep Learning Lab](https://mipt.ru/english/research/labs/neural-networks-and-deep-learning-lab) at [MIPT](https://mipt.ru/english/) within [iPavlov](http://ipavlov.ai/) project (part of [National Technology Initiative](https://asi.ru/eng/nti/)) and in partnership with [Sberbank](http://www.sberbank.com/).
