Python pipelinesΒΆ

Open In Colab

Python models could be used without .json configuration files.

The code below is an alternative to building insults_kaggle_bert model and using it with

from deeppavlov import build_model

model = build_model('insults_kaggle_bert', download=True)

At first, define variables for model components and download model data.

[ ]:
from deeppavlov.core.commands.utils import expand_path
from deeppavlov.download import download_resource


classifiers_path = expand_path('~/.deeppavlov/models/classifiers')
model_path = classifiers_path / 'insults_kaggle_torch_bert'
transformer_name = 'bert-base-uncased'

download_resource(
    'http://files.deeppavlov.ai/deeppavlov_data/classifiers/insults_kaggle_torch_bert_v5.tar.gz',
    {classifiers_path}
)

Then, initialize model components.

[ ]:
from deeppavlov.core.data.simple_vocab import SimpleVocabulary
from deeppavlov.models.classifiers.proba2labels import Proba2Labels
from deeppavlov.models.preprocessors.torch_transformers_preprocessor import TorchTransformersPreprocessor
from deeppavlov.models.torch_bert.torch_transformers_classifier import TorchTransformersClassifierModel


preprocessor = TorchTransformersPreprocessor(
    vocab_file=transformer_name,
    max_seq_length=64
)

classes_vocab = SimpleVocabulary(
    load_path=model_path/'classes.dict',
    save_path=model_path/'classes.dict'
)

classifier =  TorchTransformersClassifierModel(
    n_classes=classes_vocab.len,
    return_probas=True,
    pretrained_bert=transformer_name,
    save_path=model_path/'model',
    optimizer_parameters={'lr': 1e-05}
)

proba2labels = Proba2Labels(max_proba=True)

Finally, create model from components. Element is a wrapper for a component. Element receives the component and the names of the incoming and outgoing arguments. Model combines Elements into pipeline.

[ ]:
from deeppavlov import Element, Model

model = Model(
    x=['x'],
    out=['y_pred_labels'],
    pipe=[
        Element(component=preprocessor, x=['x'], out=['bert_features']),
        Element(component=classifier, x=['bert_features'], out=['y_pred_probas']),
        Element(component=proba2labels, x=['y_pred_probas'], out=['y_pred_ids']),
        Element(component=classes_vocab, x=['y_pred_ids'], out=['y_pred_labels'])
    ]
)

model(['you are stupid', 'you are smart'])