REST API¶
Each DeepPavlov model can be easily made available for inference as a REST web service. The general method is:
python -m deeppavlov riseapi <config_path> [-d] [-p <port>] [--https] [--key <SSL key file path>] \
[--cert <SSL certificate file path>]
-d
: downloads model specific data before starting the service.-p <port>
: sets the port to<port>
. Overrides default value fromdeeppavlov/utils/settings/server_config.json
.--https
: use https instead of http. Overrides default value fromdeeppavlov/utils/settings/server_config.json
.--key <SSL key file path>
: path to SSL key file. Overrides default value fromdeeppavlov/utils/settings/server_config.json
.--cert <SSL certificate file path>
: path to SSL certificate file. Overrides default value fromdeeppavlov/utils/settings/server_config.json
.
The command will print the used host and port. Default web service properties
(host, port, POST request arguments) can be modified via changing
deeppavlov/utils/settings/server_config.json
file.
Warning
Starting from the 1.0.0rc2 model response format in riseapi mode matches Chainer
response format. To start model with the old format, give the COMPATIBILITY_MODE
environment variable any
non-empty value (e.g. COMPATIBILITY_MODE=true python -m deeppavlov riseapi ...
).
COMPATIBILITY_MODE
will be removed in DeepPavlov 1.2.0.
API routes¶
/probe¶
Send POST request to <host>:<port>/probe
to check if API is working. The
server will send a response ["Test passed"]
if it is working. Requests to
/probe
are not logged.
/api¶
To get model argument and response names send GET request to <host>:<port>/api
. Server
will return dict with model input and output names.
/docs¶
To interact with the REST API via graphical interface open
<host>:<port>/docs
in a browser (Swagger UI).
/metrics¶
Endpoint to monitor a running service using Prometheus. Metrics:
http_requests_count
: Counter, tracks number of processed requests. Labels:endpoint
,status_code
.http_requests_latency_seconds
: Histogram, tracks responses latency (only with 200 status code). Labels:endpoint
.http_requests_in_progress
: Gauge, tracks inprogress requests. Labels:endpoint
.
Advanced configuration¶
By modifying deeppavlov/utils/settings/server_config.json
you can change
host, port, POST request arguments and other properties of the API service.
Properties from common_defaults
section are used by default unless
they are overridden by model-specific properties, provided in model_defaults
section of the server_config.json
. Model-specific properties are bound
to the model by server_utils
label in metadata
section of the model
config. Value of server_utils
label from model config should match with
properties key from model_defaults
section of server_config.json
.
For example, adding metadata/server_utils
key to kbqa/kbqa_cq.json
with value KBQA will initiate the search of KBQA tag
at model_defaults
section of server_config.json
. Therefore, if this
section is present, all parameters with non empty (i.e. not ""
,
not []
etc.) values stored by this tag will overwrite the parameter values
in common_defaults
.
If model_args_names
parameter of server_config.json
is empty string,
then model argument names are provided as list from chainer/in
section of
the model config file, where arguments order corresponds to model API.
When inferencing model via REST api, JSON payload keys should match
model arguments names from chainer/in
section.
If model_args_names
parameter of server_config.json
is list, its values
are used as model argument names instead of the list from model config’s
chainer/in
section.
Here are POST request payload examples for some of the library models:
Model |
POST request JSON payload example |
---|---|
One argument models |
|
NER model |
{“x”:[“Elon Musk launched his cherry Tesla roadster to the Mars orbit”]} |
Intent classification model |
{“x”:[“I would like to go to a restaurant with Asian cuisine this evening”]} |
Automatic spelling correction model |
{“x”:[“errror”]} |
Ranking model |
{“x”:[“What is the average cost of life insurance services?”]} |
Goal-oriented bot |
{“x”:[“Hello, can you help me to find and book a restaurant this evening?”]} |
Multiple arguments models |
|
Question Answering model |
{“context_raw”:[“After 1765, growing philosophical and political differences strained the relationship between Great Britain and its colonies.”],
“question_raw”:[“What strained the relationship between Great Britain and its colonies?”]}
|
REST API Usage Example¶
To start server with squad_bert
model run:
python -m deeppavlov riseapi squad_bert -id
To get response from this model on another terminal run:
curl -X POST http://0.0.0.0:5000/model -H 'Content-Type: application/json' -d '{
"context_raw": [
"All work and no play makes Jack a dull boy.",
"I used to be an adventurer like you, then I took an arrow in the knee."
],
"question_raw": [
"What makes Jack a dull boy?",
"Who I used to be?"
]
}'