{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Context Question Answering\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deeppavlov/DeepPavlov/blob/master/docs/features/models/SQuAD.ipynb)\n",
    "\n",
    "[![Medium](https://img.shields.io/badge/Medium-12100E?style=for-the-badge&logo=medium&logoColor=white)](https://medium.com/deeppavlov/developing-qa-systems-for-any-language-with-deeppavlov-a9033d5231a8)\n",
    "\n",
    "# Table of contents \n",
    "\n",
    "1. [Introduction to the task](#1.-Introduction-to-the-task)\n",
    "\n",
    "2. [Get started with the model](#2.-Get-started-with-the-model)\n",
    "\n",
    "3. [Models list](#3.-Models-list)\n",
    "\n",
    "4. [Use the model for prediction](#4.-Use-the-model-for-prediction)\n",
    "\n",
    "    4.1. [Predict using Python](#4.1-Predict-using-Python)\n",
    "    \n",
    "    4.2. [Predict using CLI](#4.2-Predict-using-CLI)\n",
    "     \n",
    "5. [Train the model on your data](#5.-Train-the-model-on-your-data)\n",
    "    \n",
    "    5.1. [from Python](#5.1-Train-your-model-from-Python)\n",
    "    \n",
    "    5.2. [from CLI](#5.2-Train-your-model-from-CLI)\n",
    "    \n",
    "6. [Evaluate](#6.-Evaluate)\n",
    "    \n",
    "    6.1. [from Python](#6.1-Evaluate-from-Python)\n",
    "    \n",
    "    6.2. [from CLI](#6.2-Evaluate-from-CLI)\n",
    "\n",
    "# 1. Introduction to the task\n",
    "\n",
    "Context Question Answering is a task of finding a fragment with an answer to a question in a given segment of context.\n",
    "\n",
    "**Context**:\n",
    "\n",
    "```\n",
    "In meteorology, precipitation is any product of the condensation \n",
    "of atmospheric water vapor that falls under gravity. The main forms \n",
    "of precipitation include drizzle, rain, sleet, snow, graupel and hail… \n",
    "Precipitation forms as smaller droplets coalesce via collision with \n",
    "other rain drops or ice crystals within a cloud. Short, intense periods \n",
    "of rain in scattered locations are called “showers”.\n",
    "```\n",
    "\n",
    "**Question**:\n",
    "```\n",
    "Where do water droplets collide with ice crystals to form precipitation?\n",
    "```\n",
    "\n",
    "**Answer**: \n",
    "```\n",
    "within a cloud\n",
    "```\n",
    "\n",
    "Datasets that follow this task format:\n",
    "\n",
    "- [Stanford Question Answering Dataset (SQuAD) (EN)](https://rajpurkar.github.io/SQuAD-explorer/)\n",
    "\n",
    "- [SberQuAD (RU)](https://paperswithcode.com/dataset/sberquad)\n",
    "\n",
    "# 2. Get started with the model\n",
    "\n",
    "First make sure you have the DeepPavlov Library installed.\n",
    "[More info about the first installation.](http://docs.deeppavlov.ai/en/master/intro/installation.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then make sure that all the required packages for the model are installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov install squad_bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`squad_bert` is the name of the model's *config_file*. [What is a Config File?](http://docs.deeppavlov.ai/en/master/intro/configuration.html) \n",
    "\n",
    "Configuration file defines the model and describes its hyperparameters. To use another model, change the name of the *config_file* here and further.\n",
    "The full list of the models with their config names can be found in the [table](#3.-Models-list).\n",
    "\n",
    "# 3. Models list\n",
    "\n",
    "The table presents a list of all of the Context Question Answering models available in DeepPavlov Library.\n",
    "\n",
    "| Config name  | Dataset | Language | Model Size | F1 score | EM  |\n",
    "| :--- | --- | --- | --- | --- | ---: |\n",
    "| squad_bert | SQuAD v1.1 | En | 1.3 GB | 88.86 | 81.49 |\n",
    "| qa_squad2_bert | SQuAD v2.0 | En | 1.3 GB | 83.56 | 75.54 |\n",
    "| qa_multisberquad_bert | MultiSQuAD | Multi | 2 GB | 80.76 | 63.81 |\n",
    "| squad_ru_bert | SberQuAD | Ru | 2.0 GB | 84.71 | 66.21 |\n",
    "| squad_ru_convers_distilrubert_2L | SberQuAD | Ru | 1.2 GB | 65.20 | 44.52 |\n",
    "| squad_ru_convers_distilrubert_6L | SberQuAD | Ru | 1.6 GB | 80.57 | 61.54 |\n",
    "\n",
    "\n",
    "# 4. Use the model for prediction\n",
    "\n",
    "## 4.1 Predict using Python\n",
    "\n",
    "After [installing](#2.-Get-started-with-the-model) the model, build it from the config and predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import build_model\n",
    "\n",
    "model = build_model('squad_bert', download=True, install=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Input**: List[context], List[question]\n",
    "\n",
    "**Output**: List[answer, start_character, logit]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['a library for NLP and dialog systems'], [14], [200928.390625]]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model(['DeepPavlov is a library for NLP and dialog systems.'], ['What is DeepPavlov?'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Predict using CLI\n",
    "\n",
    "You can also get predictions in an interactive mode through CLI (Command Line Interface)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov interact squad_bert -d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`-d` is an optional download key (alternative to `download=True` in Python code). The key `-d` is used to download the pre-trained model along with embeddings and all other files needed to run the model.\n",
    "\n",
    "Or make predictions for samples from *stdin*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov predict squad_bert -f <file-name>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Train the model on your data\n",
    "\n",
    "\n",
    "## 5.1 Train your model from Python\n",
    "\n",
    "### Provide your data path\n",
    "\n",
    "To train the model on your data, you need to change the path to the training data in the *config_file*.\n",
    "\n",
    "Parse the *config_file* and change the path to your data from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~/.deeppavlov/downloads/squad/\n"
     ]
    }
   ],
   "source": [
    "from deeppavlov import train_model\n",
    "from deeppavlov.core.commands.utils import parse_config\n",
    "\n",
    "model_config = parse_config('squad_bert')\n",
    "\n",
    "#  dataset that the model was trained on\n",
    "print(model_config['dataset_reader']['data_path'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide a *data_path* to your own dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download and unzip a new example dataset\n",
    "!wget http://files.deeppavlov.ai/datasets/squad-v1.1.tar.gz\n",
    "!tar -xzvf \"squad-v1.1.tar.gz\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you want to provide your own dataset, it should have the same format as the SQuAD dataset downloaded in this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# provide a path to the train file\n",
    "model_config['dataset_reader']['data_path'] = '/contents/train-v1.1.json'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SQuAD dataset info\n",
    "\n",
    "There are *two* versions of the SQuAD dataset available for training at the moment: \n",
    "\n",
    "- [SQuAD 1.1](https://arxiv.org/abs/1606.05250) contains 107,785 question-answer pairs on 536 articles. Dataset size: `33.52 MiB`.\n",
    "\n",
    "- [SQuAD 2.0](https://arxiv.org/abs/1806.03822) combines all of the questions from SQuAD 1.1 with over 50,000 un-answerable questions written adversarially by crowdworkers. Dataset size: `44.34 MiB`.\n",
    "\n",
    "### Train the model using new config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = train_model(model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use your model for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['a library for NLP and dialog systems'], [14], [200928.390625]]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model(['DeepPavlov is a library for NLP and dialog systems.'], ['What is DeepPavlov?'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 Train your model from CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m deeppavlov train squad_bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Evaluate\n",
    "\n",
    "## 6.1 Evaluate from Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deeppavlov import evaluate_model\n",
    "\n",
    "model = evaluate_model('squad_bert', download=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 Evaluate from CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m deeppavlov evaluate squad_bert -d"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}