deeppavlov.core.data

DatasetReader, Vocab, DataLearningIterator and DataFittingIterator classes.

class deeppavlov.core.data.dataset_reader.DatasetReader[source]

An abstract class for reading data from some location and construction of a dataset.

class deeppavlov.core.data.data_fitting_iterator.DataFittingIterator(data: List[str], doc_ids: Optional[List[Any]] = None, seed: Optional[int] = None, shuffle: bool = True, *args, **kwargs)[source]

Dataset iterator for fitting estimator models, like vocabs, kNN, vectorizers. Data is passed as a list of strings(documents). Generate batches (for large datasets).

Parameters
  • data – list of documents

  • doc_ids – provided document ids

  • seed – random seed for data shuffling

  • shuffle – whether to shuffle data during batching

shuffle

whether to shuffle data during batching

random

instance of Random initialized with a seed

data

list of documents

doc_ids

provided by a user ids or generated automatically ids

class deeppavlov.core.data.data_learning_iterator.DataLearningIterator(data: Dict[str, List[Tuple[Any, Any]]], seed: Optional[int] = None, shuffle: bool = True, *args, **kwargs)[source]

Dataset iterator for learning models, e. g. neural networks.

Parameters
  • data – list of (x, y) pairs for every data type in 'train', 'valid' and 'test'

  • seed – random seed for data shuffling

  • shuffle – whether to shuffle data during batching

shuffle

whether to shuffle data during batching

random

instance of Random initialized with a seed

class deeppavlov.core.data.simple_vocab.SimpleVocabulary(special_tokens: Tuple[str, ...] = (), max_tokens: int = 1073741824, min_freq: int = 0, pad_with_zeros: bool = False, unk_token: Optional[str] = None, freq_drop_load: Optional[bool] = None, *args, **kwargs)[source]

Implements simple vocabulary.

Parameters
  • special_tokens – tuple of tokens that shouldn’t be counted.

  • max_tokens – upper bound for number of tokens in the vocabulary.

  • min_freq – minimal count of a token (except special tokens).

  • pad_with_zeros – if True, then batch of elements will be padded with zeros up to length of the longest element in batch.

  • unk_token – label assigned to unknown tokens.

  • freq_drop_load – if True, then frequencies of tokens are set to min_freq on the model load.