deeppavlov.core.data¶
DatasetReader, Vocab, DataLearningIterator and DataFittingIterator classes.
-
class
deeppavlov.core.data.dataset_reader.
DatasetReader
[source]¶ An abstract class for reading data from some location and construction of a dataset.
-
class
deeppavlov.core.data.data_fitting_iterator.
DataFittingIterator
(data: List[str], doc_ids: List[Any] = None, seed: int = None, shuffle: bool = True, *args, **kwargs)[source]¶ Dataset iterator for fitting estimator models, like vocabs, kNN, vectorizers. Data is passed as a list of strings(documents). Generate batches (for large datasets).
Parameters: - data – list of documents
- doc_ids – provided document ids
- seed – random seed for data shuffling
- shuffle – whether to shuffle data during batching
-
shuffle
¶ whether to shuffle data during batching
-
random
¶ instance of
Random
initialized with a seed
-
data
¶ list of documents
-
doc_ids
¶ provided by a user ids or generated automatically ids
-
class
deeppavlov.core.data.data_learning_iterator.
DataLearningIterator
(data: Dict[str, List[Tuple[Any, Any]]], seed: int = None, shuffle: bool = True, *args, **kwargs)[source]¶ Dataset iterator for learning models, e. g. neural networks.
Parameters: - data – list of (x, y) pairs for every data type in
'train'
,'valid'
and'test'
- seed – random seed for data shuffling
- shuffle – whether to shuffle data during batching
-
shuffle
¶ whether to shuffle data during batching
-
random
¶ instance of
Random
initialized with a seed
- data – list of (x, y) pairs for every data type in
-
class
deeppavlov.core.data.sqlite_database.
Sqlite3Database
(save_path: str, table_name: str, primary_keys: List[str], keys: List[str] = None, unknown_value: str = 'UNK', *args, **kwargs)[source]¶ Loads and trains sqlite table of any items (with name
table_name
and pathsave_path
).Primary (unique) keys must be specified, all other keys are infered from data. Batch here is a list of dictionaries, where each dictionary corresponds to an item. If an item doesn’t contain values for all keys, then missing values will be stored with
unknown_value
.Parameters: - save_path – sqlite database path.
- table_name – name of the sqlite table.
- primary_keys – list of table primary keys’ names.
- keys – all table keys’ names.
- unknown_value – value assigned to missing item values.
- **kwargs – parameters passed to parent
Estimator
class.
-
class
deeppavlov.core.data.vocab.
DefaultVocabulary
(save_path: str, load_path: str, level: str = 'token', special_tokens: List[str] = [], default_token: str = None, tokenizer: Callable = None, min_freq: int = 0, **kwargs)[source]¶ Implements vocabulary of tokens, chars or other structeres.
Parameters: - level – level of operation can be tokens (
'token'
) or chars ('char'
). - special_tokens – tuple of tokens that shouldn’t be counted.
- default_token – label assigned to unknown tokens.
- tokenizer – callable used to get tokens out of string.
- min_freq – minimal count of a token (except special tokens).
- level – level of operation can be tokens (
-
class
deeppavlov.core.data.simple_vocab.
SimpleVocabulary
(special_tokens: Tuple[str, ...] = (), max_tokens: int = 1073741824, min_freq: int = 0, pad_with_zeros: bool = False, unk_token: Optional[str] = None, freq_drop_load: Optional[bool] = None, *args, **kwargs)[source]¶ Implements simple vocabulary.