Add TokenIndexers and Tokenizers
Showing
- combo/data/__init__.py 2 additions, 5 deletionscombo/data/__init__.py
- combo/data/token.py 0 additions, 2 deletionscombo/data/token.py
- combo/data/token_indexers/__init__.py 1 addition, 3 deletionscombo/data/token_indexers/__init__.py
- combo/data/token_indexers/base_indexer.py 0 additions, 6 deletionscombo/data/token_indexers/base_indexer.py
- combo/data/token_indexers/pretrained_transformer_fixed_mismatched_indexer.py 120 additions, 0 deletions...dexers/pretrained_transformer_fixed_mismatched_indexer.py
- combo/data/token_indexers/pretrained_transformer_indexer.py 243 additions, 0 deletionscombo/data/token_indexers/pretrained_transformer_indexer.py
- combo/data/token_indexers/pretrained_transformer_mismatched_indexer.py 113 additions, 6 deletions...ken_indexers/pretrained_transformer_mismatched_indexer.py
- combo/data/token_indexers/token_characters_indexer.py 146 additions, 3 deletionscombo/data/token_indexers/token_characters_indexer.py
- combo/data/token_indexers/token_const_padding_characters_indexer.py 62 additions, 0 deletions.../token_indexers/token_const_padding_characters_indexer.py
- combo/data/token_indexers/token_features_indexer.py 73 additions, 3 deletionscombo/data/token_indexers/token_features_indexer.py
- combo/data/token_indexers/token_indexer.py 124 additions, 0 deletionscombo/data/token_indexers/token_indexer.py
- combo/data/tokenizers/__init__.py 2 additions, 0 deletionscombo/data/tokenizers/__init__.py
- combo/data/tokenizers/character_tokenizer.py 11 additions, 0 deletionscombo/data/tokenizers/character_tokenizer.py
- combo/data/tokenizers/pretrained_transformer_tokenizer.py 488 additions, 0 deletionscombo/data/tokenizers/pretrained_transformer_tokenizer.py
- combo/data/tokenizers/tokenizer.py 15 additions, 0 deletionscombo/data/tokenizers/tokenizer.py
- combo/utils/cached_transformers.py 0 additions, 0 deletionscombo/utils/cached_transformers.py
- combo/utils/sequence.py 14 additions, 0 deletionscombo/utils/sequence.py
- requirements.txt 4 additions, 1 deletionrequirements.txt
Please register or sign in to comment