Finalised subword splitting.
Showing
- README.md 30 additions, 7 deletionsREADME.md
- pyproject.toml 1 addition, 1 deletionpyproject.toml
- src/lambo/data/token.py 5 additions, 1 deletionsrc/lambo/data/token.py
- src/lambo/examples/__init__.py 1 addition, 0 deletionssrc/lambo/examples/__init__.py
- src/lambo/examples/run_training_splitting.py 9 additions, 10 deletionssrc/lambo/examples/run_training_splitting.py
- src/lambo/examples/run_usage.py 2 additions, 14 deletionssrc/lambo/examples/run_usage.py
- src/lambo/segmenter/lambo.py 21 additions, 13 deletionssrc/lambo/segmenter/lambo.py
- src/lambo/subwords/__init__.py 3 additions, 0 deletionssrc/lambo/subwords/__init__.py
- src/lambo/subwords/model.py 17 additions, 11 deletionssrc/lambo/subwords/model.py
- src/lambo/subwords/preprocessing.py 39 additions, 20 deletionssrc/lambo/subwords/preprocessing.py
- src/lambo/subwords/splitter.py 23 additions, 13 deletionssrc/lambo/subwords/splitter.py
- src/lambo/subwords/train.py 5 additions, 5 deletionssrc/lambo/subwords/train.py
- src/lambo/utils/download.py 34 additions, 1 deletionsrc/lambo/utils/download.py
- src/lambo/utils/generate_languages_txt.py 6 additions, 0 deletionssrc/lambo/utils/generate_languages_txt.py
Please register or sign in to comment