Adding Lambo tokenizer

dependency update
setting encoding to utf-8

Update dataset.py

Update setup.py

Update dataset.py

Łukasz Pszenny authored 2 years ago and Łukasz Pszenny committed 2 years ago

adding scripts to train new UD with instruction, fixing numpy problem on Mac(M1), fixing link to download models

Łukasz Pszenny authored 2 years ago and Łukasz Pszenny committed 2 years ago

Update README.md

small fix in performance.md

small fix in performance.md - got rid of models that were trained on data with sentences of form "____"

Error while too many tokens are processed by BERT