Adding Lambo tokenizer

dependency update
setting encoding to utf-8

Update dataset.py

Update setup.py

Update dataset.py

Łukasz Pszenny authored 2 years ago and Łukasz Pszenny committed 2 years ago

adding scripts to train new UD with instruction, fixing numpy problem on Mac(M1), fixing link to download models

Error while too many tokens are processed by BERT

instructions.

Extend installation and pre-trained models documentation, fix installation, update trainer to be in sync with AllenNLP 1.2.1.

(cherry picked from commit b17ac2317086531ecd3279ff5c47a6944e32838e)