Skip to content
Snippets Groups Projects
Commit 31efb090 authored by Adam Radziszewski's avatar Adam Radziszewski
Browse files

fix get_next_token having sentences delete their tokens before giving tokens out

parent d91e0cce
Branches
No related merge requests found
......@@ -290,6 +290,7 @@ Token* BufferedSentenceReader::get_next_token()
if (s) {
std::copy(s->tokens().begin(), s->tokens().end(),
std::back_inserter(token_buf_));
s->release_tokens();
} else {
more = false;
}
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment