Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
T
toki
Manage
Activity
Members
Labels
Plan
Issues
2
Issue boards
Milestones
Wiki
Redmine
Code
Merge requests
0
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Analysers
toki
Commits
fb529811
Commit
fb529811
authored
14 years ago
by
ilor
Browse files
Options
Downloads
Patches
Plain Diff
Handle U+200B (zero width space) in Toki
parent
ad4f1d62
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
libtoki/tokenizer/whitespacetokenizer.cpp
+3
-1
3 additions, 1 deletion
libtoki/tokenizer/whitespacetokenizer.cpp
with
3 additions
and
1 deletion
libtoki/tokenizer/whitespacetokenizer.cpp
+
3
−
1
View file @
fb529811
...
...
@@ -119,7 +119,9 @@ namespace Toki {
int
ws
=
0
;
int
nl
=
0
;
while
(
input
().
has_more_chars
())
{
UChar
u
=
input
().
peek_next_char
();
if
(
u
==
0xfeff
)
{
//BOM mark (aka ZERO WIDTH NO-BREAK SPACE)
if
(
u
==
0xfeff
||
u
==
0x200b
)
{
//U+FEFF BOM mark (aka ZERO WIDTH NO-BREAK SPACE)
//U+200B ZERO WIDTH SPACE
//do not increment ws. BOM's are skipped entirely
input
().
get_next_char
();
}
else
if
(
!
u_isUWhiteSpace
(
u
))
{
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment