Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
C
combo
Manage
Activity
Members
Labels
Plan
Issues
20
Issue boards
Milestones
Wiki
Redmine
Code
Merge requests
2
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Syntactic Tools
combo
Commits
8f0f24ac
Commit
8f0f24ac
authored
4 years ago
by
Mateusz Klimaszewski
Browse files
Options
Downloads
Patches
Plain Diff
Simplify batch sampler.
parent
2b72300b
Branches
Branches containing commit
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
combo/data/samplers/samplers.py
+1
-14
1 addition, 14 deletions
combo/data/samplers/samplers.py
with
1 addition
and
14 deletions
combo/data/samplers/samplers.py
+
1
−
14
View file @
8f0f24ac
...
...
@@ -35,17 +35,4 @@ class TokenCountBatchSampler(allen_data.BatchSampler):
return
batches
def
get_num_batches
(
self
,
instances
:
Sequence
[
data
.
Instance
])
->
int
:
dataset
=
list
(
instances
)
batches
=
[]
batch
=
[]
words_count
=
0
lengths
=
[
len
(
instance
.
fields
[
"
sentence
"
].
tokens
)
for
instance
in
dataset
]
argsorted_lengths
=
np
.
argsort
(
lengths
)
for
idx
in
argsorted_lengths
:
words_count
+=
lengths
[
idx
]
batch
.
append
(
idx
)
if
words_count
>
self
.
_word_batch_size
:
batches
.
append
(
batch
)
words_count
=
0
batch
=
[]
return
len
(
batches
)
return
sum
(
1
for
_
in
self
.
get_batch_indices
(
instances
))
This diff is collapsed.
Click to expand it.
Preview
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment