Consider merging these comparisons with "in" to '0 in (weight_norm, adam_norm)'
163 adam_step.add_(p.data, alpha=group['weight_decay'])
164
165 adam_norm = torch.norm(adam_step)
166 if weight_norm == 0 or adam_norm == 0:167 trust_ratio = 1
168 else:
169 trust_ratio = weight_norm / adam_norm
Description
To check if a variable is equal to one of many values, combine the values into a tuple and check if the variable is contained in it instead of checking for equality against each of the values.
This is faster, less verbose, and more readable.
Bad practice
if x == 1 or x == 2 or x == 3:
do_something()
Recommended
if x in (1, 2, 3):
do_something()