Consider merging these comparisons with "in" to '0 in (weight_norm, adam_norm)'
160 adam_step.add_(p.data, alpha=group['weight_decay'])
161
162 adam_norm = torch.norm(adam_step)
163 if weight_norm == 0 or adam_norm == 0:164 trust_ratio = 1
165 else:
166 trust_ratio = weight_norm / adam_norm
Description
To check if a variable is equal to one of many values, combine the values into a tuple and check if the variable is contained in it instead of checking for equality against each of the values.
This is faster, less verbose, and more readable.
Bad practice
if x == 1 or x == 2 or x == 3:
do_something()
Recommended
if x in (1, 2, 3):
do_something()