This would happen because by default Max "captures" the keyboard for their shortcuts (i.e. '0' for the Render to Texture dialog).
When the interface enters in certain controls (as a numeric edit box), it's necessary to release the keyboard from Max, and capture it from the control (and back when the focus changes).
The Style Editor does the same, capturing and releasing the kebyboard when needed.
But Probably some Max event is recapturing the keyboard while the editor is open, causing this issue.
At least we can reproduce the same steps, it's difficult to fix it.
Anyway, this should happen only ocasionally, and the workaround is simple: just close and open the Style Editor.