So Having your library window open, while adjusting parameters of a forest object, is causing that dialog box to be displayed behind the library window, which cannot be moved or closed, until you click the OK button in the dialog box. Easily solved, if you find yourself in this situation, just press the Return/Enter button and the dialog with focus, will close... Yes it is hard to know what appears to be freezing Max, but pressing Enter/Return, is the first solution to try.
I have a question though.. Why is the library window open at that time... The reason I ask is, because that dialog usually only appears, once a library item is chosen, after which the library window closes.. or you adjust parameters of the forest object, above a certain limit.... and this might explain why Rokas is having a problem, trying to replicate it.