This is an internal error of the shader for Mental Ray. Theorically, it must never happen.
What type of objects are you using as trees: billboards, meshes, proxies... ? What version of Forest and Max have you installed ?
To fix this problem as soon as possible, please send us a copy of the scene. If the file is too big to email it, i can give you a link to our FTP server.