Well, we've found what is causing the warning message (and probably the backburner problem):
We create an user data block in the Mental Ray shader, to store certain data required by Forest Color. For some reason, this block seems too large to Mental Ray, although it's only 23 Mb. and only exists one by plant model. The size of the data block is proportional to the tree's mesh size. This is the reason it fails only with certain models (plants of roughly 2 Mill. faces).
Checking the Mental Ray documentation, this seems the only method to create user data blocks, so from our side everything seems correct. I have not idea why they have introduced that ridiculous size limit. We'll open a support case in Autodesk about this issue.
Please, try the attached shader (FP 4.0.3, Max 2013/2014 64 bit only) and tell me if something changes. This shader doesn't create the data block, but the Tint/Map by Element functionality of Forest Color is disabled, so use it only if you don't use that features. You must copy the DLL at <maxdir>\NVIDIA\shaders_autoload\mentalray\shaders.