I had a problem on one of my project, when i had to select 1 FP object to render mask on Vray 5 (A FP with trees, separate from a building). The id object with multimatte didn´t work either, and in the end i had to set a wirecolour on the FP and everything else was pure black.
Since i cannot share the image or files, i´ve tried to replicate the problem with a simple scene and it is not working also. When i select "render mask" with the green box objects on the scene, it selects the other FP object (Pink boxes), just like my project scene.
The files and image are attached.