This is a limitation caused by the fact that RailClone is a seen by the G-Buffer as a single object. According to Cebas:
The 3dsmax G-Buffer does not support antialiasing among Material-IDs on the same object. This may cause jagged edges on the produced layers/channels if objects use multiple materials with different Material-IDs.
There are 2 workarounds:
- Split the object into multiple objects if possible, so that each object only has a single Material ID
- use a render element instead (like Matte, or MultiMatte if it is V-Ray)
You can see the same antialiasing problem if a standard mesh has two materials applied - for example this box has a different material inset onto the front face.
You can see the aliasing in that area is disabled too.
You could therefore use a Render Element as suggested by Cebas, or as an alternative you can convert a RailClone object to standard Max instances using RailClone tools.
I hope that helps, please let me know if I can help further with this.