Hi,
This is a limitation caused by the fact that RailClone is a seen by the G-Buffer as a single object. According to Cebas:
The 3dsmax G-Buffer does not support antialiasing among Material-IDs on the same object. This may cause jagged edges on the produced layers/channels if objects use multiple materials with different Material-IDs.
There are 2 workarounds:
- Split the object into multiple objects if possible, so that each object only has a single Material ID
 - use a render element instead (like Matte, or MultiMatte if it is V-Ray)
 
You can see the same antialiasing problem if a standard mesh has two materials applied - for example this box has a different material inset onto the front face. 

You can see the aliasing in that area is disabled too. 
You could therefore use a Render Element as suggested by Cebas, or as an alternative you can convert a RailClone object to standard Max instances using 
RailClone tools. I hope that helps, please let me know if I can help further with this. 
Many thanks!
Paul