Hi, it's normal than IR takes more time to initialize with RC objects, because they must be rebuilt when render starts (and once more when it stops).
Instead, poly objects are static geometry which is initialized much faster.
Anyway, this should be noticeable only when IR starts. Once running, perfomance should be similar, at least you modify some parameter in the RC object (then it must be rebuilt again).
Please, would you detail a bit more you experience with this ?
Does it happen also moving the IR view ?