It wont do any harm perse, other than perhaps gum up the ring and or powervalve sooner. Arguably it could affect your jetting too.. Probably the biggest debate would be on issues that might occur when you switch from castor to syth.
Just go with the Motul or XPS. You could sell the castor on ebay or EKN or something. Keep a bottle or two for the KPV if you still have it.