How old is the chassis? Are your king pins ok? Are your spindle bearings good, and with no play?
Any signs of wear at your tie rod ends’ himes joints?
Your chassis should normally see changes or increments of 2 at any direction
I would totally set it up at 4 and 4, and that should be pretty good.
Please explain how you’re setting it on the ground, or on a jig, or table,and taking measurements?
Where are you setting the kart on? Remember that it has to flex, and roll, so that it’s not on a bind, plus where is the driver siting on? Make sure your driver is sitting down correctly, with his/her legs fully extended.
If no matter what you do, you’re still getting 6mm in increments, then there is something seriously wrong with the chassis, or loose/worn out components.