Hi Ian. Karts can be set up for dirt although most on this forum are more familiar with their use on asphalt. This site regularly does stories on the So Cal Oval Karters who run on dirt and I am sure there are other organizations as well. Perhaps you could get in touch with folks there for recommendations.
I would say the biggest issues are lack of a differential and, more importantly, safety. Karts are low to the ground and hard to see by persons in cars, especially around turns, etc. If noise is an issue in your neighborhood then I would suggest a 4-stroke engine such as the Briggs LO206.
That said, have fun. Welcome to the sport! And please don’t get the impression karters are a bunch of cranky naysayers!