What does Thoreau mean when he says "nor can he get off by hammering marble instead?"?

Thoreau says "Every man is the builder of a Temple, called his body, to the god he worships, after a style purely his own, nor can he get off by hammering marble instead." I understand the first part of the quote, but what does he mean when he says "nor can he get off by hammering marble instead?"