Rosemary doesn't tolerate freezing weather in some regions. Consider skipping moving it indoors and out with the seasons by growing it indoors all year long.