Not stupid at all! Time travel as a concept would take a baffling amount of work. The timeline of the universe is a number of causally related quantum states: one state creating the next.
For time travel to work, you'd be taken "back" to a different quantum state of the universe, by unmooring yourself and your time travel device from a given organization of cosmic waveforms, and moving to another. But you also have to move *through space* or you'll fall out of geo-sync with the Earth!
@MLClark this is a great answer, thank you