If we consider the Universe to be a single, unbroken chain of cause and effect, and time is just our perception of that sequence as we move forward along the chain, then there’s no need to worry. By this interpretation, the past, present and future must all be fixed by the immutable laws of physics. Even seemingly random events, such as radioactive decay, would be predetermined and must play out in exactly the same way, even though we can’t predict them.
If you travel back in time to meet your younger self, it must follow that this has already happened in your past. If you don’t remember it, then maybe you were in disguise or you wiped your memory afterwards, but whatever effect this had on the future is already locked in.