If the earth’s orbit is affected, then the moon’s orbit will be affected too. Moon is actually drifting away at 3.8 cm per year rate. This is however due to the tidal effects on the earth due to moon’s gravity. Such effects causes earth to bulge at its surface (causing tides) on both ends. But as the earth has larger water and it rotates faster than the moon, the bulge recedes a little compared to the crust. The water mass then exerts a pull on the moon and accelerates it. The moon now steals the earth’s energy and angular momentum here and goes to a slightly higher orbit. The moon’s gravity in turn pulls on the bulge and creates a friction to slow down the earth’s rotation. If you further read about the tidal forces between Earth, Moon and Sun and you will understand this better. This will be a good start.