The Moon is generally thought to have formed from the debris ejected by the impact of a planet-sized object with the proto-Earth towards the end of planetary accretion. Models of the impact process predict that the lunar material was disaggregated into a circumplanetary disk and that lunar accretion subsequently placed the Moon in a near-equatorial orbit. Forward integration of the lunar orbit from this initial state predicts a modern inclination at least an order of magnitude smaller than the lunar value--a long-standing discrepancy known as the lunar inclination problem. Here we show that the modern lunar orbit provides a sensitive record of gravitational interactions with Earth-crossing planetesimals that were not yet accreted at the time of the Moon-forming event. The currently observed lunar orbit can naturally be reproduced via interaction with a small quantity of mass (corresponding to 0.0075-0.015 Earth masses eventually accreted to the Earth) carried by a few bodies, consistent with the constraints and models of late accretion. Although the encounter process has a stochastic element, the observed value of the lunar inclination is among the most likely outcomes for a wide range of parameters. The excitation of the lunar orbit is most readily reproduced via collisionless encounters of planetesimals with the Earth-Moon system with strong dissipation of tidal energy on the early Earth. This mechanism obviates the need for previously proposed (but idealized) excitation mechanisms, places the Moon-forming event in the context of the formation of Earth, and constrains the pristineness of the dynamical state of the Earth-Moon system.