La Trobe needed a partner, primarily to undertake the upbringing of his children and, as was typical of the era, he turned to the woman who had had much to do with the successful raising of his eldest child Agnes. This was Rose Isabelle de Meuron, Sophie’s youngest sibling and a widow herself. They married in Switzerland on 3 October 1855, nearly two years after Sophie’s death. A marriage with a deceased wife’s sister was not recognised in Britain at that time. In fact, it was illegal in Britain until the law was changed in 1907. But at that time, it was one of the ‘Prohibited Degrees of Marriage and Incest’ listed in the Book of Common Prayer of the Church of England and set in British law which prevented marriages between those connected through affinity, that is, by marriage but not by blood.

This marriage could have been a contributing reason why La Trobe was never offered another government posting, but since it was three years after his resignation that he had remarried, it would seem that the Colonial Office had no intention of giving him another position. Probably the most decisive factor in this was lack of a patron. In addition, there were a number of contributing factors which did much to colour the Colonial Office’s opinion of him. He was unpopular with many of the colonists in Port Phillip who did not hesitate to express their dissatisfaction to the Governor in Sydney and the Colonial Secretary in London; there were petitions to the Colonial Office to have him recalled, and even a serial advertisement run in The Argus in 1853: ‘Wanted a Governor. Apply to the people of Victoria’. He was frequently away from his post, exploring the colony and ‘inspecting’, away from the ties of his administrative responsibilities; and he had left a chaotic situation on the goldfields which resulted in the oppression and bloodshed of the Eureka Stockade under his successor, Governor Hotham.

Due to the lack of a pension at this time, La Trobe and his large family – including two more daughters Margaret and Isabelle – lived in a number of rented houses, one of the most notable being Ightham Mote in Kent, one of the few moated manor houses still remaining in Britain. Another residence was Whitbourne Court, site of the manor house of the Bishop of Hereford. In his final years, the family lived in Clapham House, a manor house in the village of Litlington near Eastbourne in Sussex.

It was there that Charles Joseph La Trobe died aged 74 on 4 December 1875, and was buried in the local churchyard of St Michael the Archangel. In 1879, Rose La Trobe built to his memory the Chapelle de l’Ermitage overlooking the glorious lake in Neuchâtel.