It has taken a bit of time and digging, but here are details of the Primroses:
Henry William Primrose, born 22 August 1846 in Barnbougle, Midlothian, Scotland, died 17 June 1923 in London. Educated at Glenalmond, and Balliol College, Oxford. Later Rt Hon Sir Henry Primrose, Treasury civil servant, and chairman of the Board of Inland Revenue.
There is no-one named CE Primrose in the 1871 London census, but allowing for a misprint of his first initial, GE Primrose would be Gilbert Edward Primrose, born Barnbougle, Midlothian, Scotland, on 27 February 1848, younger brother of HW Primrose (above). Educated at Glenalmond College. Went to Australia where he settled at Emu Creek, near Brisbane. He later returned to England and died in Worthing on 16 February 1935.