The cabinet seceretary role is "massively underpaid", according to Gus O'Donnell, who held the position from 2005-2011.
A public recruitment campaign is underway to replace Simon Case as cab sec and head of the civil service, with £200,000 a year on offer for the role.
Lord O'Donnell, who is part of the Civil Service Commission recruitment panel that will interview shortlisted candidates to be Case's, told BBC Radio 4's The Westminster Hour that the role is "an amazing job" but also "incredibly demanding".
"It's a huge job and it's massively underpaid in my view – given I've been paid a lot more since, to do a lot less," he added.
Case announced a week ago that he would stand down as cab sec at the end of the year for health reasons. His most recently published salary details show his earnings at £200,000-£205,000 in 2022-23.
O'Donnell told the BBC that whoever replaces Case will need to have a "good relationship" with the prime minister's chief of staff, now Morgan McSweeney. O'Donnell, who was speaking to the BBC before Gray's resignation at the weekend, said Gray "knows the civil service backwards".