Labour MPs Paul Flynn and Gordon Prentice, both members of the influential Public Administration Committee, have signed up to a motion of no confidence, which will be tabled on Monday or Tuesday by Tory Douglas Carswell.
Cabinet ministers, MPs and his predecessor Betty Boothroyd all called time on his calamitous handling of the MPs expenses scandal.
Mr Davies, the MP for Shipley, said: 'The Speaker hasn't just lost the confidence of the House, he has lost the confidence of people in the country.
'The institution of Parliament has become discredited. It is going to take years to recover. If we're going to turn it around, an essential first step is to remove a Speaker who has come to symbolise all that is wrong with Parliament.'
