According to the lines in the poem, the phantom listeners are the ghosts who lived in r the lonely house of the forest. They were the host and the owner of the house while they were alive. But even after their death they continue to live there as a group.
In the dark silent night they listen attentively to the sound of the traveller which vibrates the air, by standing as a crowd under the dull moonlight on the dark stair that leads down to the empty hall.
