PNE took the lead with a goal from Ryan Crosdale on 10 minutes with Leicester equalised on 15 minutes has Shane Byrne.
PNE regained the lead on 24 minutes with a penalty from Michael McLellan before Oliver Norburn made it 2-2.
The second half saw the Foxes take the lead for the first time when Jeff Schlupp made it 3-2 before the Lilywhites made it 3-3 on 90 minutes via Luke Clarke to take the game into Extra Time.
Unfortunately luck deserted PNE in Extra Time as they were reduced to ten men on 113 minutes when Billington was carried off and all subs had been used.
Just before the end Leicester was awarded a penalty and scored to make it 4-3. It looked like Preston might get a second penalty at the death but was waived away by the referee. The protests by PNE captain Clucas led to him being red carded and PNE playing the last few seconds with nine men.
Leicester will now play Sheffield United at home on Saturday for a place in the Semi-Final.