if you simply remove the battery but connect the circuit, the current will flow, in the opposite direction which the battery provided, because of the capacitor (which is discharging). however since the cap. is simply a reserve of power (or electrons), as it discharges it gives up electrons, reducing its potential and thus reducing the current. The current asymptotically comes to a stop, much like how it behaves when the capacitor is charging. If you dont connect the circuit after you remove the battery, but you just leave the capacitor there, it wont discharge immediately but it may experience dielectric breakdown in which the electrons essentially "jump" between the two plates, taking a path through the dielectric material (which can be air) in order to discharge the potential. This is the phenomenon which results in lightning. This would NOT, however, occur in a vacuum, but it would occur even in a vacuum if you shined light above a threshhold frequency onto the anode. This is a specific application of the photoelectric effect and is something ive seen twice on AAMC passages so id be familiar with it.
Also, spooge you mentioned "In reality the potential remains for a while after being removed from the source" - in theory the potential doesnt just remain for a while, it remains forever - the potential as well as the current declines asymptotically during discharge and charging of the cap. we say that the cap. is completely discharged however when the current it produces is "close enough" to 0.