This Access behavior you describe is the default behavior and somewhat difficult to circumvent except by using an unbound form, which is labour intensive. What I might do in this case is add a Yes/No field 'Confirmed' to the table. When the form closes always run something like...
CurrentDB.Execute "DELETE FROM YourTable WHERE Not Confirmed"
To ensure a record is saved, set it's Confirmed field to True.