I'm guessing you mean that after closing the form and then subsequently re-opening it, all the "deleted" numbers are back, which is the way you example you posted worked. That's because the RemoveItem (and AddItem) only affects the RowSource until the form is closed. When it re-opens the RowSource is as listed in the Properties Pane. You can only change this in Design View.
If you want to permanently deleted the values you'll have to base your ListBox on a table and actually delete the valuse from the table, then Requery then ListBox.