Looks like if you already have some values in a field, and if you add input mask, you can make a mistake and add duplicate values, because of input mask.

So I had to remove input mask and set it as text.
This happened with the phone numbers. I already had list of numbers in my table with xxx-xxx-xxxx format, and the input mask was (xxx) xxx-xxxx, therefore it couldn't catch the duplicates.

Just wanted to share this.

I think best way to overcome this would be by telling access to match numbers only or disregard anything in between number like dashes, dots, etc.