So two things - 1) I didn't even know we could use Regexp this easily. Very awesome. Thanks for sharing! :)
2) The function you shared doesn't quite render the result this post was going for:
debug.Print udfRemoveOrdinalNumber2("test of the 4th place on the 3rd spot")
Expected output ->...
In case this helps anyone else (I stumbled across this when trying solve a very similar scenario).. Here's some code I wrote that does the same thing, not 100% tested across every conceivable case so I've added a 'limiter' to prevent any strange infinite loops which can probably be removed...