In the before update of the sub form:
Private Sub Form_BeforeUpdate(Cancel As Integer)
If DCount("[InvoiceID]", "Transactions", "[InvoiceID] = " & Forms!MainForm![InvoiceID]) > 10 Then
MsgBox "You can't have more than 10 transactions per Invoice"
Cancel = True
End If
End Sub