I have a report with grouped records and a calculated text box below each group indicating the number of records in that group.

How do I sort the report by the number of records in each group?

The number of records in a group isn't calculated until the report is formatted so there is no way to get the report to sort by this number.

You would need to write a query to calculate the number of records in the group. Then join that query to the query you are using as the report's recordsource. You would then have the number of records in a group available before the report starts and will be able to sort on it since it is in the report's recordsource.

The counting query would be:
Select fld1, fld2, Count(*) AS countOfGroup
From YourTable
Group by fld1, fld2;

Then join the counting query to your main query on fld1 and fld2. Each row in the recordset will then contain the count for the entire group.