Let me try to explain this.
I have a Book Table with a BookID, Title, ISBN, & Rented (Boolean) value.
I also have a Transaction Table with a TransID, Date, and Student ID Field.
I also have a Student Table with a StudentID, First and Last Name.

I have a subform that does the following:
Once you click on a student, a subform below (which is based on a query), lists all the transactions that that student has made (renting and returning books).
But I want to be able to create an expression that will signal whether the transaction that occurred was a result of a book being rented or the book being returned (want it to be displayed in the subbform).

Thanks

Perhaps you should store the "rented" value in the Transaction Table. Otherwise it sounds like you might need a way to relate the Book Table and Transaction Table by some foreign key. Then you could query them together to display the information.

Assuming that your Rented field is set to true when a book is rented and to false when the book is returned, you can use that field. If you add an order by clause to the subform's recordsource query, you can sort first by the Rented field and second by the Title. That way all the books still "out" will show up first in the list.