Poor Performance in Recordset Loop

Cullihall

Registered User.
Local time
Today, 17:01
Joined
Oct 28, 2009
Messages
33
Hi. I'm wondering if someone would please let me know if there is a better way to run the loop in the following code. My code is extremely slow on a core 2 duo 3.16GHz cpu with 2G RAM. My goal is to loop through a table of approx 5000 records, perform a calculation for a field and update that field for the current record, then move to the next record.
Code:
' Set the recordset the the appropriate table
Set rs1 = CurrentDb.OpenRecordset("t_POCheckAllItemsCalculations", dbOpenDynaset)
    ' Select first record in the recordset
    rs1.MoveFirst
        ' Loop until it is not the end of the recordset
        Do While Not rs1.EOF
        ' Reset variables
        ' Fill variable with the current date
        theDate = Date
        ' Fill variable with the current day of the week
        theWkDay = Format(theDate, "dddd")
        ' Set temp variable to 1
        tempVar = 1
        ' Set cases to be sold to 0
        qCsToBeSold = 0
        ' Get the current record number
        curRecord1 = rs1.AbsolutePosition + 1
            On Error Resume Next
                ' Loop through until the days until delivery has been reached
                Do While tempVar <= rs1.Fields("[Days Until Delivery]").Value + 1
                ' Look up the value from the table using a formula
                qCsToBeSold = qCsToBeSold + DLookup("[" & theWkDay & " Average]", "[t_CombinedVelocityTotals]", "[Item Code] = " & rs1.Fields("[Item Code]").Value)
                ' Add one to the date variable in order to move to the next day of the week
                theDate = theDate + 1
                ' Extract the day of the week from the date variable
                theWkDay = Format(theDate, "dddd")
                ' Add 1 to the temp variable to use as a counter
                tempVar = tempVar + 1
                Loop
    ' Populate the table with the records
    With rs1
        .Edit
        ![Cases to be Sold] = qCsToBeSold
    End With
    ' Update the table
    rs1.Update
    ' Move to next record
    rs1.MoveNext
        Loop
 
Calculations shouldn't be stored. If you need to calculate something, it should be done in a query. What exactly are you trying to calculate?
 
can you not achieve this functionality with an "update" query?
 
Calculations shouldn't be stored. If you need to calculate something, it should be done in a query. What exactly are you trying to calculate?

I have a table containing average cases sold for each day of the week (Sunday to Saturday). What I'm doing is calculating the estimated cases to be sold from today until the next delivery of product([Days Until Delivery]) . So if today is Wednesday, I want it to add the average cases sold on Wednesday, the average cases sold on Thursday, and so on, until it reaches the date of my next delivery.
I honestly don't know for sure but I don't think this can be done in a query.
 
Some Java colleages of mine have advised that if you are going to reuse the values stored within a recordset several times, you could realize a performance increase by extracting values from the recordset and populating them into an array (or structure) before setting the recordset equal to nothing

This would theoretically improve performance because only the relevent recordset properties would be transferred to the array (or structure), less memory would be consumed once the array (or structure) had been created and the recordset reset to nothing), and you could eliminate all of the recordset lock handling from background processes.

I don't think this applies to you because you are only using the recordset once, but thought I would share the concept so you could consider it for future projects.

=======
To echo gemma-the-husky, I personally use UPDATE queries when attempting to do the operation your code describes


=======
Though Scooter has a point, I have chosen to store some calculated values to improve report performance times. They key here is to balance the increased storage needs against the run time calculation performance hits. I also limit precalculated values to elements unlikely to change such as mapping a person's age to an age cohort (e.g. 0, 1-4,...65+). Calling an unchanging, commonly used, precalculated value makes more sense to me than running the same calculation continuously.
________
W transmission
 
Last edited:
=======
Though Scooter has a point, I have chosen to store some calculated values to improve report performance times. They key here is to balance the increased storage needs against the run time calculation performance hits. I also limit precalculated values to elements unlikely to change such as mapping a person's age to an age cohort (e.g. 0, 1-4,...65+). Calling an unchanging, commonly used, precalculated value makes more sense to me than running the same calculation continuously.

Oh, I agree, (I store calculations in a few tables as well for performance hits). Its just generally easier to tell ppl to not store them as a general rule :)

I have a table containing average cases sold for each day of the week (Sunday to Saturday). What I'm doing is calculating the estimated cases to be sold from today until the next delivery of product([Days Until Delivery]) . So if today is Wednesday, I want it to add the average cases sold on Wednesday, the average cases sold on Thursday, and so on, until it reaches the date of my next delivery.
I honestly don't know for sure but I don't think this can be done in a query.

Well, off the top of my it shouldn't be that hard. Let me see if I got this straight:

You have a table that has the average cases sold by day (Is this static, or does it change with each passing day?). First, you want to calculate the difference between today and the next delivery. This can be accomplished by using the DateDiff function. You then want to figure out the sum of the average of the days between today and the next delivery day. Am I close?
 
Cullihall,

This is more of a business requirements question, but have you clarified whether by "average" you need a mathematical "mean", "median", or "mode"?

Different kinds of averages will give you different values and have different responses to outlier events.

As Scooterbug recommended, I would highly recommend you become familliar with the DateDiff method as it is incredibly valuable when averaging values by day of week. I've used it in the past to calculate average ship times and adjust raw calculations by excluding weekends.


=====
Scooterbug,

As they say "great minds think alike" I'm glad I've made it up to your level. *GRIN* Let's hope it's not a fluke.
________
Java
 
Last edited:
To begin, I'd like to say that I appreciate all your time.

Let's see if I can clarify what I'm trying to accomplish. The code I originally posted does work properly but it's really slow. Because I'm relatively new at vb , I just thought it might be a good idea to see if someone could tell me if my code could be changed to improve performance.

I have two tables (t_POCheckAllItemsCalculations and t_CombinedVelocityTotals). The t_CombinedVelocityTotals table has all my products with their average units sold for each day of the week. The t_POCheckAllItemsCalculations table has all the records which require updating, the delivery date and the days from now until that delivery date. I have to calculate the estimated cases that will sell from now until the next delivery based on my per day averages in the t_CombinedVelocityTotals table and put that result in the t_POCheckAllItemsCalculations table for each record.

Example: Product X sells 5 on Sunday, 10 on Monday, 7 on Tuesday, 3 on Wednesday, 4 on Thursday, 12 on Friday, and 8 on Saturday. For each record in t_POCheckAllItemsCalculations: If today is Wednesday and the days until delivery is 4, then I add the Wednesday number, the Thursday number, the Friday number, and the Saturday number and update the record in the t_POCheckAllItemsCalculations table.

The speed is slow probably because I take the first record that needs updating and Lookup that product to get the average units sold for each day(one day at a time) that falls within the range of today and the delivery date.

Hope this makes sense.
 
it makes sense..but you are right, using dlookups is more than likely what is slowing it down. Dlookups should be used sparingly...what you are trying to accomplish can be done with a bit of code and a well defined query.

This is what is floating around in my head. First, Calculate the number of days between today and the next delivery day. You can use the DateDiff() function for that. Then, format the current date to return the weekday number (Format(date(),"w"). Then, format the delivery date. Now, you have the range of days that you need to calculate the sum. Have a field in your t_CombinedVelocityTotals that store the numeric value of the day. You can now create a query and put the critiera of:
Between [DayNumericValue] and [DeliveryDayOfValue].

With that query, you can turn on Totals (The Sigma button on the query builder). Set the DayValueField Total line to WHERE, add your product, estimated Quantity. Set the total for the EstimatedQuantity to Sum, and it should give you a list of all your products added up for that time frame.
 
That sounds really good Scooterbug but would this work if I have 20,000 products in the t_CombinedVelocityTotals table, all with different averages for each day? All the products could have different delivery dates as well.
 
with different averages, yes. It should work with different delivery dates as well. I gotta head out....but if nobody jumps in, I'll see what I can come up with tomorrow.

Could you post a sample db with a few rows of data from the tables in question? Would help to see how the data is setup. Also, not everybody runs A2007, so if you do, try saving it as a MDB. You will get more responses.
 
Here's a sample. It's a big database so I've removed many tables and forms that don't apply here.
Click the 'Suggested Inbound Items' button to open another form. Click the 'Build Table' button to run the code I originally posted in this thread. All of the tables and queries in question are in the 'POCheck' group in the database window.
 

Attachments

Couple of questions..how do you derive the Average for each day? Do you run some kind of query each day to calculate the total? I ask because for what i'm thinking, the table setup needs to be changed a bit. Queries can be built to find the averages, and should be used in this scenario.
 
Couple of questions..how do you derive the Average for each day? Do you run some kind of query each day to calculate the total? I ask because for what i'm thinking, the table setup needs to be changed a bit. Queries can be built to find the averages, and should be used in this scenario.

The average per day is calculated in another database. I just import the table into this database.
 
Alright, check this out.

I had to create a new table for the Averages. Basically, I created a table where there are 4 fields fields. Primary Key, ProductID, AvgAmount, DayOfWeek. I simply did an update query for each day to dump the data from the orgininal table into the new table.

I then created a function that takes in the productID and deliverydate. (Check to Module for exact code).

Using the data fed into it, it calculates which days you need to add up to get the total. It then runs a query using that data to return the sum of the averages and passes that number back. Check query1 to see the results. You might want to double check my math to make sure it's valid.

I dont know what kind of time it will take to run through a large number of records, but give it a shot and compare.
 

Attachments

Alright, check this out.

I had to create a new table for the Averages. Basically, I created a table where there are 4 fields fields. Primary Key, ProductID, AvgAmount, DayOfWeek. I simply did an update query for each day to dump the data from the orgininal table into the new table.

I then created a function that takes in the productID and deliverydate. (Check to Module for exact code).

Using the data fed into it, it calculates which days you need to add up to get the total. It then runs a query using that data to return the sum of the averages and passes that number back. Check query1 to see the results. You might want to double check my math to make sure it's valid.

I dont know what kind of time it will take to run through a large number of records, but give it a shot and compare.

That's awesome Scooterbug!. Thank you very much. There's a considerable increase in performance. I really appreciate the time and effort you have put into this. That says a lot about the kind of people on this forum.
 
Glad it worked out for ya and happy to help out :)

Oh, forgot to mention, the function only works if the deliveryDate is a week or less. If you need to calculate for a greater time frame, you are going to have to add code to test to see if the DateDiff() is greater than 7 and adjust the sql statement.
 
Last edited:

Users who are viewing this thread

Back
Top Bottom