Hello,
I have a performance, probably related to my query and calculation structure and would be seeking your advice.
The DB is basically an airline schedule, with to make things simple, an entry for every flight. (tblSchedule). This table has information of origin and destination, airline, flight times, and aircraft type (to make things simple). The table relates to several other tables (tblAirport, tblAirline, tblAircrafttype), for the obvious information reltated to those. Other than that, there is a tblAircraftConfig (which has information, related to Airline and Aircrafttype - among other things the number of seats for that aircraft type and airline).
Now I need to do several evaluations for the (very large) tblSchedule. Those evaluations are - in principle - straightforward, but due to the sheer mass of data and the way I probably do things, give me a sheer headache, in terms of runtime.
The calculations, actually need to be done on every entry in the schedule table: Here is the data that is calculated:
- Flighttime : (by simply substracting departure and arrival times, all part of tblSchedule, but need a lookup of time difference, between arrival and departure airport).
- Number of seats : actually a lookup in the tblConfiguration, based on airline and aircrafttype, part of tblSchedule), but since a weighted average of found entries has to be calculated, I do this via a function call
- Available seat miles (most tricky one): a multiplication of the seats (see above) with the flight distance. This flight distance is calculated using a formula (with all kinds of sin and cos and atn functions), grouped in a function call, which needs to work on a set of geographic longitude and latitude data of the arrival and departure airport. This data is included within tblAirport.
At the end of the day, I don't need these indicators per flight (single tblSchedule record), but summed across flights, e.g; per a/c type and airline, per city pair, per airline and departure etc.
As a said, some rather complicated formulae for a database application, but nothing to win the noble prize with. However, the number of calculations to be carried out just kills the run time.
- First shot: create a "full query" with a defined query, include all n:1 relationships into the query, and for those, n:m relationships, make a davg lookup. Calculate the indicators for every flight, and sum up in a report. Result: awful runtime.
- Second shot: create a grouped query, summing things within the query. Problem is the lookup, which I need to make into the n:m relationship of the a/c config. I worked a way around this by creating a formula, which does this lookup for me with either the dlookup formula or a recordset and then run through the records. Anyway, runtime is disgusting.
- Third shot: create a formula call for the entire data, which does the same thing that the query does, with running through the data in a recordset (which is handed over to the formula). The formula, just hands back the readily calculated number. Runtime: let's not talk about it.
So this is a call for help. Doesn't seem like a complicated problem, but that's the difference between theory and reality.
The main issue - I believe is the number of lookups and 'complicated' calculations - e.g. calculating the distance between two airports. The thing is, the number of calculations is also exaggerated by the fact, that it repeats the calculation for every single flight. Now there are a lot fewer airport pairs than flights, of course. So I tried:
- Fourth shot: Make a split query. First of all, a SELECT First query, which generates a list of airport pairs, and calculates the airport distance only for the pair, once and for all in the query. Then base the second query on this first query selecting the rest of the schedule data (the flight time, the seats etc.). In theory, a lot fewer calculations need to be made, sinc eflight distance is calculated only for 250 airport pairs, instead of 25,000 flight entries. But: result - pathetic.
HELP!
Thanks
Jan
I have a performance, probably related to my query and calculation structure and would be seeking your advice.
The DB is basically an airline schedule, with to make things simple, an entry for every flight. (tblSchedule). This table has information of origin and destination, airline, flight times, and aircraft type (to make things simple). The table relates to several other tables (tblAirport, tblAirline, tblAircrafttype), for the obvious information reltated to those. Other than that, there is a tblAircraftConfig (which has information, related to Airline and Aircrafttype - among other things the number of seats for that aircraft type and airline).
Now I need to do several evaluations for the (very large) tblSchedule. Those evaluations are - in principle - straightforward, but due to the sheer mass of data and the way I probably do things, give me a sheer headache, in terms of runtime.
The calculations, actually need to be done on every entry in the schedule table: Here is the data that is calculated:
- Flighttime : (by simply substracting departure and arrival times, all part of tblSchedule, but need a lookup of time difference, between arrival and departure airport).
- Number of seats : actually a lookup in the tblConfiguration, based on airline and aircrafttype, part of tblSchedule), but since a weighted average of found entries has to be calculated, I do this via a function call
- Available seat miles (most tricky one): a multiplication of the seats (see above) with the flight distance. This flight distance is calculated using a formula (with all kinds of sin and cos and atn functions), grouped in a function call, which needs to work on a set of geographic longitude and latitude data of the arrival and departure airport. This data is included within tblAirport.
At the end of the day, I don't need these indicators per flight (single tblSchedule record), but summed across flights, e.g; per a/c type and airline, per city pair, per airline and departure etc.
As a said, some rather complicated formulae for a database application, but nothing to win the noble prize with. However, the number of calculations to be carried out just kills the run time.
- First shot: create a "full query" with a defined query, include all n:1 relationships into the query, and for those, n:m relationships, make a davg lookup. Calculate the indicators for every flight, and sum up in a report. Result: awful runtime.
- Second shot: create a grouped query, summing things within the query. Problem is the lookup, which I need to make into the n:m relationship of the a/c config. I worked a way around this by creating a formula, which does this lookup for me with either the dlookup formula or a recordset and then run through the records. Anyway, runtime is disgusting.
- Third shot: create a formula call for the entire data, which does the same thing that the query does, with running through the data in a recordset (which is handed over to the formula). The formula, just hands back the readily calculated number. Runtime: let's not talk about it.
So this is a call for help. Doesn't seem like a complicated problem, but that's the difference between theory and reality.
The main issue - I believe is the number of lookups and 'complicated' calculations - e.g. calculating the distance between two airports. The thing is, the number of calculations is also exaggerated by the fact, that it repeats the calculation for every single flight. Now there are a lot fewer airport pairs than flights, of course. So I tried:
- Fourth shot: Make a split query. First of all, a SELECT First query, which generates a list of airport pairs, and calculates the airport distance only for the pair, once and for all in the query. Then base the second query on this first query selecting the rest of the schedule data (the flight time, the seats etc.). In theory, a lot fewer calculations need to be made, sinc eflight distance is calculated only for 250 airport pairs, instead of 25,000 flight entries. But: result - pathetic.
HELP!
Thanks
Jan
