NTDLL.DLL / OLEAUT32.DLL crashes with O365 64bit MS Access application

[YossiDukes]

I'm running an access application in RDP and CITRIX environments.
It inconsistently crashes on a specific form after performing the Form_Load code while adding a new record and returning to internal access code. The form is bound to a SQL db view. Permissions are ok since it works intermittently. I have ensured the form is not corrupted (saveastext/loadtext).
This problem has been dogging us for the last 3 months. It seemed that a /decompile, compact and repair would resolve the problem, but for some users, it inconsistently keeps coming back.

I did a verification of windows install (sfc /VERIFYONLY) and no errors were reported.

Any suggestions to route this problem out would be greatly appreciated

Yossi

RDP ENVIRONMENT
Faulting application name: MSACCESS.EXE, version: 16.0.13127.21348, time stamp: 0x604411ff
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d
Exception code: 0xc0000374
Fault offset: 0x00000000000f70d3
Faulting process id: 0x598
Faulting application start time: 0x01d76d0b18b3c615
Faulting application path: C:\Program Files\Microsoft Office\root\Office16\MSACCESS.EXE
Faulting module path: C:\windows\SYSTEM32\ntdll.dll
Report Id: e5222485-5b43-4d1f-a3d7-49d941f2171e
Faulting package full name:
Faulting package-relative application ID:

Faulting application name: MSACCESS.EXE, version: 16.0.13127.21348, time stamp: 0x604411ff
Faulting module name: OLEAUT32.dll, version: 10.0.14393.4402, time stamp: 0x6087963c
Exception code: 0xc0000005
Fault offset: 0x00000000000896ab
Faulting process id: 0x2408
Faulting application start time: 0x01d76c55560cf8a4
Faulting application path: C:\Program Files\Microsoft Office\root\Office16\MSACCESS.EXE
Faulting module path: C:\windows\System32\OLEAUT32.dll
Report Id: 58e092b2-d84e-11eb-aaa7-000d3a849c18
Faulting package full name:
Faulting package-relative application ID:

DESKTOP ENVIRONMENT
Faulting application name: MSACCESS.EXE, version: 16.0.13127.21668, time stamp: 0x60bb1689
Faulting module name: ntdll.dll, version: 10.0.18362.1621, time stamp: 0x9c439dc3
Exception code: 0xc0000028
Fault offset: 0x00000000000fbe28
Faulting process id: 0x8f4
Faulting application start time: 0x01d76dbd95cc2d7d
Faulting application path: C:\Program Files\Microsoft Office\Root\Office16\MSACCESS.EXE
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 690a8526-a9af-458f-9363-d55ce52a1111
Faulting package full name:
Faulting package-relative application ID:

CITRIX ENVIRONMENT
Faulting application name: MSACCESS.EXE, version: 16.0.13628.20528, time stamp: 0x603ffc45
Faulting module name: OLEAUT32.dll, version: 10.0.14393.4402, time stamp: 0x6087963c
Exception code: 0xc0000005
Fault offset: 0x00000000000896a8
Faulting process id: 0xea8
Faulting application start time: 0x01d76c5840aa93a0
Faulting application path: C:\Program Files\Microsoft Office\Root\Office16\MSACCESS.EXE
Faulting module path: C:\windows\System32\OLEAUT32.dll
Report Id: 7fada191-d84c-11eb-a874-000d3a84f4bc
Faulting package full name:
Faulting package-relative application ID:

 

[The_Doc_Man]

OK, I'm going to look at the reports you have made one at a time.

RDP(1st) - exception code 0xC0000374 is a HEAP CORRUPTION, faulting app NTDLL.DLL
RDP(2nd) - exception code 0xC0000005 is a memory fault, faulting app OLEAUT32.DLL
Desktop - exception code 0xC0000028 is a stack fault, faulting app NTDLL.DLL
CITRIX - exception code 0xC0000005 is a memory fault, faulting app OLEAUT32.DLL

The exception codes have in common that something is abusing memory. They are all faults centered around the way that memory is being accessed/referenced. The ...005 cases are MOST likely to be either a reference to address 0 (which is illegal in any standard Windows process) or an attempt to write in a segment of memory that has been hardware write-protected. (Can't tell which.) The other two cases are HEAP and STACK faults which usually are related to a bad address. (If you are not familiar, the HEAP is like a scratch-pad/work area and the STACK is how Windows keeps track of the subroutines that called other subroutines in a last-in, first-out order.)

NTDLL.DLL and OLEAUT32.DLL are known components that are sensitive to abuse of memory. It's a toss-up when I search the web as to which of them reports more errors of this type.

The first way to try to fix this is to attempt to update Windows. NTDLL.DLL is, essentially, a Windows NT DLL kernel, i.e. a module that helps your process interface to other .DLL files. Absolutely can't do without it. OLEAUT32.DLL is the Ace database OLE kernel, a way to interface with OLE situations. This one is used very often as well. Be sure that you do not need to patch either of these two modules.

The second possible issue is that something you are calling as an add-on or third-party software needs updating. If you have any non-MS software on your system, see if newer versions/updates are available.

You don't say what the DB does, so I'll ask. Does this DB have anything to do with web pages? If so, browser add-ons are at the top of the suspect list.

Now the payoff: What is really happening? Well, nobody knows because despite these two .DLL files being called out a lot, the problem is that we need another reporting layer. You see, ALL of those errors say more or less the same thing: Access crashed because a go-between module was given a bad value as input. But it doesn't say from where that bad value originated because we only see TWO layers - the host task and the module containing the faulting address. We don't see what was passed because that would take a 3rd layer.

ALL of those errors can be easily described as "abuse of memory by passing bad 'ByRef' arguments." Just passing the bad args to different routines depending on what was actually done.

 

[YossiDukes]

Firstly, thank you so much for your detailed explanation of these internals. This is truly an education

Patching - first - are these dlls out of date? Based on Win10 installation, what are the latest versions of these DLLs. Regardless, this would have to be done by IT, as it is not in my power to do anything of that level in either the RDP or CITRIX environments.

DB - nothing to do with web pages. just the repository for the data. The form in question is bound to a view.

Add-ons

• we are using a 3rd party control in the form (EXONTROL's ExGRID control v17.0.0.2). Although the control is on the form, we have commented out any code referencing it and the error still occurs.
• We are also using the latest vbWatchdog add-in (v3.8.3.0), an more extensive Error handler than VB's which allows us to dump memory and call stack details when an error occurs

My concern is that the problem only manifests itself on this one form (out of over 600), yet inconsistently. FYI - If I delay the last mouse click in a series of clicks by 10-15 seconds, Access does not crash. But if I don't delay while clicking the series of mouse clicks to get to the form and click the [Add] button, Access crashes.

Is there some tool that can be running that would log the calls to these DLLs which could determine where things are going off the rails?

That being said, taking your comment on "bad ByRef arguments" as a lead, I will take a closer look at the calls made in this form to see if there's something nasty.

Any other hints or thoughts on where to take this is greatly appreciated.

Yossi

 

[The_Doc_Man]

The problem is hard to trace because Windows doesn't QUITE give us the information needed for this. BUT if it is a single form, you must ask yourself what is ON that form. You point out a 3rd party control being on the form. It is not enough to comment out referencing code. If you are no longer using the code, take it out of your VBA Window: Tools>>References list. Close the entire DB and then open it again.

As to "latest versions:...

Faulting application name: MSACCESS.EXE, version: 16.0.13127.21668, time stamp: 0x60bb1689
Faulting module name: OLEAUT32.dll, version: 10.0.14393.4402, time stamp: 0x6087963c
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d

But when I ask Google what is the latest version of software, it tells me:

MSACCESS.EXE - 16.0.14026.20270
OLEAUT32.DLL - 10.0.16299.1087
NTDLL.DLL - 10.0.19041.423

If your IT guys want to do a Windows Update then all three of those should be updated automagically. NOTA BENE: Updating everything on the Windows side of the problem still doesn't guarantee a fix - but as often as this problem gets reported to Micro$oft, they will have patched everything they could find. So if the problem persists, consider that it is in your 3rd party software.

Also, if you have a warranty service on your 3rd party stuff, open a ticket to ask them if they have received reports of crashes like this with their software. Unless they are sleazeballs, they will tell you what they know about it. We might have trouble searching this on the web if they happen to not put their trouble reports in a web-reachable site. If so, we would never be able to give you the help you need. But you did ask where to go to find out more. That's the best place.

So the short-winded version of that?
1. Get your IT guys to run an update cycle
2. Call your 3rd-party s/w vendors.

 

[The_Doc_Man]

One more thought: When I said "Bad ByRef arguments" I wasn't specifically referring to your code. Based on the bugs noted online, this might be bad calls in your 3rd party software. It won't hurt to check your ByRef arguments, but the compiler should stop any BLATANTLY bad calls.

 

[YossiDukes]

The problem is hard to trace because Windows doesn't QUITE give us the information needed for this. BUT if it is a single form, you must ask yourself what is ON that form. You point out a 3rd party control being on the form. It is not enough to comment out referencing code. If you are no longer using the code, take it out of your VBA Window: Tools>>References list. Close the entire DB and then open it again.

As to "latest versions:...

Faulting application name: MSACCESS.EXE, version: 16.0.13127.21668, time stamp: 0x60bb1689
Faulting module name: OLEAUT32.dll, version: 10.0.14393.4402, time stamp: 0x6087963c
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d

But when I ask Google what is the latest version of software, it tells me:

MSACCESS.EXE - 16.0.14026.20270
OLEAUT32.DLL - 10.0.16299.1087
NTDLL.DLL - 10.0.19041.423

If your IT guys want to do a Windows Update then all three of those should be updated automagically. NOTA BENE: Updating everything on the Windows side of the problem still doesn't guarantee a fix - but as often as this problem gets reported to Micro$oft, they will have patched everything they could find. So if the problem persists, consider that it is in your 3rd party software.

Also, if you have a warranty service on your 3rd party stuff, open a ticket to ask them if they have received reports of crashes like this with their software. Unless they are sleazeballs, they will tell you what they know about it. We might have trouble searching this on the web if they happen to not put their trouble reports in a web-reachable site. If so, we would never be able to give you the help you need. But you did ask where to go to find out more. That's the best place.

So the short-winded version of that?
1. Get your IT guys to run an update cycle
2. Call your 3rd-party s/w vendors.

Hi Doc,

Thank you again for your insight.

Regarding the 3rd party control: This grid control is used in numerous places across our application. All startup, initialization and population of the control is consistent across all the other forms and yet the problem is only manifesting itself on this one form. To eliminate it as a potential cause of the problem, I have removed the control along with all references to the control from this form and am still able to cause MSACCESS to crash referencing ntdll.dll in our CITRIX environment:

Faulting application name: MSACCESS.EXE, version: 16.0.13628.20528, time stamp: 0x603ffc45
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d
Exception code: 0xc0000028
Fault offset: 0x00000000000a5640
Faulting process id: 0x28f4
Faulting application start time: 0x01d771a0cc9a6726
Faulting application path: C:\Program Files\Microsoft Office\Root\Office16\MSACCESS.EXE
Faulting module path: C:\windows\SYSTEM32\ntdll.dll
Report Id: 2379c91b-dd94-11eb-a876-000d3ae8dfea
Faulting package full name:
Faulting package-relative application ID:

Just as an observation, the only other unique characteristic of this troublesome form that I can quickly identify is that it is using a common sub-form three times with different filtering criteria.

I will look into getting IT to update the CITRIX environment DLLs and report back any further findings .

 

[The_Doc_Man]

Might be a pain in the toches but if this 3rd-party control works for every other form where it appears, consider dissolving and recreating the one form that doesn't work. You also mentioned that there is a time factor in this operation. "Races" are not unknown in computer circles. I'll have to go into some weeds for this though.

Basically, anything you do that is "pure" Access cannot have a race because Access is written as a single-threaded program. Everything inside of Access is synchronous. The problem comes when you have external elements that potentially involve Object Linking and Embedding (OLE) because that means that Access potentially needs to wait for an external program to load and execute. The OLEAUT32.DLL file is the OLE Access Utility for 32-bit Access. NTDLL.DLL is also an interface - of a more general nature - for DLL support files to access the NT kernel.

OK, you aren't on NT... but actually you and everyone else on Win10 ARE using the NT kernel, just a modified version of it. That kernel change occurred with the introduction of Win NT, which was the first TRUE rewrite of Windows since the advent of Windows as a layer over MS-DOS. But that's how long the particular architecture has been around.

Back to your problem... this is the most difficult kind of debugging there is. IF rebuilding a new form from scratch doesn't help, the only other thing to do is to populate your form's event code with something to event-log "waypoints." Basically, create a table with two fields: a SINGLE and an INTEGER or LONG field. (I always use LONG, but that's me...) Then at the start of the run, erase the table. Call it WAYPT, call the fields TS and EVTNUM.

Either in a public module or in the class module of the offending form:

Public Sub LogWayPnt( WptNum as LONG )
Dim WRS as DAO.Recordset

    Set WRS as CurrentDB.OpenRecordset( "WAYPT", dbOpenDynaset )
    With WRS
        .Append
        !EvtNum = WptNum
        !TS = Timer()
        .Update
        .Close
    End With

End Sub

Then in the event code on that form you call LogWayPnt with an argument that is a different number at each different point where you call it. Each call point then becomes a waypoint that you have passed in your code. It's like a mile marker at the side of the highway. Important: Each call uses a constant number as the "actual argument" but each call that is a different location uses a different constant. It is perfectly normal to see some waypoints more often than others. Don't use a call inside a short loop unless that loop calls a complex subroutine at each iteration.

The time stamp will be floating-point seconds (and fractions thereof) since midnight of the current date. The timer is accurate to about 1/60th of a second. This time never exceeds 87600, which is the number of seconds in a day, so SINGLE is sufficient precision.

In this case it is imperative to close the recordset with each call because if you crash it might be asynchronous. You want to do your best to "leave a trail of bread crumbs." If your crash consistently stops just after a particular waypoint, you have a hint as to where to focus next. For long, complex VBA code, it is perfectly fine to have more than one waypoint (each with different numbers, of course...) so that you can track progress. And if you are trying to figure out WHERE in a routine you have the problem, it is OK (for testing purposes) to add extra waypoints, again with unique numbers so that you can find them again.

Note that I'm suggesting the TIMER() function rather than NOW() because NOW truncates time to the second where as TIMER has better time resolution. And to be honest, you don't care about the date. It would also be OK if you REALLY wanted to fancy to put a TEXT string rather than a waypoint number. I just use the numbers to minimize the size of the table and also to minimize the work done for each waypoint.

If after several tries this offers no insight, then I don't know where to go from there. Take out the waypoint code in that case. You won't need it for this problem.

 

[YossiDukes]

Hi Doc,

Thank you again for your insight.

Regarding the 3rd party control: This grid control is used in numerous places across our application. All startup, initialization and population of the control is consistent across all the other forms and yet the problem is only manifesting itself on this one form. To eliminate it as a potential cause of the problem, I have removed the control along with all references to the control from this form and am still able to cause MSACCESS to crash referencing ntdll.dll in our CITRIX environment:

Faulting application name: MSACCESS.EXE, version: 16.0.13628.20528, time stamp: 0x603ffc45
Faulting module name: ntdll.dll, version: 10.0.14393.4350, time stamp: 0x606eaf8d
Exception code: 0xc0000028
Fault offset: 0x00000000000a5640
Faulting process id: 0x28f4
Faulting application start time: 0x01d771a0cc9a6726
Faulting application path: C:\Program Files\Microsoft Office\Root\Office16\MSACCESS.EXE
Faulting module path: C:\windows\SYSTEM32\ntdll.dll
Report Id: 2379c91b-dd94-11eb-a876-000d3ae8dfea
Faulting package full name:
Faulting package-relative application ID:

Just as an observation, the only other unique characteristic of this troublesome form that I can quickly identify is that it is using a common sub-form three times with different filtering criteria.

I will look into getting IT to update the CITRIX environment DLLs and report back any further findings .

Just a further note/observation - it seems that when this specific form is opened on an existing record, it opens without incident.
Form_Load code checks if the form is to be opened in Add mode and if so, it uses the DoCmd.GoToRecord , , acNewRec

 

[YossiDukes]

The journey continues...

After removing the grid and still encountering crashes, this eliminates the GRID as a potential culprit.

I focused on the Form_Load's docmd.gotorecord acnew when the form is open to insert a new record. When it was commented out no more crashes.

Traced code after the gotorecord and saw it went into the subform's OnCurrent code 6 times (even though there are only 3 references to this subform on this form). Once I removed the master/child relationships to the three subforms, no more crashes.

I checked the data in the subform's referenced table and noted there are 2 records with wonky years (Year 209, Year 107). I corrected the years, put the relationships back, removed the SourceObject reference on these subforms and changed the Form_Load code to assign the SourceObject when the form gets loaded. No more crashes in DEV (so far).

 

[The_Doc_Man]

Excellent detective work. Sounds like you have a handle on it.

 

[YossiDukes]

The journey continues...

After removing the grid and still encountering crashes, this eliminates the GRID as a potential culprit.

I focused on the Form_Load's docmd.gotorecord acnew when the form is open to insert a new record. When it was commented out no more crashes.

Traced code after the gotorecord and saw it went into the subform's OnCurrent code 6 times (even though there are only 3 references to this subform on this form). Once I removed the master/child relationships to the three subforms, no more crashes.

I checked the data in the subform's referenced table and noted there are 2 records with wonky years (Year 209, Year 107). I corrected the years, put the relationships back, removed the SourceObject reference on these subforms and changed the Form_Load code to assign the SourceObject when the form gets loaded. No more crashes in DEV (so far).

New Day - same problem unfortunately

 

[CJ_London]

Subforms open before main forms and will go through the current event at least twice . Once when it opens and again when the main form opens. If there is another subform, it will go through a third time (as will the other subform. And again for each subsequent subform

 

[YossiDukes]

So chasing this down further, I have determined the problem occurs because of:
a. Form has a Tab Control
b. Tab Control has multiple pages
c. 3 of the pages reference the same sub-form with different filtering criteria (there are reported issues with Sub-Forms on a Tab Control)
d. If form is passed openarg to add a new record, Form_Load uses a docmd.gotorecord ,,acnew
e. Form_Load calls function to configure a combo control for each sub-form
f. Access/NTDLL.DLL intermittently crashes after returning control to user when calls from step (e) are run. Commenting out the 3 calls prevents crash

As a workaround:
a. If form is passed openarg to add a new record, I do not call the configuration function from step (e)
b. If a new record is saved, I run the configuration function from step (e) to ensure the combo controls in each subform are populated.

This solution
a. avoids the crashing scenario when loading the form and setting up the combo controls in the subforms
b. ensures the combo controls are initialized after saving a new record

Testing this so far looks promising. Will report back if this solves the crashing issue

 

[The_Doc_Man]

Here's a crazy thought... make two COPIES of the sub-form with different names (and thus independent class modules and controls). Use a different copy on each tab. IF that works, your problem was PROBABLY an incompletely qualified object reference.