FSO VBA Question (1 Viewer)

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
Hi All,

I have a module in my Access Database that looks for a text file in a folder, renames it (due to the weird extension the original file has .QAN) as a .TXT file, grabs the data and deletes the file. This is called on a timer, and usually works well. However, if the module falls over, the files build up in the folder and we have to remove the files manually and move them back one by one. Can be hundreds of files if the system is running overnight.

This is the code we use
Code:
Dim fs As New Scripting.FileSystemObject
Dim f As File
Dim fldr As Folder

Set fldr = fs.GetFolder("C:\Results")
If fldr.Files.Count > 0 Then
  For Each f In fldr.Files
          Name f As "C:\Results\SuperQ.txt"
      Next f
       
End If
   Set f = Nothing

Does anyone have an idea how to just rename the first file in the folder, or any random file in the folder, to "SuperQ.txt", run the data grab, then rename the next file etc? I was think of a loop, but, I'm not sure how to go about this.

Or any other suggestions on how to do this if there are multiple files in the source folder?

Thanks a lot in advance for any help you can give.
 
Last edited:

arnelgp

..forever waiting... waiting for jellybean!
Local time
Tomorrow, 00:02
Joined
May 7, 2009
Messages
19,233
you only need to rename those with .QAN extension?
Code:
Dim fs As New Scripting.FileSystemObject
 Dim f As File
 Dim fldr As Folder

 Set fldr = fs.GetFolder("C:\Results")
If fldr.Files.Count > 0 Then
  For Each f In fldr.Files
        If Right$(f, 4) = ".QAN" Then
          Name f As Replace$(f, ".QAN", ".TXT")
        End If
      Next f
        
 End If
   Set f = Nothing
End Sub
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
Thanks for the reply arnelgp. Unfortunately, all the files going into this folder will have the .QAN extension
 

arnelgp

..forever waiting... waiting for jellybean!
Local time
Tomorrow, 00:02
Joined
May 7, 2009
Messages
19,233
sorry, the first file (random .QAN file) will be renamed SuperQ.txt.
how about the succeeding files what it will be renamed?
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
At the moment, if there is more than one file the code falls over, (can't have more than 1 file called super.txt). The files are generated by an analytical instrument every 90 seconds or so. If all is working well, the generated .QAN file is renamed, the data is taken out of the file and the super.txt file is deleted. That works well most of the time, but, if there is an error (happens a few times a week) and the files build up in C:\Results and the data import doesn't work. We have to manually move the files to another folder, then move them back one by one to the C:\Results folder and the process restarts.

The main reason the file is renamed is that .QAN is not a recognised file type, but it is a specific type of text file.

Could we just rename all of the .QAN files .TXT files and then loop through the data grab? I'm not sure how to go about that. I can post the rest of the code if that helps?
 

arnelgp

..forever waiting... waiting for jellybean!
Local time
Tomorrow, 00:02
Joined
May 7, 2009
Messages
19,233
ok post more code, so i will give you the full code (not installment).
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
This is the full code and i have attached an example QAN file, had to rename it, wouldn't let me post a QAN file (the import part of the code works well, if there is only 1 file in the C:\Results folder):

Code:
Private Sub Form_Timer() 'set timer to x milliseconds accordingly



Dim fs As New Scripting.FileSystemObject
Dim f As File
Dim fldr As Folder

Set fldr = fs.GetFolder("C:\Results")
If fldr.Files.Count > 0 Then
  For Each f In fldr.Files
          Name f As "C:\Results\SuperQ.txt"
      Next f
     
End If
   Set f = Nothing
 


'This code made with MUCH help from Wayne Ryan at the Access World Forums.
'It opens a text file made by SuperQ software, containing individual analysis
'results and parses out the desired sample information to the table "new XRF Results"
'and the desired sample data to the table "XRF Results Concentration". The 2 tables
'are in a One-to-Many relationship, and the ResultID, an autonumber ID in "XRF Results",
'ties the related records together.


 
If fldr.Files.Count > 0 Then

Dim dbs As DAO.Database          'Pull up database
Dim rst As DAO.Recordset         'Pull up individual recordset for table
Dim ResultID As Long             'The AutoNumber record ID
Dim SampleName As String         'The name of the actual sample analyzed
Dim ResultDate As String        'Date/Time value of the converted string for date of analysis
Dim Time As String
Dim FinalWeight As String
Dim MeasureOriginName As String  'Analytical program used to analyze sample in SuperQ
Dim Fe As Double
Dim Init As String
Dim LOI As String
Dim Fe2O3 As Double                 'Result concentration, in %, from the analysis
Dim SiO2 As Double                 'Result concentration, in %, from the analysis
Dim CaO As Double                 'Result concentration, in %, from the analysis
Dim MnO As Double                 'Result concentration, in %, from the analysis
Dim Al2O3 As Double                 'Result concentration, in %, from the analysis
Dim TiO2 As Double                 'Result concentration, in %, from the analysis
Dim MgO As Double                 'Result concentration, in %, from the analysis
Dim P2O5 As Double                 'Result concentration, in %, from the analysis
Dim SO3 As Double                 'Result concentration, in %, from the analysis
Dim K2O As Double                 'Result concentration, in %, from the analysis
Dim V2O5 As Double                 'Result concentration, in %, from the analysis
Dim Cr2o3 As Double                 'Result concentration, in %, from the analysis
Dim CoO As Double                 'Result concentration, in %, from the analysis
Dim NiO As Double                 'Result concentration, in %, from the analysis
Dim CuO As Double                 'Result concentration, in %, from the analysis
Dim ZnO As Double                 'Result concentration, in %, from the analysis
Dim As2O3 As Double                 'Result concentration, in %, from the analysis
Dim PbO As Double                 'Result concentration, in %, from the analysis
Dim BaO As Double                'Result concentration, in %, from the analysis
Dim Na2O As Double
Dim Cl As Double
Dim SnO As Double
Dim SQL As String                'The SQL statement used to pull data into the table "XRF Results Concentration"



Dim buffer As String

Set dbs = CurrentDb
Set rst = dbs.OpenRecordset("tblXRFResults")   'Set database table
DoCmd.SetWarnings False              'Turn off warning so that an Access message box does not appear
                                             'for each record appended to the "tblXRFResultsConcentration" table


Open "C:\Results\SuperQ.txt" For Input As #1    'Open the text file made by SuperQ to import the data



Line Input #1, buffer
While Not EOF(1)
   Select Case Mid(buffer, 1, 2)
      Case "Sa"
       SampleName = RTrim(Mid(buffer, 24, 35))
   
     
      Case "Ap"
        MeasureOriginName = RTrim(Mid(buffer, 24, 20))
      Case "Me"
        ResultDate = RTrim(Mid(buffer, 24, 10))
        Time = RTrim(Mid(buffer, 34, 11))
      Case "In"
        Init = RTrim(Mid(buffer, 24, 10))
       Case "Fi"
        FinalWeight = RTrim(Mid(buffer, 24, 10))
      Case "LO"
        LOI = RTrim(Mid(buffer, 24, 10))
   
     
        Set rst = dbs.OpenRecordset("tblXRFResults")
        rst.AddNew
        rst!ID = SampleName
        rst!SampleName = SampleName
        rst!ResultDate = ResultDate
        rst!Time = Time
        rst!MeasureOriginName = MeasureOriginName
        rst.Update
        rst.Close
        Set rst = dbs.OpenRecordset("tblXRFResults")
        rst.MoveLast
        ResultID = rst!ResultID
        rst.Close
 
      Case "Si"
        SiO2 = RTrim(Mid(buffer, 17, 9))
      Case "Fe"
        Fe2O3 = RTrim(Mid(buffer, 17, 9))
      Case "Ca"
        CaO = RTrim(Mid(buffer, 17, 9))
      Case "Mn"
        MnO = RTrim(Mid(buffer, 17, 9))
      Case "Al"
        Al2O3 = RTrim(Mid(buffer, 17, 9))
      Case "Ti"
        TiO2 = RTrim(Mid(buffer, 17, 9))
      Case "Mg"
        MgO = RTrim(Mid(buffer, 17, 9))
      Case "P2"
        P2O5 = RTrim(Mid(buffer, 17, 9))
      Case "SO"
        SO3 = RTrim(Mid(buffer, 17, 9))
      Case "K2"
        K2O = RTrim(Mid(buffer, 17, 9))
      Case "V2"
        V2O5 = RTrim(Mid(buffer, 17, 9))
      Case "Cr"
        Cr2o3 = RTrim(Mid(buffer, 17, 9))
      Case "Co"
        CoO = RTrim(Mid(buffer, 17, 9))
      Case "Ni"
        NiO = RTrim(Mid(buffer, 17, 9))
      Case "Cu"
        CuO = RTrim(Mid(buffer, 17, 9))
      Case "Zn"
        ZnO = RTrim(Mid(buffer, 17, 9))
      Case "As"
        As2O3 = RTrim(Mid(buffer, 17, 9))
      Case "Pb"
        PbO = RTrim(Mid(buffer, 17, 9))
      Case "Ba"
        BaO = RTrim(Mid(buffer, 17, 9))
      Case "Na"
        Na2O = RTrim(Mid(buffer, 17, 9))
      Case "Cl"
        Cl = RTrim(Mid(buffer, 17, 9))
      Case "Sn"
        SnO = RTrim(Mid(buffer, 17, 9))
   
          SQL = "INSERT INTO [tblXRFResultsConc] (ResultID, Fe2O3, SiO2, CaO, MnO, Al2O3, TiO2, MgO, P2O5, SO3, K2O, V2O5, Cr2O3, CoO, NiO, CuO, ZnO, As2O3, PbO, BaO, Na2O, Cl, SnO)" & _
                     "Values(" & ResultID & ", " & _
                                 Fe2O3 & ", " & _
                                 SiO2 & ", " & _
                                 CaO & ", " & _
                                 MnO & ", " & _
                                 Al2O3 & ", " & _
                                 TiO2 & ", " & _
                                 MgO & ", " & _
                                 P2O5 & ", " & _
                                 SO3 & ", " & _
                                 K2O & ", " & _
                                 V2O5 & ", " & _
                                 Cr2o3 & ", " & _
                                 CoO & ", " & _
                                 NiO & ", " & _
                                 CuO & ", " & _
                                 ZnO & ", " & _
                                 As2O3 & ", " & _
                                 PbO & ", " & _
                                 BaO & ", " & _
                                 Na2O & ", " & _
                                 Cl & ", " & _
                                 SnO & ");"
                             
     
        DoCmd.RunSQL SQL
     
      End Select
   Line Input #1, buffer

   Wend
Close #1                                        'Close the text file.


Dim fso, f1, S

  Set fso = CreateObject("Scripting.FileSystemObject")

  Set f1 = fso.GetFile("C:\Results\SuperQ.txt")

  f1.Delete

  End If

   Set fldr = Nothing
    Set fs = Nothing
    Set rst = Nothing

End Sub
 

Attachments

  • Fe Ore.txt
    1 KB · Views: 492
Last edited:

arnelgp

..forever waiting... waiting for jellybean!
Local time
Tomorrow, 00:02
Joined
May 7, 2009
Messages
19,233
make a backup of your db first.
then overwrite your timer with this one:
Code:
Private Sub Form_Timer() 'set timer to x milliseconds accordingly
Dim fs As New Scripting.FileSystemObject
Dim f As File
Dim fldr As Folder

'arnelgp
Dim lngTimer As Long, i As Long
Dim colFiles As New Collection
lngTimer = Me.TimerInterval
'kill the timer
Me.TimerInterval = 0
'end of arnelgp code

Set fldr = fs.GetFolder("C:\Results")
If fldr.Files.Count > 0 Then
    For Each f In fldr.Files
        'arnelgp
        'put all files to collection object
        'Name f As "C:\Results\SuperQ.txt"
        colFiles.Add f
    Next f
     
End If
   Set f = Nothing
 


'This code made with MUCH help from Wayne Ryan at the Access World Forums.
'It opens a text file made by SuperQ software, containing individual analysis
'results and parses out the desired sample information to the table "new XRF Results"
'and the desired sample data to the table "XRF Results Concentration". The 2 tables
'are in a One-to-Many relationship, and the ResultID, an autonumber ID in "XRF Results",
'ties the related records together.

Dim dbs As DAO.Database          'Pull up database
Dim rst As DAO.Recordset         'Pull up individual recordset for table
Dim ResultID As Long             'The AutoNumber record ID
Dim SampleName As String         'The name of the actual sample analyzed
Dim ResultDate As String        'Date/Time value of the converted string for date of analysis
Dim Time As String
Dim FinalWeight As String
Dim MeasureOriginName As String  'Analytical program used to analyze sample in SuperQ
Dim Fe As Double
Dim Init As String
Dim LOI As String
Dim Fe2O3 As Double                 'Result concentration, in %, from the analysis
Dim SiO2 As Double                 'Result concentration, in %, from the analysis
Dim CaO As Double                 'Result concentration, in %, from the analysis
Dim MnO As Double                 'Result concentration, in %, from the analysis
Dim Al2O3 As Double                 'Result concentration, in %, from the analysis
Dim TiO2 As Double                 'Result concentration, in %, from the analysis
Dim MgO As Double                 'Result concentration, in %, from the analysis
Dim P2O5 As Double                 'Result concentration, in %, from the analysis
Dim SO3 As Double                 'Result concentration, in %, from the analysis
Dim K2O As Double                 'Result concentration, in %, from the analysis
Dim V2O5 As Double                 'Result concentration, in %, from the analysis
Dim Cr2o3 As Double                 'Result concentration, in %, from the analysis
Dim CoO As Double                 'Result concentration, in %, from the analysis
Dim NiO As Double                 'Result concentration, in %, from the analysis
Dim CuO As Double                 'Result concentration, in %, from the analysis
Dim ZnO As Double                 'Result concentration, in %, from the analysis
Dim As2O3 As Double                 'Result concentration, in %, from the analysis
Dim PbO As Double                 'Result concentration, in %, from the analysis
Dim BaO As Double                'Result concentration, in %, from the analysis
Dim Na2O As Double
Dim Cl As Double
Dim SnO As Double
Dim SQL As String                'The SQL statement used to pull data into the table "XRF Results Concentration"



Dim buffer As String

Set dbs = CurrentDb
Set rst = dbs.OpenRecordset("tblXRFResults")   'Set database table

'arnelgp
For i = 1 To colFiles.Count
'If fldr.Files.Count > 0 Then
   
    DoCmd.SetWarnings False              'Turn off warning so that an Access message box does not appear
                                                 'for each record appended to the "tblXRFResultsConcentration" table
    'arnelgp
    'just makesure SuperQ.txt does not exists
    If Len(Dir("C:\Results\SuperQ.txt")) > 0 Then Kill "C:\Results\SuperQ.txt"
   
    'arnelgp
    'rename it here
    Name colFiles(i) As "C:\Results\SuperQ.txt"
   
    Open "C:\Results\SuperQ.txt" For Input As #1    'Open the text file made by SuperQ to import the data
   
   
   
    Line Input #1, buffer
    While Not EOF(1)
       Select Case Mid(buffer, 1, 2)
          Case "Sa"
           SampleName = RTrim(Mid(buffer, 24, 35))
       
         
          Case "Ap"
            MeasureOriginName = RTrim(Mid(buffer, 24, 20))
          Case "Me"
            ResultDate = RTrim(Mid(buffer, 24, 10))
            Time = RTrim(Mid(buffer, 34, 11))
          Case "In"
            Init = RTrim(Mid(buffer, 24, 10))
           Case "Fi"
            FinalWeight = RTrim(Mid(buffer, 24, 10))
          Case "LO"
            LOI = RTrim(Mid(buffer, 24, 10))
       
         
            Set rst = dbs.OpenRecordset("tblXRFResults")
            rst.AddNew
            rst!ID = SampleName
            rst!SampleName = SampleName
            rst!ResultDate = ResultDate
            rst!Time = Time
            rst!MeasureOriginName = MeasureOriginName
            rst.Update
            rst.Close
            Set rst = dbs.OpenRecordset("tblXRFResults")
            rst.MoveLast
            ResultID = rst!ResultID
            rst.Close
     
          Case "Si"
            SiO2 = RTrim(Mid(buffer, 17, 9))
          Case "Fe"
            Fe2O3 = RTrim(Mid(buffer, 17, 9))
          Case "Ca"
            CaO = RTrim(Mid(buffer, 17, 9))
          Case "Mn"
            MnO = RTrim(Mid(buffer, 17, 9))
          Case "Al"
            Al2O3 = RTrim(Mid(buffer, 17, 9))
          Case "Ti"
            TiO2 = RTrim(Mid(buffer, 17, 9))
          Case "Mg"
            MgO = RTrim(Mid(buffer, 17, 9))
          Case "P2"
            P2O5 = RTrim(Mid(buffer, 17, 9))
          Case "SO"
            SO3 = RTrim(Mid(buffer, 17, 9))
          Case "K2"
            K2O = RTrim(Mid(buffer, 17, 9))
          Case "V2"
            V2O5 = RTrim(Mid(buffer, 17, 9))
          Case "Cr"
            Cr2o3 = RTrim(Mid(buffer, 17, 9))
          Case "Co"
            CoO = RTrim(Mid(buffer, 17, 9))
          Case "Ni"
            NiO = RTrim(Mid(buffer, 17, 9))
          Case "Cu"
            CuO = RTrim(Mid(buffer, 17, 9))
          Case "Zn"
            ZnO = RTrim(Mid(buffer, 17, 9))
          Case "As"
            As2O3 = RTrim(Mid(buffer, 17, 9))
          Case "Pb"
            PbO = RTrim(Mid(buffer, 17, 9))
          Case "Ba"
            BaO = RTrim(Mid(buffer, 17, 9))
          Case "Na"
            Na2O = RTrim(Mid(buffer, 17, 9))
          Case "Cl"
            Cl = RTrim(Mid(buffer, 17, 9))
          Case "Sn"
            SnO = RTrim(Mid(buffer, 17, 9))
       
              SQL = "INSERT INTO [tblXRFResultsConc] (ResultID, Fe2O3, SiO2, CaO, MnO, Al2O3, TiO2, MgO, P2O5, SO3, K2O, V2O5, Cr2O3, CoO, NiO, CuO, ZnO, As2O3, PbO, BaO, Na2O, Cl, SnO)" & _
                         "Values(" & ResultID & ", " & _
                                     Fe2O3 & ", " & _
                                     SiO2 & ", " & _
                                     CaO & ", " & _
                                     MnO & ", " & _
                                     Al2O3 & ", " & _
                                     TiO2 & ", " & _
                                     MgO & ", " & _
                                     P2O5 & ", " & _
                                     SO3 & ", " & _
                                     K2O & ", " & _
                                     V2O5 & ", " & _
                                     Cr2o3 & ", " & _
                                     CoO & ", " & _
                                     NiO & ", " & _
                                     CuO & ", " & _
                                     ZnO & ", " & _
                                     As2O3 & ", " & _
                                     PbO & ", " & _
                                     BaO & ", " & _
                                     Na2O & ", " & _
                                     Cl & ", " & _
                                     SnO & ");"
                                 
         
            DoCmd.RunSQL SQL
         
          End Select
       Line Input #1, buffer
   
       Wend
    Close #1                                        'Close the text file.
   
    'arnelgp
    'this is redundant, there is already a fs object.
    '
    'Dim fso, f1, S
    '
    '  Set fso = CreateObject("Scripting.FileSystemObject")
    '
    '  Set f1 = fso.GetFile("C:\Results\SuperQ.txt")
    '
    '  f1.Delete
     
      'arnelgp
    If Len(Dir("C:\Results\SuperQ.txt")) > 0 Then Kill "C:\Results\SuperQ.txt"


  'End If
Next

   Set fldr = Nothing
    Set fs = Nothing
    rst.Close
    Set rst = Nothing
    Set dbs = Nothing
   
    'reinstate the timer
    Me.TimerInterval = lngTimer
End Sub
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
Thanks a lot, I will give it a go and report back.
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
Thanks a lot, it seems to work fine. Only issue is it is very slow, >5 minutes to import one file, but, this could be due to the fact that I am working remotely and my data is pretty slow.
 

arnelgp

..forever waiting... waiting for jellybean!
Local time
Tomorrow, 00:02
Joined
May 7, 2009
Messages
19,233
can you post a sample of the .QAN file.
maybe we can open it as Recordset.
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
I have attached an example file, but I had to change the extension, wouldn't let me upload as a .QAN file
 

Attachments

  • QB100833.txt
    1 KB · Views: 466

gemma-the-husky

Super Moderator
Staff member
Local time
Today, 17:02
Joined
Sep 12, 2006
Messages
15,640
Can you really not open a file with a .QAN extension, or indeed any extension, and read it in as a text file a row at a time?

eg

Code:
Sub readQANFile

Dim f As Long
Dim fname As String
Dim s As String
Dim lin As String

fname = CurrentProject.Path & "\" & "QB100833.qan"
f = FreeFile

Open fname For Input As #f

s = ""
While Not EOF(f)
   Line Input #f, lin
    s = s & lin & vbCrLf
Wend
Close #f

MsgBox s

End Sub
 

moke123

AWF VIP
Local time
Today, 12:02
Joined
Jan 11, 2013
Messages
3,912
How long does it take to process one file? Does it run 24/7 or just for a certain period of time?

My thoughts are to break this up into 2 processes.

The first process would run on a short timer and when run would just check for the existence of a QAN file and if found rename it using Now(), or something similiar, in the filename and move it to a different directory.

The second process would sort the files in order and import the data. I might then move the files to another directory just as a backup in case of a problem.

Access being single threaded may present a problem running the 2 processes concurrently but maybe it would work with 2 instances of access, one to simply rename and move the files and the second instance to process them.
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
Can you really not open a file with a .QAN extension, or indeed any extension, and read it in as a text file a row at a time?

eg

Code:
Sub readQANFile

Dim f As Long
Dim fname As String
Dim s As String
Dim lin As String

fname = CurrentProject.Path & "\" & "QB100833.qan"
f = FreeFile

Open fname For Input As #f

s = ""
While Not EOF(f)
   Line Input #f, lin
    s = s & lin & vbCrLf
Wend
Close #f

MsgBox s

End Sub
HI Dave,

Thanks for your input. I did try that years go when I first stumbled on it, without much success. The file is a bit weird, not really a text file, no delimiters etc. The import side of things works fine if there is only one file, but falls over if there is more than one.
 

spectrolab

Registered User.
Local time
Tomorrow, 00:02
Joined
Feb 9, 2005
Messages
116
How long does it take to process one file? Does it run 24/7 or just for a certain period of time?

My thoughts are to break this up into 2 processes.

The first process would run on a short timer and when run would just check for the existence of a QAN file and if found rename it using Now(), or something similiar, in the filename and move it to a different directory.

The second process would sort the files in order and import the data. I might then move the files to another directory just as a backup in case of a problem.

Access being single threaded may present a problem running the 2 processes concurrently but maybe it would work with 2 instances of access, one to simply rename and move the files and the second instance to process them.
Hi moke123,

The process runs 24/7, when the instrument is running. It generates the .QAN file roughly every 90 seconds.

Your process could work, I didn't think of it that way.

All of the data generated by the instrument is stored in another database (unfortunately, this is proprietary and we can't directly query it) there isn't a reason to keep the results file after we get the data from it.

The system works 98% of the time, until we have an unsupported data type or something strange in the file, or someone closes the database. The instrument keeps running and keeps adding files to the folder, that's when my code falls down.

The solution from arnelgp above works (thanks again!), but the databases is massive (over 1 million records in each table) and I'm accessing it remotely, so it runs a bit slow. I will try and run it on the local machine with a direct connection to the SQL backend tomorrow and see how it goes.
 

moke123

AWF VIP
Local time
Today, 12:02
Joined
Jan 11, 2013
Messages
3,912
Just for my own edification I just tested with 2 seperate database instances.
DB1 checked the folder every 10 secs for a file Q.txt and if found renamed it and moved it to another folder.
Db2 checked the folder every 5 secs for Q.txt and if not found created one. In your situation, this would be the import code db. I just created a file for testing purposes. You would move to the next file in sequence and import it.

At least I now know you can run 2 instances at the same time independantly.

My guess would be that the code for importing would fail easier than code to rename and move a file.
After importing you could delete the file. If the import code failed and stopped functioning at least the file move code would still run and then restarting the import code would pick up where it left off.

Just a thought, hope it makes sense.
 

Isaac

Lifelong Learner
Local time
Today, 09:02
Joined
Mar 14, 2017
Messages
8,774
@spectrolab
You need to re-think your approach.

It's slow because you're engaging in something called RBAR (row by agonizing row) in SQL, or pretty much anything else.

Rather than processing these by SQL inserts one row at a time: First, import the whole raw data into a database table. Then, harness the power of queries and SQL to do the more delicate processing.

This would be the standard ETL approach and probably best/fastest.
 

gemma-the-husky

Super Moderator
Staff member
Local time
Today, 17:02
Joined
Sep 12, 2006
Messages
15,640
they are actually tiny files.

This is the example posted by the OP

I would be inclined to store the first 6 rows in a table, and then store all the readings in a linked table.

Sample name 2139730962
Application As Fe Ore
Measurement time 2/10/2021 8:33:37 AM
Initial weight 0.70g
Final weight 7.48g
LOI (Balance) 1.24%
Fe2O3 92.150 %
SiO2 2.956 %
CaO 0.015 %
MnO 0.044 %
Al2O3 3.164 %
TiO2 0.038 %
MgO 0.183 %
P2O5 0.089 %
SO3 0.015 %
K2O 0.005 %
V2O5 0.014 %
Cr2O3 0.012 %
NiO -0.007 %
CuO 0.007 %
ZnO 0.001 %
PbO -0.007 %
As2O3 0.009 %
Na2O 0.007 %
Cl 0.005 %
ZrO2 0.052 %
SnO -0.001 %
SrO 0.004 %
 

gemma-the-husky

Super Moderator
Staff member
Local time
Today, 17:02
Joined
Sep 12, 2006
Messages
15,640
There's a few issues with your data, but I have loaded your sample file into a database, with a different table structure. This is mdb format. Change the file name in the module.

A few notes.

1. you have multiple spaces between columns. The awf editor removes multiple spaces. When I realised this, I changed my code to tab characters with spaces, then replaces multiple spaces with a single space. I thought they were tabs, but I think they seem to be spaces.

2. in your sample the final weight is greater than the initial weight. Not sure if that is right or not. I have multiplied these by 1000, so 0.7g becomes 700 mg. I thought that might be easier to work with. I left the LOI, I didn't know whether that was a calculation based on the weights.

3. The results include an additional separator for the % sign, which I didn't notice for a while. There are also a few negative percents.

4. The row processing is rather simpler to do that your version. Let me know if anything is confusing.

Anyway this version reads in a QAN file, assumes there is always the same six descriptors at the top, stores this detail in a header table, and and then stores the remaining analysis in a vertical structure, in a details table. I get the sum of the percentages to be 98.755%, which I checked against the original QAN file. It's much easier to work with now, as you can filter/sort/total by chemical.

This imports the QAN file in a fraction of a second. I haven't added a timer, but very quick. This is RBAR, to use @Isaac phrase, but it's not slow, although you do need coding.
 

Attachments

  • qan reader.mdb
    196 KB · Views: 465

Users who are viewing this thread

Top Bottom