Good morning folks,
From some of the responses here, PM's and even emails, it seems I have ruffled a few feathers. I really didn't understand where the ire was coming from until I had some time to think on it.
From what I have given you, all the code I have been provided DOES work. But, like is used to tell my customers while I was on active duty (Logistics/Supply): "I gave you what you asked for, not what you need."
When I tried to plug in my site, the code provided would either fail silently or error on various lines. What was even more curious to me was that EVERY bit of code, when processing the extracted HTML would only return 25 records when there are over 19k.
Since the site is password protected and uses two-tier security - in this case a one-time pin - giving everyone access was not possible. I was able to give one person the "keys to the kingdom" and he, with is very impressive knowledge of website structure, was able to figure out what the issue(s) are.
I will try to explain it here, but when I get it wrong, understand that I am doing trouble shooting by braille here and I am only making a vain attempt to explain what has been explained to me. I know I am going to muck this up, but what the hell...
I assumed that web-scrapping covered ALL methods of getting info from the web, which is why I used that term in the title. The site I am using is more a less a front-end to another site that accesses/stores the raw-data. While I was talking to Edgar, he kept using the term API which confused me because I was assured by the company that there were in fact no API's and that the company who developed the front-end site would not make one. No idea why, but they were quite clear that it was a no-go. Simply getting a connection to their server was also a non-starter since the ability to export the data was provided. In other words, IT made the data accessible; job done. However, the export is hideous and to "clean" the data takes days due to the amount of human interaction.
He corrected me and said these were REST API's - which was clear as dog-shit mud until I did a google search after he explained it to me. This explained why I was only getting the 25 records and why web-scraping would not work. To programmatically "click" the "Next" button to load the next 25 records, process them and repeat would have been painful.
He has found a way to leverage this REST API and get up to 100 records in JSON format at a time, but is still working on how to get a better product.
I hope this clears things up, if not, ask and I will try to explain it, or even better, Edgar will. I like learning new things, but this has me feeling 'purt near stupid...
