In the previous parts of this story, I talked about the problem to be solved, and how we did the migration of the raw data to the staging file. Now we'll talk about what needed to be done with that data.
The data we had was all in a single staging file. We needed to extract fields and records and write them to the appropriate PICK files (think "tables" if you're from the relational world). I had to write a program that would take each block of text, and the control codes that said when to change print direction or issue a linefeed, and keep track of the relative offset. For backwards printed text, I had to reverse the order of the letters, all while tracking relative position. I also setup a control file where we defined a "ruler" for the fixed length fields in the print report from the BASIC Four machine. I then extracted out the data, and for each field, wrote it to the appropriate PICK files.
Data Cleansing
Along the way, we ran into some data cleansing issues.
The first one was that spaces needed to be trimmed out, especially leading and trailing spaces. But then things got very interesting.
The system was used to track donations, but it didn't do arithmetic on them. As a result, there was no harm in using upper case "O" instead of a zero, or lower case "L" instead of a 1 (1, l - see they look very much the same!) This meant that wherever there was a field that should have been numeric, you had to do some careful checking. We discovered that we could automatically replace a couple of these characters and recheck that it was numeric and that would get most of the cases.
The other thing that had happened was that people would use the arrow keys. Let's say you were spelling the word "Hello" but you hit the letter "p" (right beside "o") instead. What you really should do is use the backspace key to erase the p, then type the o. Instead, these staff would press the back-arrow key and type "o". This moved the cursor over the p and displayed the o in its place, so it looked right but in the data, you had these characters: "Hellp.o" where the '.' was actually a low ASCII control character. With text this was not so bad, but when you did it with numbers, it created an interesting problem!
We also had to deal with dates. The Microdata couldn't handle lower case dates. "14 Feb 2018" would confuse it, but "14 FEB 2018" was just fine. The BASIC Four was taking dates as text, so it didn't care. It didn't have to do math, it just had to print it. So we had a cleanup around dates, including the back-arrow problem noted above. If you spelled a date "14 FEV 20018" the BASIC Four didn't care. It would simply print it. The Microdata was unable to convert it and gave you an empty string.
Also, from time to time, we had a corrupted record from the transfer into the Microdata. Despite all the delays we put in, sometimes the Microdata would lose a character or two. I would have to go in manually, figure out what the correct positioning was, put some placeholder data in, record the donor information, and we'd have to go back to the BASIC Four to make sure we updated the correct data manually.
Application Development
Finally, we had transferred all the data and were ready to develop the application in Data BASIC. The first order of the day was two simple programs. One to capture information for a new donation, and one to print out a receipt that could be mailed to them. We actually setup a test account that had a copy of the transferred data and started working in there.
Down the hall from their office, Terry Winter had fairly narrow book room. They would send out books as an offer with donations of a certain amount or more. The room had a shelf for boxes of books, and enough room for a table and a chair. That was where I worked. It also had a door to another small room that had the Microdata and the Printronix chain printer as well as the air conditioning and power supply. The cold air would seep through door from the computer room to the book room and I'd sometimes work with my jacket or a sweater on. From time to time, there would be a knock at the door and I'd help a shipper to load a couple of boxes of books on the shelves. Still, it was a good paying job in a recession and I loved the opportunity to create a brand new application for a customer!
Agile Before Agile
This was long before Agile was a thing, but I would make it a point of choosing a key set of features in consultation withe the customer, developing code to the point where the user could see a prototype, then showing it to them and getting their feedback before proceeding. The feature set to develop was always negotiated with the customer. They were holding on to donation data, so as soon as the data entry program had all the essentials in it, they took it and started entering the data. The first version, having been rushed out the door, had some annoyances for the users that impeded productivity, so we focused on those for the next version. The receipt printing program followed immediately on the heels of the first donation entry program version, as they needed to print and ship receipts, then we did the changes to deal with the annoyances. After that, we started building out new features and functionality.
When Agile methodology first came out, we had a bit of trouble understanding what the hype was. We weren't full-on Agile by today's standards, but the concepts were baked into our DNA! I've had to deal with waterfall mode (it still has its place), and I can tell you which approach suits me better!
So that was my first data migration, and my first full application written from scratch!
A couple years later, I was still occasionally doing support for them. Joan Winter called me up and told me about a bug they had encountered. I told her what program to go to, roughly what line number, asked her to read me the code, told her what to change and got her to compile and catalog the program. I fixed the bug over the phone from memory! For me, software is like an old friend. I know it intimately, and can quickly pull it back up from memory. OK, I am weird... I'll admit it!
While the news occasionally caught some big televangelists doing inappropriate things, Terry was the real deal. He didn't drive a Rolls Royce, he drove a station wagon. For quite a while it had plastic in a window because a thief had broken into it. He wasn't about ego. He insisted on the local churches funding his crusades and would only have 1 offering taken, on the last day of his crusades. He was serious about reaching out to Canadians with the gospel. He was not as big as Billy Graham, but he had the same integrity. In December, 1998 Terry Winter passed away suddenly from an aneurysm. I consider myself fortunate to have known him and his family!
No comments:
Post a Comment