On or about August 8, 2009, Mark Garlanger proposed a new disk image format for archiving H17 disks. What follows here is a description of some interesting lessons learned while trying to create disk images as samples of Mark's proposed format. A team of three, Mark, Chris Elmquist, and myself have been cooperating on this project since its beginning.
The first routine contrived to read H17 disks used SY.DVD and H17 ROM subroutines. Some of the problems encountered suggested that the H17 ROM sync detection process would have to be used and that some re-ordering of recovered sectors would be necessary. This did not yield an image based on "everything" on a disk, but rather an image based on data recovered only if sync characters were detected by H17 ROM routines and H17 controller hardware. Our goal was to "capture everything on the disk," so this result was not the desired one.
In discussing this with Mark, he indicated that some of the problems were likely related to the fact that bit patterns recovered from streaming magnetic media (including rotating diskettes) can suffer from framing errors. That is, the serial bit stream that makes up the 8-bit bytes may not start at the original byte boundaries but may instead be captured with a bit left over from the previous byte or start on some bit of the current byte other than the original first bit and end with a bit or bits taken from the following byte. Mark's information and encouragement led to renewed coding efforts to read a full track at a time.
When attempting to read a full track with custom disk assess routines, the resulting data capture frequently does not appear to be valid. Study showed what did, in fact, turn out to be framing errors in the serial data from the disk as described above. Furthermore, these errors are not consistent from run to run nor from sector to sector or even from header to sector in a single run.
For example, after waiting for a sector hole to pass, one would expect a 3200 byte read to capture an entire track including padding, headers, sync characters, etc. While this approach occasionally produces the desired result, for most runs it does not. Oh, it captures everything, but the data just doesn't appear to be usable. The following is representative of the first 64 bytes of one such unsuccessful attempt. This read began immediately after a sector hole passed.
offset 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
00000000 <00>00 00 00 00 00 00 00 00 80 7e 00 00 03 06 00 ..........~.....
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80 ................
00000020 7e be 92 75 9f 82 e6 32 13 15 82 94 11 11 82 94 ~¾.u..æ2........
00000030 10 83 94 e6 5c 12 6d 50 94 e6 67 12 ff 01 e5 7b ...æ\.mP.æg.ÿ.å{
|
Based on the H17 ROM code and code from the disk initialization program INIT.ABS, the expected result would be 10 null bytes preceding a sync character (0fdH). Instead the data contained a string of nulls followed by the hex values 80 7e. Presuming that the bits come off of the disk least significant first and that we have a 1-bit framing error, we can begin to reconstruct our data.
The bit pattern for 80h is 1000 0000. If this pattern were read lsb first it would have been 0000 0001. The following 7eh (0111 1110) would have remained 0111 1110 (because, for this byte and several others, the pattern for the reversed order happens to be the same as that of the original order). Taken together they would have been 0000 0001 - 0111 1110. If there had been a one bit framing error causing a one bit shift to the left we could shift the pattern one bit to the right an obtain the correct sequence of 0000 0000 - 1011 1111. The "left over" bit (0 in this case) would be shifted to the following byte. When assembled (i.e. reversed) by the hardware and presented to the software the byte pattern would have been 0000 0000 - 1111 1101 or a null followed by our desired fd (hex) sync character. A possible conclusion might be that we need to perform this restoration throughout the captured data and thus obtain our desired accurate "full track" capture. This appears to be the case in the above example. Following our restored sync char, if we restore the next four bytes in a similar fashion, we end up with the header information, i.e. sync, vol id = 0, track # = 0, sector # = 6, checksum = 0ch. Next is another stream of nulls. The first non-null pattern we find is our old friend 80 7e (hex), which we now know we can restore to 00 fd. Problem solved, or is it?
Larger samples reveal that even within a single track read, the bit-shift framing error is not consistent.
offset 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 00000100 e6 1e 99 66 af 0c 85 83 22 29 a9 27 29 90 16 10 æ..f¯...")©')... 00000110 d0 78 13 85 ff 17 9f 80 61 ae 94 e6 1e 99 66 af Ðx..ÿ...a®.æ..f¯ 00000120 8c 1c 00 10 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000140 00 00 00 00 00 00 00 00 00 40 3f 00 c0 81 03 00 .........@?.À... 00000150 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 80 ................ 00000160 fe 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 þ#¦#¦#¦#¦#¦#¦#¦# 00000170 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 ¦#¦#¦#¦#¦#¦#¦#¦# 00000180 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 a6 23 ¦#¦#¦#¦#¦#¦#¦#¦# |
In this sample, from the same run, we seen the end of the 256 byte sector followed by a long stream of nulls. The first non-null pattern in not our expected old friend 80 7e but rather 40 3f. This pattern requires a 2-bit framing error correction. Once applied we see that we are processing vol 0, track 0, sector 7, checksum 0e. If we assume that we need to apply the 2-bit framing error correction to the sector data portion of the record, we would be in error. After a stream of nulls (with a spurious hex 10 in the midst) we see the pattern 80 fe. This is not the familiar 80 7e of the first sample. Upon further inspection, however, we see that if we apply a 1-bit framing error correction to these two bytes and to the 256 following bytes, we get a null then a sync character then a pattern of 47 4c 47 4c . . . or our recognizable distribution disk filler "GLGLGLGLGL . . . " With regard to the "spurious hex 10," I expect that this is a remnant flux pattern from a previous write. I presume this is the sort of thing that the trailing and leading nulls, starting read/write only after sector hole detect, and various software time delays are intended to protect against. We only check for hole detect to start the process and not on every sector we hope to capture. We'll simply have to take such things into account.
What all of this means is that we can, indeed, capture an entire track of data at one pass, but that it will require some careful processing to recover useful information, especially the information that will be useful in Mark's proposed new format. We have to capture a track and examine the bytes in the expected area of the sector header and determine what, if any bit-framing correction is needed. We then have to examine the area of the start of sector data an determine what, if any, bit-framing correction is needed for the sector data. We must then repeat this analysis for each sector header and the start of each sector individually. By doing so, we can read "everything" on a disk, reconstruct it to a meaningful form and use it to archive a 40 track hard sectored disk. No analysis for soft-sectored disks has yet been undertaken, at least not here.
Please feel free to offer questions, comments and suggestions to me at dsemrick@gmail.net or to any of our project team via the SEBHC group.
- -
Dan Emrick
September 30, 2009