Data Base Loading

Along with getting the data into a usable form for a database there is the issue of how to identify elements and refer to them in the database and its tables. This step I call Database Loading and is closely related to identification and extraction.

To load database tables, a list of data elements, their type and, for some data types, the size (or width) of the field is needed. This list becomes the data table schema or format and tell the database how to organize and refer to the data elements.

As with extraction, this step becomes more complex for larger data sets or data in multiple files, or for data structures requiring multiple tables. I have found that preparing schema and creating tables followed by preparing data in flat ASCII files organized to compliment the schema facilitates the loading process. Small datasets can be loaded with the MySQL "insert" process, but for larger systems of 100 million or more records, creating the flat files and then using a "load process" allows the loading to be accomplished more quickly.