/
Data Loader

Data Loader

Owned by Vilhjálmur Hallgrímsson
Last updated: Mar 20, 2024

Uploading Data from CRFM Excel data source to TFM

The Data Load application imports data from spreadsheets into The Fisheries Manager database. It is essentially a tool that facilitates the transfer of data from a spreadsheet file (such as those created with Microsoft Excel, Google Sheets, or similar applications) to The Fisheries Manager database.

Using the data loader Step-by-Step CRFM version

Step 1: 

  • Delete totals and copy down the Fishery name in column A. The total number of species should be 49.

Step 2: 

  • Ensure all species have unique names. See changes in column D.

  • Ensure all species names are available in the CRFM DB.  

Step 3: 

  • Create four new columns: A, B, C, and D, where column A is an Index column

  • Insert each country name 49 times (as many as the number of species). Make sure the country name matches the spelling in the DB.

  • For each country, copy the list of species names and make sure the order is correct.

  • For each country, copy the weight.

Step 4: 

  • Order by weight and delete all zeros and blanks

  • Reorder by index

Step 5:

  • Copy the result into the TFM format and review

  • Make sure EOF is at the bottom row

  • Upload the file 

The data loader explained

  • Parser: The loader first reads the spreadsheet file using a parser. This component understands how to interpret the file format (e.g., XLSX, CSV) and extract data from it.

  • Mapper: Once the data is parsed, a mapping occurs where data from the spreadsheet is matched to the target software or database's expected data format or schema. This involves identifying which columns correspond to which fields in the software.

  • Validator: Before importing the data, the loader may validate it to ensure it meets certain criteria or rules defined by the target software. This step is crucial for maintaining data integrity and can include checks for data types, required fields, and unique constraints.

  • Importer: This component is responsible for actually inserting the data into the target software or database. This might involve creating new records, updating existing ones, or both.

Features

  • Batch Processing: Many loaders are capable of processing large volumes of data in batches to optimize performance and minimize the impact on the target system.

  • Error Handling: Good loaders have robust error-handling mechanisms that can log issues, skip problematic records, or halt the import process if necessary.

  • Custom Mapping: They often allow for custom mapping configurations so that users can specify how each column in the spreadsheet corresponds to fields in the software.

  • Automation: Some loaders can be automated to run at scheduled intervals, enabling regular data updates without manual intervention.

  • User Interface: While some loaders operate as background processes or command-line tools, others come with graphical user interfaces (GUIs) that make configuring and running imports easier for non-technical users.

This procedure describes converting the data in the document "Production by species 2011 totals.xlsx" to  the TFM format.