2. During migration
Data extraction
Collecting the data you want to migrate from one or more sources is the first step in the
migration process itself. To do so, you need to identify all your data sources, as information
is usually stored in different systems. For example, you can have a database where you
have consolidated human rights violation events in the last decade, but you still have
records from the previous years in paper. Or maybe you are partnering with an
organization that sends you this data from a remote region where you don't have your own
staff, but this data is registered in a spreadsheet. Migration can be a good opportunity to
consolidate data, especially if the records are similar but are registered in different
formats, as in this example.
If you still have paper records it is very likely that you will need to transcribe them manually
on a computer (if the number of records is very large you can consider acquiring an OCR
software). However, if you are working with digital files you should check if your source
application (the one where your data is stored) allows you to download the dataset in an
interoperable format.
The CSV format is one of the best choices as it saves the values and its structure in a simple
plain text machine-readable format. The CSV files contain values which would normally be
displayed in columns, but instead are separated by commas. For further information about
how to import and export csv files we recommend you to watch this video.
Data transformation
After the extraction, we should have all the data we want to migrate saved in one or more
interoperable files, so we can transform them with specialized tools, such as OpenRefine,
or simply with a spreadsheet software. The goal of this process is to get your data ready for
the load, in the output format and structure.
As data transformation and data cleaning constitute broad and highly specialized fields of
work, in this document we will focus on basic steps you can follow to have your data as tidy
as possible before the loading (please note that there are several methods that you can
use for the same purposes).
11