Manicuring the Data

Some people use the term manipulation when speaking about data, giving the impression that the data is somehow changed in an unscrupulous or devious manner. Perhaps a better term is manicuring, which makes the data well-shaped and lovely. No matter which term you use, however, raw data seldom meets the requirements for processing and analysis. To get something from the data, you must manicure it to meet specific needs. The following sections discuss data manicuring needs.

Dealing with missing data

To answer a given question correctly, you must have all the facts. You can guess the answer to a question without all the facts, but then the answer is just as likely to be wrong as correct. Often, someone who makes a decision, essentially answering a question, without all the facts is said to jump to a conclusion. When analyzing data, you have probably jumped to more conclusions than you think because of missing data. A data record, or one entry in a dataset (which is all the data), consists of fields that contain facts used to answer a question. Each field contains a single kind of data that addresses a single fact. If that field is empty, you don’t have the data you need to answer the question using that particular data record.

REMEMBER

As part of the process of dealing with missing data, you must know that the data is missing. Identifying that your dataset is missing information can be quite difficult because it requires you to look at the data at a low level — something that most people are unprepared to do and that is time-consuming even if you do have the required skills. Often, your first clue that data is missing is the preposterous answers that your questions elicit from the algorithm and associated dataset. When the algorithm is the right one to use, the dataset must be at fault. Here are some issues to consider:

• Essential data missing: A problem can occur when the data collection process lacks all the data necessary to answer a particular question. Sometimes you’re better off to drop a fact than to use a considerably damaged fact.
• Some data missing: Less damaged fields can have data missing in one of two ways — randomly or sequentially:
  • Randomly missing data is often the result of human or sensor error. Fixing randomly missing data is easiest. You can use a simple median or average value as a replacement. No, the dataset isn’t completely accurate, but it will likely work well enough to obtain a reasonable answer.
  • Sequentially missing data occurs during some type of generalized failure. Fixing sequentially missing data is significantly harder, if not impossible, because you lack any surrounding data on which to base any sort of guess. If you can find the cause of the missing data, you can sometimes reconstruct it.

Considering data misalignments

Data might exist for each of the data records in a dataset, but it might not align with other data in other datasets you own. For example, the numeric data in a field in one dataset might be a floating-point type (with decimal point), but an integer type in another dataset. Before you can combine the two datasets, the fields must contain the same type of data.

All sorts of other kinds of misalignment can occur. For example, date fields are notorious for being formatted in various ways. To compare dates, the data formats must be the same. However, dates are also insidious in their propensity for looking the same but not being the same. For example, dates in one dataset might use Greenwich Mean Time (GMT) as a basis, whereas the dates in another dataset might use some other time zone. Before you can compare the times, you must align them to the same time zone. It can become even weirder when dates in one dataset come from a location that uses daylight saving time (DST) but dates from another location don’t.

Even when the data types and format are the same, other data misalignments can occur. For example, the fields in one dataset may not match the fields in the other dataset. In some cases, these differences are easy to correct. One dataset may treat first and last names as a single field, while another dataset might use separate fields for first and last names. The answer is to change all datasets to use a single field or to change them all to use separate fields for first and last names. Unfortunately, many misalignments in data content are harder to figure out. In fact, it’s entirely possible that you might be unable to figure them out. However, before you give up, consider these potential solutions to the problem:

• Calculate the missing data from other data you can access.
• Locate the missing data in another dataset.
• Combine datasets to create a whole that provides consistent fields.
• Collect additional data from various sources to fill in the missing data.
• Redefine your question so that you no longer need the missing data.

Separating useful data from other data

Some organizations’ leaders are of the opinion that they can never have too much data, but an excess of data becomes as much a problem as not enough. To solve problems efficiently, an AI requires just enough data. Defining the question that you want to answer concisely and clearly helps, as does using the correct algorithm (or algorithm ensemble). Of course, the major problems with having too much data are that finding the solution (after wading through all that extra data) takes longer, and sometimes you get confusing results because you can’t see the forest for the trees.

WARNING

As part of creating the dataset you need for analysis, you make a copy of the original data rather than modify it. Always keep the original, raw data pure so that you can use it for other analysis later. In addition, creating the right data output for analysis can require a number of tries because you may find that the output doesn’t meet your needs. The point is to create a dataset that contains only the data needed for analysis, but keep in mind that the data may need specific kinds of pruning to ensure the desired output.