You may have heard big data mentioned in many specialized scientific and business publications, and you may have even wondered what the term really means. From a technical perspective, big data refers to large and complex amounts of computer data, so large and intricate that applications can’t deal with the data by simply using additional storage or increasing computer power.

Big Data

Extremely large and complex datasets that cannot be managed or processed by traditional data-handling applications.

Big data implies a revolution in data storage and manipulation. It affects what you can achieve with data in more qualitative terms (meaning that in addition to doing more, you can perform tasks better). From a human perspective, computers store big data in different data formats (such as database files and .csv files), but regardless of storage type, the computer still sees data as a stream of ones and zeros (the core language of computers). You can view data as being one of two types, structured and unstructured, depending on how you produce and consume it. Some data has a clear structure (you know exactly what it contains and where to find every piece of data), whereas other data is unstructured (you have an idea of what it contains, but you don’t know exactly how it is arranged).

• Structured data include typical examples such as database tables, in which information is arranged into columns and each column contains a specific type of information. Data is often structured by design. You gather it selectively and record it in its correct place. For example, you may want to place a count of the number of people buying a certain product in a specific column, in a specific table, or in a specific database. As with a library, if you know what data you need, you can find it immediately.
• 

  TIP

  Unstructured data consists of images, videos, and sound recordings. You may use an unstructured form for text so that you can tag it with characteristics, such as size, date, or content type. Usually, you don’t know exactly where data appears in an unstructured dataset, because the data appears as sequences of ones and zeros that an application must interpret or visualize.

  

  REMEMBER

  Transforming unstructured data into a structured form can cost lots of time and effort and can involve the work of many people. Most of the data of the big data revolution is unstructured and stored as is, unless someone renders it structured.

Think about the types of data you interact with every day—how many are structured versus unstructured?

This copious and sophisticated data store didn’t appear suddenly overnight. It took time to develop the technology to store this amount of data. In addition, it took time to spread the technology that generates and delivers data — namely, computers, sensors, smart mobile phones, and the internet and its web services. The following sections help you understand what makes data a universal resource today.

Using data everywhere

Scientists need more powerful computers than the average person because of their scientific experiments. They began dealing with impressive amounts of data years before anyone coined the term big data. At that point, the internet wasn’t producing the vast sums of data that it does today.

REMEMBER

Big data isn’t a fad created by software and hardware vendors but has a basis in many scientific fields, such as astronomy (space missions), satellite (surveillance and monitoring), meteorology (storm predictions), physics (particle accelerators), and genomics (DNA sequences).

Although an AI application can specialize in a scientific field — such as IBM’s Watson, which boasts an impressive medical-diagnosis capability because it can learn information from millions of scientific papers on diseases and medicine — the actual AI application driver often has more mundane facets. Actual AI applications are mostly prized for being able to recognize objects, move along paths, or understand what people say and speak to them. Data contribution to the actual AI renaissance that molded it in such a fashion didn’t derive from the classical sources of scientific data.

TIP

The internet now generates and distributes new data in large amounts. Our current daily data production is estimated to amount to about 2.5 quintillion (a number with 18 zeros) bytes, with the lion’s share going to unstructured data like video and audio.

All this data is related to common human activities, feelings, experiences, and relations. Roaming through this data, an AI can easily learn how reasoning and acting more human-like works. Here are some examples of the more interesting data you can find:

• Large repositories of faces and expressions from photos and videos posted on social media websites like Facebook, YouTube, and Google: They provide information about gender, age, feelings, and possibly sexual orientation, political orientation, or IQ (see “Face-reading AI will be able to detect your politics and IQ, professor says” at The Guardian.com).
• Privately held medical information and biometric data from smartwatches, which measure body data such as temperature and heart rate during both illness and good health: Interestingly enough, data from smartwatches is seen as a method to detect serious diseases, such as COVID-19, early.
• Datasets of how people relate to each other and what drives their interest from sources such as social media and search engines: For instance, a study from Cambridge University’s Psychometrics Centre claims that Facebook interactions contain a lot of data about intimate relationships.
• Information on how we speak, which is recorded by mobile phones. For example, OK Google, a function found on Android mobile phones, routinely records questions and sometimes even more, as explained in “Google’s been quietly recording your voice; here’s how to listen to — and delete — the archive” at https://qz.com/526545/googles-been-quietly-recording-your-voice-heres-how-to-listen-to-and-delete-the-archive.

Every day, users connect even more devices to the internet that start storing new personal data. There are now personal assistants that sit in houses, such as Amazon Echo and other integrated smart home devices that offer ways to regulate and facilitate the domestic environment. These are just the tip of the iceberg because many other common tools of everyday life are becoming interconnected (from the refrigerator to the toothbrush) and able to process, record, and transmit data. The internet of things (IoT) is becoming a reality.