2.2 Data-driven programming

The Importance of Large Amounts of Data in Machine Learning

In the realm of machine learning (ML), data is often referred to as the "fuel" that powers algorithms and models. The effectiveness and accuracy of machine learning systems heavily depend on the availability of large amounts of high-quality data. This section emphasizes the significance of data in ML, introduces the concept of data-driven programming, and explains fundamental concepts such as datasets, instances, attributes, input variables, and models.

The Role of Data in Machine Learning

Data serves as the foundation upon which machine learning models are built. Without sufficient data, machine learning algorithms cannot learn patterns or make predictions effectively. The importance of large datasets can be summarized in the following points:

1. Learning from Examples: Machine learning is fundamentally about learning from examples. Large datasets provide a diverse range of instances that help algorithms identify patterns and relationships within the data.
2. Improved Accuracy: Models trained on larger datasets tend to generalize better to unseen data, leading to improved accuracy in predictions. This is particularly crucial in applications such as healthcare diagnostics or financial forecasting, where precision is vital.
3. Robustness: A well-rounded dataset that encompasses various scenarios allows models to handle edge cases and anomalies more effectively. This robustness is essential for real-world applications where data can be noisy or incomplete.
4. Feature Representation: Large datasets enable the extraction of meaningful features that can significantly enhance model performance. With more data, algorithms can learn complex representations that capture underlying trends.
5. Mitigating Overfitting: Having a substantial amount of training data helps prevent overfitting, where a model learns noise instead of the underlying distribution. A larger dataset provides a better approximation of the true data distribution.

Data-Driven Programming

Data-driven programming is an approach that emphasizes the use of data as a primary driver for decision-making and algorithm development. In this paradigm, programmers focus on collecting, analyzing, and utilizing data to inform their coding practices rather than relying solely on predefined rules or logic.

Why We Need Data-Driven Programming

1. Adaptability: Data-driven programming allows systems to adapt to changing conditions by continuously learning from new data inputs.
2. Enhanced Decision-Making: By leveraging large datasets, developers can create more informed algorithms that lead to better outcomes.
3. Automation: Data-driven approaches facilitate automation by enabling machines to learn from historical data and make predictions without human intervention.
4. Scalability: As more data becomes available, systems can scale their capabilities without requiring significant changes to their underlying architecture.

Basic Concepts of Machine Learning

1. Dataset

A dataset is a structured collection of data used for training machine learning models. It consists of multiple instances (data points) organized into rows and columns, where each column represents an attribute or feature.

2. Instance

An instance refers to a single record or observation within a dataset. Each instance contains values for various attributes that describe its characteristics.

3. Attribute

Attributes are the individual variables or features that describe an instance in a dataset. They can be numerical (e.g., age, salary) or categorical (e.g., gender, color). Attributes are crucial for defining the input space for machine learning algorithms.

4. Input Variables

Input variables are specific attributes used as predictors in a machine learning model. These variables provide the necessary information for the model to learn patterns and make predictions about output variables (targets).

5. Output Variables

Output variables are the target values that the model aims to predict based on input variables. For example, in a housing price prediction model, input variables might include square footage and location, while the output variable would be the estimated price.

The Concept of a Model

In machine learning, a model is defined as a mathematical representation that maps given input variables to output variables based on learned patterns from training data. The process involves:

1. Training: During training, the model learns from labeled instances in the dataset by adjusting its parameters to minimize prediction errors.
2. Mapping: Once trained, the model can take new input variables and generate corresponding output predictions based on its learned mappings.
3. Evaluation: The performance of a model is evaluated using metrics such as accuracy, precision, recall, and F1 score on validation or test datasets.

The significance of large amounts of suitable data in machine learning cannot be overstated; it is essential for training effective models capable of making accurate predictions in real-world applications. Data-driven programming emerges as a vital methodology that leverages this abundance of data to inform algorithm development and decision-making processes.Understanding fundamental concepts such as datasets, instances, attributes, input variables, output variables, and models lays the groundwork for further exploration into machine learning techniques and their applications across various domains. As we continue to harness the power of data in machine learning, it is imperative to prioritize quality data collection and management practices to ensure robust and reliable outcomes in AI systems.

We will start this story with a brain teaser. Try to conclude from the numbers you have in the left column (we will call them inputs) how they relate to the numbers in the right column (we will call them outputs ).

We assume that this task did not bother you much and that already in the second or third row of the table you came up with the idea that the numbers in the second column, i.e. output, represent the sum of the numbers that are in the left column, i.e. at the entrance. For this task, you also know how to write an algorithm (and by squinting!) that leads you to the solution. For example, In the Python programming language , we can calculate the sum of the elements of the array [1, 9, 5, 4] by initially declaring the sum zero and then adding one element at a time until we reach the end of the array:

numbers = [1, 9, 5, 4]
sum = 0
for element in numbers:
  sum = sum + element

(The built-in sum function calculates this quickly for us and we actually use it more often.)

Now try your hand at the next brain teaser in which you need to figure out how the inputs and outputs are connected. The inputs are now photos of animals, and the outputs are the numbers 0 or 1.

I'm sure you've got a lot of ideas here. And it's very possible that they're all valid! However, since the ones are next to the photos of the cats, and the zeros are next to the rest of the photos of the animals, we wanted you to conclude that this is a task in which you need to recognize whether there is a cat in the photo or not. If there is a photo of a cat at the input, there is 1 at the output, and vice versa, if there is no photo of a cat at the input, there is 0 at the output. You will see a little later that this task is called a binary classification task and is very common in the field of machine learning.

Is it possible to create an algorithm to solve this problem? You will agree that it is important for us to be able to distinguish what is in the photos because that way we can make them easier to search and analyze. You can try to make a list of dozens of rules to determine whether there is a cat in the picture or not. Don't forget to consider the background, lighting, angle of view, and the fact that there are over 50 different types of cats.

There are actually a lot of problems like this, in which even if we try hard (we name 1000 rules!) we are not able to write algorithms that solve them precisely. Some other examples are translating from one language to another and marking faces in photos. You will agree that these tasks are also important to us because they allow us to understand content written in a language we do not speak or to improve security. That is why we describe such problems  with datasets , pairs of inputs and expected outputs, and are solved by data-driven programming techniques. In the case of the task of recognizing cats (and any other objects), a suitable set of data can be organized similarly to ours, with images at the inputs and values of 0 or 1 at the output. In the case of a translation task, it can be pairs of sentences in both languages, while in the case of a face marking task, it can be pairs of images, one without marked faces as input and one with marked faces as output. Here is an example.

Datasets that we can use to describe problems can be created through everyday activities. For example, after examining a patient in an electronic health record, a doctor records information about the patient such as age, gender, symptoms, drug allergies (all of these values are inputs), and the code corresponding to his diagnosis (output). Similarly, at airports, for each flight, information is known such as the time of departure, the airline, the type of aircraft, etc. (all these values are inputs) and information about whether that flight was delayed or not (exit).

Datasets can also be created specifically to solve a specific task. For example, the dataset we used to identify cats could have been created by a team of volunteers who looked at the images we had and followed our guidelines, for example, if there is a cat in the image, write 1, otherwise write 0, enter 1 or 0 in the exit column. For domain datasets, for example, recognizing changes in X-rays, medical experts would need to be hired who have the appropriate skills and knowledge to make decisions. A little later, you'll learn more about how datasets are created.

You're probably wondering: but how do we learn the connection between input and output in a data set? Just as there is an area that deals with the development of classical programming algorithms and analyzing their properties, there is also an area that deals with the development of data-driven algorithms and the examination of their properties. This is called  machine learning. Machine learning is at the core of all modern areas of artificial intelligence because it is closely related to data and ways of deriving knowledge from data. In the next lesson, machine learning will answer the question you are interested in.

It is important to emphasize that there are other areas that deal with data. Among them, the oldest is certainly  statistics, a branch of mathematics that deals with collecting data, describing and analyzing them, as well as drawing conclusions from data. Statistical techniques are at the heart of many machine learning algorithms.  Data Science  is a discipline that emerged as a result of the inability of individual disciplines to answer many interesting questions. For example, every company is faced with the question of how to improve its services. To do this, the company can analyze user comments on social media or sales sites. In order to process comments, it is necessary to collect them in one place and store them in a database, then organize them, for example, separate positive and negative comments, and then analyze each of these sets in a finer way to determine what users evaluate as negative or positive, for example, a specific product model or some functionality. This information should be shared with the company's management so that they can decide what to do next. The answer to the initial question, you notice, is long and requires knowledge and work with tools for downloading content from the web (the so-called scrapers), working with databases, natural language processing, as well as data display techniques that would be most informative to domain experts. In this and other data science application examples, machine learning is an indispensable part of the knowledge pathway.

Before we go any further, let's summarize what problem solving looks like with classical programming and data-driven programming.

When we solve a problem using classical programming techniques, for example, finding the largest element in a sequence of numbers, we first think about the problem and the spatial and temporal constraints we have, then we design an algorithm to solve it (for example, merge sorting), and then we program it in a programming language and save the code. We check the accuracy of the implementation on a few random entries until we make sure that everything works exactly as we expect. When we need to sort a new string, we can use the program we wrote, run it and get the appropriate solution.

When we rely on data-driven programming, we initially have only data, for example, thousands of pairs of inputs and outputs. Again, it is wise to think about the problem first. We do this now by getting to know the dataset. This is done by the exploratory analysis techniques that we will discuss later in the course, which can give us an idea of what form to look for a solution. Then, Instead of devising an algorithm to solve a problem, we  design an algorithm to learn how to solve a problem. This would mean that if we need to learn the connection between input and output and change the data set, this algorithm can again find the best connection between them. The connection that exists between input and output depends on the data (it is not static!) and therefore we need to learn it, And we don't say it directly. When we design and program an algorithm like this, we need to use the data to see how well it works. If we are not satisfied with the results, we need to take a step back and fix the algorithm or go back to the very beginning and check if there is anything else in the data that can be important for solving the problem. Unlike classical programming, this iterativeness is very present in data-driven programming.