Foundations of DL

- Overview

Deep learning (DL) is a type of machine learning (ML) research that uses artificial neural networks (ANNs) to conduct automated data analysis. It's a subset of machine learning (ML) that's based on representation learning and artificial neural networks (ANNs). 

DL is a combination of neural networks, AI, graphical modeling, optimization, pattern recognition, and signal processing. ML is about computers being able to think and act with less human intervention. DL is about computers learning to think using structures modeled on the human brain. 

Early forms of neural networks were inspired by the information processing and decentralized communication nodes in biological systems, particularly the human brain. However, current neural networks are not intended to simulate brain function in living organisms and are often considered low-quality models for this purpose.

- Deep Learning Architecture

Deep learning (DL) is a collection of layers, with the first layer being the input layer, hidden layers in between, and the final layer generating output. Each layer is made up of a group of units called neurons.

DL models can identify complex patterns in text, pictures, sounds, and other data to create accurate predictions and insights.

DL techniques can capture complex relations between non-related fields. For example, they can capture the relationship between air pressure recordings and English words, or between millions of pixels and a textual description. 

DL architectures such as deep neural networks, deep belief networks, recurrent neural networks, convolutional neural networks, and Transformers have been used in computer vision, speech recognition, natural language processing, machine translation, bioinformatics, drug design, and medicine. 

Fields such as image analysis, climate science, materials testing and board game projects produce results that rival and in some cases exceed the performance of human experts.   

Some common DL algorithms (architectures) include: 

• Fully connected networks
• Deep belief networks
• Recurrent neural networks
• Convolutional neural networks
• Generative adversarial networks
• Transformers
• Neural radiance fields

[Schematic overview of layer-wise learning of feature hierarchies. Increasingly complex features are determined from input using unsupervised learning. The features can be used for supervised task learning - Sven Behnke.]

- Deep Learning Artificial Neural Networks

Deep learning is essentially a specialized subset of machine learning characterized by the use of three or more layers of neural networks. These neural networks attempt to simulate the behavior of the human brain (albeit far from its capabilities) in order to "learn" from large amounts of data.

Deep learning artificial neural networks (ANNs), also known as deep neural networks (DNNs), are a type of machine learning and artificial intelligence (AI) that mimic the human brain to recognize, classify, and describe objects. 

They are made up of multiple layers of interconnected nodes, or artificial neurons, that are linked together by weights. The weights are positive if one node excites another, and negative if one node suppresses it.

An artificial neural network (ANN) is a computational system inspired by the fuzziness of the biological neural networks that make up animal brains. 

ANNs are based on collections of connected units or nodes called artificial neurons (blue nodes in the diagram above), which loosely model neurons in biological brains. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. 

An artificial neuron receives the signal, processes it, and can send out signals to the neurons connected to it. The "signal" at the connection is a real number, and the output of each neuron is computed by some nonlinear function of the sum of its inputs. 

These connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weight increases or decreases the signal strength at the connection. 

A neuron may have a threshold so that it only sends a signal when the aggregated signal exceeds that threshold. Typically, neurons are aggregated into layers. 

Different layers can perform different transformations on their inputs. The signal propagates from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers many times. 

ANNs are widely used in a variety of applications, including image recognition, predictive modeling, and natural language processing (NLP). Examples of important commercial applications since 2000 include handwriting recognition for check processing, speech-to-text transcription, oil exploration data analysis, weather forecasting, and facial recognition.  

- Deep Learning Vs. The Human Brain

Deep learning (DL) is a subset of machine learning (ML) that simulates the human brain by taking in large amounts of data and trying to learn from it. DL enables systems to "cluster data and make predictions with incredible accuracy." However, as incredible as DL is, it cannot tap into the human brain’s ability to process and learn information.

DL and DNN (deep neural networks) are used to solve complex real-world problems such as weather forecasting, facial recognition and chatbots, as well as perform other types of complex data analysis.

A better understanding of DL will benefit future applications of AI and machine learning-derived technologies, including fully autonomous vehicles and next-generation virtual assistants. In the future, DL may be developed to enable unsupervised learning and provide additional insights into how the human brain works. 

- The Brain and DNN Models

In order to further deepen the understanding of the scientific community, in the recently published article "Algorithmic Equivalency between the Brain and its DNN Model", researchers proposed and tested a method to understand how AI models and the human brain process information. The goal is to determine whether a DNN model recognizes things in the same way as the human brain, using similar computational steps. This work identifies similarities and differences between AI models and the human brain, taking a step towards creating AI technology that is as close as possible to the human brain processing information.

Understanding more about whether the human brain and its DNN models recognize things in the same way will lead to more accurate real-world applications using DNNs. If we gain a deeper understanding of the recognition mechanisms of the human brain, we can transfer this knowledge to DNNs, which in turn will help improve the way DNNs are used in applications such as facial recognition, which currently do not Not always accurate. 

If the goal is to create the most human-like decision-making process possible, then technology must be able to process information and make decisions at least as well as humans, and ideally better than humans.

[More to come ...]