Personal tools

DL Research and Applications

Deep Learning Neural Network_030822A
[Deep Learning Neural Network - Pinterest]

Deep Learning: A Technique for Implementing Machine Learning



- Overview

Organizations are increasingly turning to deep learning (DL) because it allows computers to learn independently and perform tasks with little supervision, bringing extraordinary benefits to science and industry. 

Unlike traditional machine learning (ML), DL attempts to mimic the way our brains learn and process information by creating artificial "neural networks" (ANNs) that can extract complex concepts and relationships from data. DL models improve on complex pattern recognition of pictures, text, sounds, and other data to generate more accurate insights and predictions. 

All recent advances in artificial intelligence (AL) in recent years are due to DL. Without DL, we would not have self-driving cars, chatbots or personal assistants like Alexa and Siri. The Google Translate app would continue to be as primitive as 10+ years ago (before Google switched to neural networks for this App), and Netflix or Youtube would have no idea which movies or TV series we like or dislike. Behind all these technologies are neural networks.


Please refer to Wikipedia: Deep Learning for more details.


- Deep Learning and Artificial Neural Network

Deep Learning (DL) is a subset of Machine Learning (ML), which is a subset of Artificial Intelligence AI). DL is just a type of ML, inspired by the structure of a human brain. The adjective "deep" refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

DL algorithms attempt to draw similar conclusions as humans would by continually analyzing data with a given logical structure. To achieve this, DL uses a multi-layered structure of algorithms called neural networks.  

DL uses ANN (Artificial Neural Network) to learn things. There are three types of artificial neural networks: input layer, hidden layer, and output layer. DL is said to be deep because of its hidden layers. Before DL, in neural networks, it was difficult to solve a complex problem with only two layers. In DL, there are multiple hidden layers that form a larger network. It has the potential to solve complex problems.

Most modern DL models are based on multi-layered neural networks such as convolutional neural networks.


- Deep Learning with Big Data on GPUs

The AI boom took off when people realized that they can utilize GPU technology to train DL models much faster than waiting days for a general-purpose CPU to complete one cycle of model training. 

Since 2016, when GPU manufacturers like NVIDIA, Intel, and others created their first AI-optimized GPUs, most AI development has been related to how to train models to be as accurate and predictive as possible.

DL is a type of ML that trains a computer to perform human-like tasks, such as recognizing speech, identifying images or making predictions. Instead of organizing data to run through predefined equations, DL sets up basic parameters about the data and trains the computer to learn on its own by recognizing patterns using many layers of processing.

DL is probably one of the hottest tech topics right now. Large corporations and young startups alike are all gold-rushing this fancy field. If you think big data is important, then you should care about DL. The Economist says that data is the new oil in the 21st Century. If data is the crude oil, databases and data warehouses are the drilling rigs that digs and pumps the data on the Internet, then think of DL as the oil refinery that finally turns crude oil into all the useful and insightful final products. 

There could be a lot of “fossil fuels” hidden underground, and there are a lot of drills and pumps in the market, but without the right refinery tools, you ain’t gonna get anything valuable. That’s why DL is important. It’s part of the data-driven big picture.

The good news is, we are not going to run out of data and our “refinery machine” is getting better and better. Today, just about doing anything online will generate data. In the meantime, as long as the data isn’t garbage-in, then there’s no garbage-out from DL. Also, this “oil refinery” is improving on both software and hardware. 

DL algorithms have improved over the past few decades and developers around the world have contributed to open source frameworks like TensorFlow, Theano, Keras, and Torch, all of which make it easy for people to build DL algorithms as if playing with LEGO pieces. 

And thanks to the demand from gamers around the world, GPUs (graphics processing units) make it possible for us to leverage DL algorithms to build and train models with impressive results in a time-efficient manner.

DL Vs. ML_122623A
[Deep Learning Vs. Machine Learning - XenonStack]

- Deep Learning Accelerators 

Machine learning (ML) is widely used in many modern AI applications. Various hardware platforms are implemented to support such applications. Among them, graphics processing unit (GPU) is the most widely used one due to its fast computation speed and compatibility with various algorithms. 

Field programmable gate arrays (FPGA) show better energy efficiency compared with GPU when computing ML algorithm at the cost of low speed. Deep learning accelerators such as GPUs, FPGAs, and more recently TPUs. More companies have been announcing plans to design their own accelerators, which are widely used in data centers. 

There is also an opportunity to deploy them at the edge, initially for inference and for limited training over time. This also includes accelerators for very low power devices. The development of these technologies will allow ML (or smart devices) to be used in many IoT devices and appliances.


- Deep Learning and Neural Networks

Neural networks are based on the principles of the human brain. A neural network consists of many neurons and connections between them. A neuron can be represented as a function with several inputs and one output. Each neuron takes parameters from inputs (each input may have a different weight, which determines its importance), performs a specific function on them and gives the result to the output. The output of one neuron can be the input for another. Thus, multi-layer neural networks are formed, which are the subject of deep learning.   

DL is ML. More specifically, DL is considered an evolution of ML. It uses a programmable neural network that enables machines to make accurate decisions without help from humans. 

DL achieves great power and flexibility by learning to represent the world as nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representations computed in terms of less abstract ones.

DL software attempts to mimic the activity in layers of neurons in the neocortex, the wrinkly 80 percent of the brain where thinking occurs. The software learns, in a very real sense, to recognize patterns in digital representations of sounds, images, and other data.

The basic idea - that software can simulate the neocortex’s large array of neurons in an artificial “neural network” - is decades old, and it has led to as many disappointments as breakthroughs. 

But because of improvements in mathematical formulas and increasingly powerful computers, computer scientists can now model many more layers of virtual neurons than ever before.


- Deep Learning Architectures

The design of the neural network is based on the structure of the human brain. Just as we use our brains to identify patterns and classify different types of information, neural networks can be taught to perform the same tasks on data.

The individual layers of neural networks can also be thought of as a sort of filter that works from gross to subtle, increasing the likelihood of detecting and outputting a correct result. The human brain works similarly. Whenever we receive new information, the brain tries to compare it with known objects. The same concept is also used by deep neural networks.

Neural networks enable us to perform many tasks, such as clustering, classification or regression. With neural networks, we can group or sort unlabeled data according to similarities among the samples in this data. Or in the case of classification, we can train the network on a labeled dataset in order to classify the samples in this dataset into different categories. Artificial neural networks (ANNs) have unique capabilities that enable DL models to solve tasks that ML models can never solve.

DL isn’t a single approach but rather a class of algorithms and topologies that you can apply to a broad spectrum of problems. While DL is certainly not new, it is experiencing explosive growth because of the intersection of deeply layered neural networks and the use of GPUs to accelerate their execution. 

Big data has also fed this growth. Because DL relies on supervised learning algorithms (those that train neural networks with example data and reward them based on their success), the more data, the better to build these deep learning structures.

There are five of the most popular DL architectures:  

  • Recurrent neural networks (RNNs)
  • Long short-term memory (LSTM)/gated recurrent unit (GRU)
  • Convolutional neural networks (CNNs)
  • Deep belief networks (DBN)
  • Deep stacking networks (DSNs)


[More to come ...]


Document Actions