Parallel Supercomputing
- [California Institute of Technology, US News]
- Overview
Parallel computing is the concurrent use of multiple processors (CPUs) to do computational work. In traditional (serial) programming, a single processor executes program instructions in a step-by-step manner. Some operations, however, have multiple steps that do not have time dependencies and therefore can be separated into multiple tasks to be executed simultaneously.
For example, adding a number to all the elements of a matrix does not require that the result obtained from summing one element be acquired before summing the next element. Elements in the matrix can be made available to several processors, and the sums performed simultaneously, with the results available faster than if all operations had been performed serially.
Parallel computations can be performed on shared-memory systems with multiple CPUs, distributed-memory clusters made up of smaller shared-memory systems, or single-CPU systems. Coordinating the concurrent work of the multiple processors and synchronizing the results are handled by program calls to parallel libraries; these tasks usually require parallel programming expertise.
Please refer to the following for more information:
- Wikipedia: Parallel Programming Model
- Parallel Programming and Multithreading
Parallel programming and multithreading are both ways to use resources to solve problems.
Parallel programming is the process of breaking down a large task into smaller tasks and solving them simultaneously. This is done by using multiple CPU cores and parallel hardware, such as multi-core processors, GPUs, and FPGAs.
Multithreading is the process of executing multiple sequential sets of instructions at the same time. This is done by allowing a single processor to spawn multiple threads. Each thread runs its own instructions, but they all share a memory space and can communicate with each other.
Multithreading is important because:
- Processors have reached their maximum clock speed.
- It allows things to appear to be happening at the same time.
- It can be used when the results of future computations depend on the results of current computations.
- It can be used when things aren't CPU intensive.
- It can be used when the overheads of using multiple processes are bad.
Parallel programming is used when there are a series of independent computations.
- Parallel Computing for Modeling and Computation
Parallel computing is a computing architecture that breaks down a problem into smaller tasks and runs them simultaneously. This allows computers to process multiple tasks at once, making parallel computing faster than a sequential computer. Parallel computing can help solve large, complex problems in a shorter time.
Parallel computing has several forms, including bit-level, instruction-level, data, and task parallelism. Data parallelism is a parallel programming model that distributes data across different nodes in the parallel execution environment, enabling simultaneous sub-computations on these distributed data across the different compute nodes.
Parallel computing has many applications, including:
- Databases and data mining
- Real-time simulation of systems
- Science and engineering
- Advanced graphics, augmented reality, and virtual reality
- Biomedical engineering
- Pharmaceutical design
- Economic forecasting
- Collaborative digital workspaces
- Supporting multimedia sharing
Some benefits of parallel computing include:
- Computers can execute code more efficiently
- Can save time and money by sorting through “big data” faster
- Can solve more complex problems
Some challenges of parallel computing include: Load balancing and Data dependencies.
Parallel programming models are closely related to models of computation. A model of parallel computation is an abstraction used to analyze the cost of computational processes, but it does not necessarily need to be practical, in that it can be implemented efficiently in hardware and/or software.
- Challenges and Applications of Parallel Supercomputers
In parallel computing, a larger task is divided into smaller subtasks, which are then distributed across multiple computers. The computers may be located in the same physical location, or they may be spread out across different geographical locations.
Some challenges of parallel computing include: Finding and expressing concurrency, Managing data distributions, Managing inter-processor communication, Balancing the computational load, and Implementing the parallel algorithm correctly.
Parallel supercomputers have been in the mainstream of high-performance computing for the past decades. However, their popularity is waning.
The reasons for this decline are many, but include factors like being expensive to purchase and run, potentially difficult to program, slow to evolve in the face of emerging hardware technologies, and difficult to upgrade without, generally, replacing the whole system.
The decline of dedicated parallel supercomputer has been compounded by the emergence of commodity-off-the-shelf clusters of PCs and workstations.
Parallel computing has many commercial applications, including:
- Big data
- Data mining
- Artificial intelligence (AI)
- Oil exploration
- Web search engines
- Medical imaging and diagnosis
- Pharmaceutical design
- Financial and economic modeling
- Advanced graphics and virtual reality
- Parallel Computing vs. Distributed Computing
Parallel computing and distributed computing are both powerful technologies that can be used for complex computations. They have some important differences, including:
- Number of computers: Parallel computing uses one computer with multiple processors. Distributed computing uses multiple distinct computers.
- Memory: In parallel computing, all processors share the same memory. In distributed computing, each computer has its own memory.
- Location: Parallel computing splits an application up into tasks that are executed at the same time. Distributed computing splits an application up into tasks that are executed at different locations using different resources.
Distributed computing is often used in tandem with parallel computing. Distributed parallel computing uses multiple computing devices to process tasks.
Distributed computing can improve the performance of solutions by taking advantage of hundreds or thousands of computers running in parallel. It offers unmatched scalability, better overall performance and more reliability. This makes it a better solution for businesses dealing with high workloads and big data.
Parallel computing works well when you have a single powerful machine with multiple processors. Distributed computing is preferable when dealing with large datasets or when the processing task cannot be efficiently handled by a single machine.
[More to come ...]
