Parallel Programming with C# and .NET Framework 4.0 - Distributed-Memory Systems

It's been a long time since my last post and I was having a tough time. And that tough time seems to be having like forever I thought I can't let them disturb my work. Then lets begin.

In my previous post I wrote about Parallel Programming and I said that I will write some posts related to Parallel Programming. In that post I wrote what Parallel Programming is and I described the simplest microprocessor architecture which is Shared-Memory Multicore.

Today I am going to continue with another topic that will be discussed in Parallel Programming which is Distributed-Memory Systems. Today also I should mention that I am referring the book "Professional Parallel Programming with C# - Master Parallel Extensions with .NET 4 - 2010" by "Gaston C. Hillar" which is a Wrox Publication. I am getting data (both information and images) for my posts about Parallel Programming from that book. I think everyone who is interested in Parallel Programming should start reading that book and a great appreciation goes out to Gaston C. Hillar.

Post 02. Distributed-Memory Systems

You can think of a Distributed-Memory System as an interconnected microprocessors with their own private memory. A distributed-memory system forces you to think about the distribution of the data, and accessing them through different ways and from different places. You can add new machines (nodes) to increase the number of microprocessors for the system and by doing that, distributed-memory systems can offer a great scalability.

Take a look at the following image.

Distributed-Memory Systems Architecture (small)

In here each microprocessor can be in a different computer, with different types of communication channels between them. The thing to note is the communication channel which interconnects the each microprocessor. One of the most popular communications protocols used to program parallel applications to run on distributed-memory systems is Message Passing Interface (MPI). And the most interesting thing with MPI and .NET is you can use MPI with C#.

Let's say if a process which is running in one of the microprocessors needs remote data, it has to communicate with the corresponding remote microprocessor through the communication channel. And you might think that it will add a additional work and a pretty good overhead, because not like in Shared-Memory Multicore, messages has to be trasfered through a communication channel. And because of that reason Distributed-Memory Systems are only used for applications that do high end calculations and for applications that has distributed processing and distributed data access.

Take a good look at the following imge. It's actually the larger image of what I explained above.

Distributed-Memory Systems Architecture (large)

It is a Distributed-Memory Computer System with three machines. Each machine has a quad-core microprocessor, and a shared-memory architecture for these cores. This way, the private memory for each microprocessor acts as a shared memory for its four cores.

And that's a brief explanation about Distributed-Memory Systems. Hope you found it interesting and feel free to give me your feedback.

Happy Coding.

Regards,

Jaliya