Week-1

Python Application Builder

In this week, I had to try different methods to share memory between Python processes as in the Week-3 of community bonding period I was experiencing issues with the Python shared memory module. I implemented the shared memory architecture through the following three ways:

Pickling:

The main advantage of this method was that it can dump any type of Python objects without any hassle which are easily shared between processes. The main problem with this method was the access time overhead and synchronization problems. As VisualCircuit requires each block to be run at fast iterations and handle race conditions, this was not a suitable method to go proceed with.

Shared Arrays:

The multiprocessing library in Python 3.8 provides a SharedArray() object which can be shared between processes with memory locks. This was the most appropriate solution to proceed with but implementation of this library is new and broken. I tried various tweaks and multiple methods using processes, sub-processes and multiprocessing pool to make it work but unfortunately none of them worked. Another problem was that each function has to take arguments in a specific way (def_param=(array, lock)), so the name of the memory needed to be known before the execution of the program. Following were the problems with this approach:

1) As I had many wires and each wire needed a separate identifier, I could not overwrite the current name using a loop because that would replace the previous shared memory with the current one.

2) If I created a list of wires, then I would have to place the index of that wire inside each function definition of a block, which is not possible because the indices of the list will change with each configuration.

3) If I passed one large block of shared memory to all processes along with an index telling which index of the memory contains the data for that specific process, the issue here would be that I would only have one lock for the entire shared memory. The overhead here would be huge because totally independent read/write operations would be blocking each other. Which means, only one process at a time can read or write to the memory or in other words: acquire the lock.

I also asked a question on stackoverflow but did not get any response. The question can be found here.

I also tried using different versions of Python and Python 3.8.3 on Windows and Linux but they all had the same problem. The memory was not being shared across processes and for some reason if one process acquires a lock and then releases it, the other process could not acquire the lock unless the previous process exited. The implementation can be found here.

A manual working 2-block prototype can be found here.

Bug in Python Shared Memory Module:

Python 3.8 introduced the multiprocessing.shared_memory library, which is the first step to implementing IPC tools for communication of unrelated processes. The tool was to be built on the library, however, everything went wrong. The implementation of shared memory in this library is incorrect – memory object is not properly shared among multiple processes and the shared memory object is deleted even if the process just wants to stop using the object without the intention of deleting it. Despite the presence of two calls close() and unlink(), regardless of their call or non-call, the object is deleted when any of the processes using the object terminates.

Memory Maps:

The advantages of using memory maps were their speed as they are allocated in RAM and they are well tested and work great. The issue here again was that only one process could lock the memory at a time and that is why this approach had to be discarded for only this reason.

The only solution I could find that worked was using the POSIX_IPC module for Python. This was the most suitable approach to proceed with, however, it required much more work.

Why use POSIX_IPC implementation?

The transfer and sharing of objects between processes through shared memory are much more efficient than serialization and deserialization, as it is practically free, therefore it fits great for implementing a low-latency high-bandwidth data exchange between processes within a single node. There were a lot of benefits in this case:

Benefits:

Lightweight Processes: As the processes were completely independent now and just needed the wire identifier to work. They no longer needed to inherit all the memory from their parent process. Much Faster: As POSIX_IPC is a low level library, the interprocess communication has very low overhead and is much faster than the traditional Python interprocess communication. Modular: Each process only requires the name of the shared memory and the Wire class handles everything from reading, writing to memory synchronization using semaphores. One of the downsides of this approach is that it does not support Windows as Windows is not POSIX compliant. However, for now, this tool is not aimed at being cross platform. In future, an equivalent library such as win32 might be used to make it cross platform.

At the end of Week-1, a working 2-block prototype using the POSIX_IPC module was developed. It can be found here.

The Youtube video is attached below: