Hacker Read

theden | karma 972 | avg karma 5.34 · 2021-12-21 07:47:57

I haven't grokked shared memory in python yet to implement it (https://docs.python.org/3/library/multiprocessing.shared_mem...). Apparently there is also a known bug? (https://stackoverflow.com/questions/14124588/shared-memory-i...), but an improvement I'd like to learn and implement. Though the file I have is just over 1GB, so it's not prohibitive yet.

Funny thing is even though it's throwaway code, ensuring everything worked as expected felt really high-stake since a bug would mean a found key would be lost!

Edit: Also not sure if shared memory would be slower in python3 (or if it was, whether it would matter in this use-case), but an interesting thing to profile.

reply

wyldfire | karma 20828 | avg karma 4.03 · 2021-12-21 08:17:41

multiprocessing relies on the OS' fork() to share the memory transparently to the child(ren). The pages containing the memory will refer back to the same physical page until they're written to. This is what grandparent meant by "sharing memory".

multiprocessing also provides a way to access the OS' explicit shared memory usually used as an IPC mechanism.

reply

uniqueuid | karma 3267 | avg karma 4.88 · 2021-12-21 08:23:23

Right, thanks for clearing that up.

multiprocessing's "shared memory" facility is for writable memory.

What I described is extremely handy as you simply move the parsing code up in the script, before your function definition, and "magically" gain memory efficiency.

reply