Using UNIX pipes - Simple use of zeppy¶
As the code is being written the tutorial/examples are being written. So there is a qulaity of exploration to this section. Let try to run the simples code possible with this package
Note: You can find this jupyter notebook
at docs/tutorial_docs/simple_zeppy.ipynb
[1]:
# setting the path name so that the tutorial works from this directory
# if you install zeppy with `pip install zeppy` you will not need to do this
import sys
# pathnameto_zeppy = 'c:/eppy'
pathnameto_zeppy = '../../'
sys.path.append(pathnameto_zeppy)
[2]:
from zeppy import ppipes
import time
the function waitsome
will take wait for the time in seconds
[3]:
# from ppipes.py
def waitsome(seconds):
"""wait for some seconds"""
time.sleep(seconds)
return seconds
Let us call the function 5 times and see how long it takes
[4]:
waitlist = [1, 2, 3, 2, 1]
starttime = time.time()
for seconds in waitlist:
ppipes.waitsome(seconds)
endtime = time.time()
print(f'time to run this = {(endtime-starttime):.3f} seconds')
time to run this = 9.024 seconds
it took 9.01 seconds to run -> pretty close 1+2+3+2+1 = 9 seconds Let us try this again using the parallelpipe
function
[5]:
waitlist = [1, 2, 3, 2, 1]
starttime = time.time()
result = ppipes.ipc_parallelpipe(ppipes.waitsome, waitlist)
endtime = time.time()
print(f'time to run this = {(endtime-starttime):.3f} seconds')
print(result)
time to run this = 3.485 seconds
[1, 2, 3, 2, 1]
It took about 3.39 seconds. This is because the ipc_parallelpipe
ran the function 5 times too. But it ran it in parallel - simultanesouly Now let us run it with verbose=True
, so we can see what is going on under the hood.
[6]:
waitlist = [1, 2, 3, 2, 1]
starttime = time.time()
result = ppipes.ipc_parallelpipe(ppipes.waitsome, waitlist, verbose=True)
endtime = time.time()
print(f'time to run this = {(endtime-starttime):.3f} seconds')
print(f'result = {result}')
started worker 0
started worker 1
started worker 2
started worker 3
started worker 4
starting sink
started ventilator
time to run this = 3.460 secondsrunning item: 1, in worker: 2
sent result of item: 1, in worker: 2 to sink
running item: 2, in worker: 1
sent result of item: 2, in worker: 1 to sink
running item: 4, in worker: 3
sent result of item: 4, in worker: 3 to sink
running item: 0, in worker: 0
sent result of item: 0, in worker: 0 to sink
running item: 3, in worker: 4
sent result of item: 3, in worker: 4 to sink
number of calculations = 5
Total taken time for all calcs: 3351 msec
result = [1, 2, 3, 2, 1]
The output tells you what is happening:¶
it starts up 5 workers to run
waitsome
5 timesit starts up the sink. The sink collects the results form the workers
Then it starts the ventilator that sends the 5 tasks to the workers
the tasks are distributed evenly in no particular order between the workers
notice that the first worker got it’s task before the ventilator finished distributed all the tasks and printed
started ventilator
Now let us try this with only 2 workers
[7]:
waitlist = [1, 2, 3, 2, 1]
starttime = time.time()
result = ppipes.ipc_parallelpipe(ppipes.waitsome, waitlist, verbose=True, nworkers=2)
endtime = time.time()
print(f'time to run this = {(endtime-starttime):.3f} seconds')
print(result)
started worker 0
started worker 1
starting sink
started ventilator
time to run this = 5.248 secondsnumber of calculations = 5
Total taken time for all calcs: 5166 msec
running item: 0, in worker: 1
sent result of item: 0, in worker: 1 to sink
running item: 2, in worker: 1
sent result of item: 2, in worker: 1 to sink
running item: 4, in worker: 1
sent result of item: 4, in worker: 1 to sink
running item: 1, in worker: 0
sent result of item: 1, in worker: 0 to sink
running item: 3, in worker: 0
sent result of item: 3, in worker: 0 to sink
[1, 2, 3, 2, 1]
Looks like worker 0
made 3 runs, and worker 1
made 2 runs. The whole thing too 5.14 secs. Still quicker than 9 seconds
This is fun. Can we do some energyplus runs now. Ha! hold on. Let us move slowly. Here is a different function to try
[8]:
def wait_add(first, second):
"""wait for the sum of first and second. return the sum"""
seconds = first + second
return waitsome(seconds)
How are we going to send two args to the function
[9]:
waitlist = [(1, 0), (1, 1), (2, 1), (2, 0), (0, 1)]
starttime = time.time()
result = ppipes.ipc_parallelpipe(ppipes.wait_add, waitlist, verbose=True)
endtime = time.time()
print(f'time to run this = {(endtime-starttime):.3f} seconds')
print(result)
started worker 0
started worker 1
started worker 2
started worker 3
started worker 4
starting sink
started ventilator
time to run this = 3.464 secondsrunning item: 1, in worker: 0
sent result of item: 1, in worker: 0 to sink
running item: 2, in worker: 2
sent result of item: 2, in worker: 2 to sink
running item: 0, in worker: 1
sent result of item: 0, in worker: 1 to sink
number of calculations = 5
Total taken time for all calcs: 3352 msec
[1, 2, 3, 2, 1]
running item: 4, in worker: 4
sent result of item: 4, in worker: 4 to sink
running item: 3, in worker: 3
sent result of item: 3, in worker: 3 to sink
How about a function that has named arguments. Let us try that
Can we send named arguments to the workers ? Suppose we have a function called wait_add_mult
[10]:
def wait_add_mult(first, add=0, mult=1):
"""calculate the result=(first+add)*mult. Then waitsome(result)"""
result=(first + add) * mult
return waitsome(result)
And we want to call it in the following ways:
wait_add_mult(1, add=3)
wait_add_mult(1, mult=3)
wait_add_mult(1, add=2, mult=3)
wait_add_mult(1)
We have to send the args to it in the following way
wait_add_mult(1, add=3) # {'args':(5,), 'kwargs':{'add':3}}
wait_add_mult(1, mult=3) # {'args':(5,), 'kwargs':{'mult':3}}
wait_add_mult(1, add=2, mult=3) # {'args':(5,), 'kwargs':{'add':2, 'mult':3}}
wait_add_mult(1, 2, 3) # {'args': (5, 2, 3)}
[11]:
waitlist = [
{'args':(1,), 'kwargs':{'add':3}},
{'args':(1,), 'kwargs':{'mult':3}},
{'args':(1,), 'kwargs':{'add':2, 'mult':3}},
{'args': (1, 2, 3)},
]
func = ppipes.wait_add_mult
result = ppipes.ipc_parallelpipe(func, waitlist, nworkers=None, verbose=True)
print(result)
started worker 0
started worker 1
started worker 2
started worker 3
starting sink
started ventilator
[4, 3, 9, 9]running item: 0, in worker: 0
sent result of item: 0, in worker: 0 to sink
running item: 1, in worker: 0
sent result of item: 1, in worker: 0 to sink
running item: 3, in worker: 3
sent result of item: 3, in worker: 3 to sink
running item: 2, in worker: 1
sent result of item: 2, in worker: 1 to sink
number of calculations = 4
Total taken time for all calcs: 9466 msec
Thats all for now. Take a look at the next chapter