The proposed example is not what one should really do with signals, it is just for the purpose of studying.
Well remember this point it will prove important later: http://docs.python.org/dev/library/signal.html#signal.getsignal
There is no way to “block” signals temporarily from critical sections (since this is not supported by all Unix flavors).
The idea transforming a process as a pseudo hardware component
Signals are like wires normally that carries a rising edge. On low level architecture you may use a wire to say validate when results are safe to propagate. and a wire to clear the results.
I do a simple component that just sets to 1 the bit at the nth position of a register according to the position of the wire/signal (could be used for multiplexing).
Here is the code:
#!/usr/bin/env python3.3
import signal as s
from time import sleep
from time import asctime as _asctime
from random import randint
import sys
asctime= lambda : _asctime()[11:19]
class Processor(object):
def __init__(self,signal_map,
slow=False, clear_sig=s.SIGHUP, validate_sig=s.SIGCONT):
self.cmd=0
self.slow=slow
self.signal_map = signal_map
self.clear_sig = clear_sig
self.validate_sig = validate_sig
self.value = 0
self._help = [ "\nHow signal are wired"]
self._signal_queue = []
self.current_signal=None
if validate_sig in signal_map or clear_sig in signal_map:
raise Exception("Dont wire twice a signal")
def top_half(sig_no, frame):
## UNPROTECTED CRITICAL SECTION
self._signal_queue.append(sig_no)
## END OF CRITICAL
for offset,sig_no in enumerate(signal_map):
s.signal(sig_no, top_half)
self._help += [ "sig(%d) sets v[%d]=%d"%(sig_no, offset, 1) ]
self._help += [ "attaching clearing to %d" % clear_sig]
s.signal(clear_sig, top_half)
self._help += [ "attaching validating to %d" % validate_sig ]
s.signal(validate_sig,top_half)
self._help = "\n".join( self._help)
print(self._help)
def bottom_half(self):
sig_no = self._signal_queue.pop()
now = asctime()
seen = self.cmd
self.cmd += 1
treated=False
self.signal=None
if sig_no in self.signal_map:
offset=self.signal_map.index(sig_no)
beauty = randint(3,10) if self.slow else 0
if self.slow:
print("[%d]%s:RCV: sig%d => [%d]=1 in (%d)s" % (
seen,now,sig_no, offset, beauty
))
sleep(beauty)
self.value |= 1 << offset
now=asctime()
print("[%d]%s:ACK: sig%d => [%d]=1 (%d)" % (
seen,now,sig_no, offset, beauty
))
treated=True
if sig_no == self.clear_sig:
print("[%d]%s:ACK clearing value" % (seen,now))
self.value=0
treated=True
if sig_no == self.validate_sig:
print("[%d]%s:ACK READING val is %d" % (seen,now,self.value))
treated=True
if not treated:
print("unhandled execption %d" % sig_no)
exit(0)
wired=Processor([ s.SIGUSR1, s.SIGUSR2, s.SIGBUS, s.SIGPWR ])
while True:
s.pause()
wired.bottom_half()
sys.stdout.flush()
Now, let's do some shell experiment:
$ ./signal_as_wire.py& [4] 9332 660 jul@faith:~/src/signal 19:04:03 $ How signal are wired sig(10) sets v[0]=1 sig(12) sets v[1]=1 sig(7) sets v[2]=1 sig(30) sets v[3]=1 attaching clearing to 1 attaching validating to 18 660 jul@faith:~/src/signal 19:04:04 $ for i in 1 12 10 7 18 1 7 30 18; do sleep 1 && kill -$i %4; done [0]19:04:31:ACK clearing value [1]19:04:32:ACK: sig12 => [1]=1 (0) [2]19:04:33:ACK: sig10 => [0]=1 (0) [3]19:04:34:ACK: sig7 => [2]=1 (0) [4]19:04:35:ACK READING val is 7 [5]19:04:36:ACK clearing value [6]19:04:37:ACK: sig7 => [2]=1 (0) [7]19:04:38:ACK: sig30 => [3]=1 (0) [8]19:04:39:ACK READING val is 12
Everything works as it should, no? :) I have brilliantly used signals to transmit data asynchronously to a process. With 1 signal per bit \o/
What about «not being able to block the signal»
$ for i in 1 12 10 7 18 1 7 30 18; do echo "kill -$i 9455; " ; done | sh [0]22:27:06:ACK clearing value [1]22:27:06:ACK clearing value [2]22:27:06:ACK: sig7 => [2]=1 (0) [3]22:27:06:ACK: sig30 => [3]=1 (0) [4]22:27:06:ACK: sig10 => [0]=1 (0) [5]22:27:06:ACK: sig12 => [1]=1 (0) [6]22:27:06:ACK READING val is 15
Oh a race condition, it already appears with the shell launching the kill instruction sequentially: the results are out of order. Plus you can clearly notice my critical section is not small enough to be atomic. And I lost signals :/
Is python worthless?
Not being able to block your code, is even making a top half/bottom half strategy risky. Okay, I should have used only atomic operations in the top half (which makes me wonder what operations are atomic in python) such has only setting one variable and doing the queuing in the while loop, but I fear it would have been worse.
Which means actually with python, you should not play with signals such as defined in stdlib since without blocking you have systematical race conditions or you risk loosing signals if you expect them to be reliable.
I am playing with signals as I would be playing with a m68K interrupt (I would still block signal before entering the critical section). To achieve the blocking and processing of pending signals I would need POSIX.1 sigaction, sisget, sigprocmask, sigpending.
Why python does not support them (in the stdlib)?
Well python is running on multiple operating systems, some do support POSIX.1 some don't. As signals are not standardized the same way except for POSIX compliant systems with the same POSIX versions, therefore it should not be in st(andar)dlib. And since it is *that* risky I would advocate not allowing to put a signal handler at first place (except for alarm maybe). But, take your own risk accordingly :)
If you feel it is a problem, then just remember binding C code to python is quite easy, and that on POSIX operating system we have everything we need. This solution given in stackoverflow is funky but less than having unprotected critical section: http://stackoverflow.com/a/3792294/1458574.
5 comments:
I'm not sure you fully understand how signals work, conceptually. Signals are not at all like "wires normally that carries a rising edge". Signals are a level-triggered delivery mechanism, meaning they persist in a pending state until acknowledged.
There's also no guarantee that multiple instances of the same signal will result in multiple calls of the handler, as the OS is free to coalesce multiple events of the same signal into one. There's also zero guarantee on the order for regular signals, which is part of your problem with your second test.
However, most importantly, there's zero guarantee about when a signal will be delievered, unless you block it. The last signal delivered to your application will never be processed, if it is delivered while bottom_half is executing. This is responsible for the "missing" signals in your second test.
There's nothing wrong with your top-half handler. CPython promises you that signal handlers always run on the main thread, thus they're always serialized. CPython list is a data structure written in C, thus individual operations on it are "atomic" from an interpreter perspective. The append() can't be re-entered by the pop() (or vice-versa), so they're safe. That wouldn't necessarily be true for other collections, though.
As such, blocking signals and the like won't fix what you're trying to do with your code. It's inherently incompatible with reliable signal delivery. It's notionally compatible with POSIX realtime signal delivery, but you have to step outside Python to get that.
Generally, there are two accepted safe approaches to signal handlers:
1) Set a (sig_atomic_t, in C) global variable for processing through the main loop of your application. This can require special care to be race-safe, depending on what your main loop does.
2) Create a pipe (via os.pipe() and fcntl.fcntl()) and perform a non-blocking write on the pipe in the handler. Non-blocking reads and I/O multiplexers (select.select()) can then be used to signal the main loop about the signal.
Due to the limitations Python imposes upon you, 2 is almost always the better choice.
Thanks Adam, I think I tried to chew a bigger bite I could swallow. And your explanation brings some lights to my confused mind.
Well, I surely better understand how signal works after I wrote this: from time to time, I like to try challenging myself doing funny code based on analogies to see if my brain map works. Anyway my analogy was fishy: wires are inherently working in parallel and in a carefully wired world without any coupling/interaction, while signals are inherently sequential and they can collide. I was heading for disaster since the beginning I admit. I think using a socket based architecture with one socket per wire would have been less stupid :P
I was trying the 1) solution when I discovered using a pipe would be the race safe way anyway.
Than I talked to a friend of mine of my not that wonderful idea and he pointed to me that the pipe solution is doing a lot of kernel space/user space costly context switching. And if you look at signals in python3.3 the signal handling for threads is quite better, plus with locks and memory sharing, I guess a multi-threaded signal handling would be a better architecture (one thread per signal and I have all that I need, but a benchmark could prove useful (plus I don't know how the GIL will interact)).
I think my mistake is that an architecture -even if it looks complex- should always be the closest possible to the problem you want to solve. And I think signals were the worst idea possible. A problem should be stated in terms of properties, and signals don't have the properties needed for handling a parallel problem. The overhead for handling signal complexity is too high.
I do admit I failed on this one :)
But I'll come back with another great idea of mine such as multi agent scheduling.
The overhead of a signal is so high that the additional cost of a pipe read/write is entirely negligible. It's not something to be worried about, and the only added overhead is one system call. The overhead forced upon you by the Python interpreter is far worse.
Signal handling and multi-threaded programs simply do not mix, and even on the few platforms where it does work it's almost impossible to build a safe solution. What Python forces you to do is really the best solution. I would ignore most of the 3.3 additions.
Still https://gist.github.com/3952658 the piped version is still quite unreliable, when signal are sent in close timing but it works better.
I guess signals should be used with parsimony :) A SIGHUP, a SIGLARM, a SIGUSR1/2 at most, with reasonable hypothesis on the rate at which you deliver signals to avoid collisions.
The solution with select.select is less reliable than the pause version. https://gist.github.com/4014563
I guess in C it would work, but allocating this bunch of sigaction structures (one per signal) and setting the sigset gives me headaches. I think I'll forget about the signals and play with sockets or zmq in the future for delivering asynchronous information on mutiple channels.
Soz for the magic number, but who needs errno?
@least it was fun to do.
You can't conclude anything from that code, because it still relies on the order in which the signals are delivered. You may not rely on that ever. Any code that does is soliciting undefined behavior from the operating system. It's just as likely to send demons out your nose as it is to work correctly.
Post a Comment