That's not how it's supposed to work. I have that checked on several Fusion boards and I'm never notified when someone removes themselves from a PT. There's another spot to stop the PT flasher from showing.
Under Ignore PT Cleanup:
This option prevents the PT flasher from displaying when a person leaves a private topic you are participating in.
If you have "PT Popup Alert" checked, then you will get the PT alert each time you receive a PT.
If this option is selected, you will recieve a Popup Alert on the index page to notify you.
Then you have "Clear PT Flasher"
By default, the PT flasher will remain visible until you have read every new PT topic/reply. This option will clear the PT flasher after viewing your list of private topics. New topics and replies will remain marked as new in the list.
If you leave this unchecked, the PT flasher will never be cleared until you've read all your PTs.
Out of curiosity, which version of Fusion are you running?