Courier Email ClientSupport forum for the Courier Email Client.
Mission Statement
WindowsBBS is an online community dedicated to easily accessible technical support for those using Microsoft operating systems and other Windows software.
Our goal is to become the leading resource for computer users that require assistance with their day-to-day computer usage, including full support for networking PC's, virus & malware removal, system upgrades and general support questions.
I used to have one enormous filter definition and it started getting pretty slow, so I broke it up into several smaller ones. There are two main ones and from those, certain rules have a "Switch to Filter" action, so the procedure moves out of the mainstream to do some of the more nit picky stuff. Having changed the structure, I can have it get pretty far down into the nitty gritty and still process messages in the blink of an eye.
For example, in certain email groups, there are certain topics that come up over and over, which I don't want to read about. So for messages from those groups, we sort of step aside for a moment and look for keywords in the message body. Scanning through the entire message body looking for certain words or phrases is pretty intensive, so this structure has really been working out great. Courier no longer has to search through every message that comes through looking for "kidney stone." (Eww.) It only goes through the messages from that one group where this topic recently came up (this is a temporary rule, and I'll take it off when it stops getting hits). Anyway...
Recently, I realized that I can't seem to make the process come back to the original filter. Old programmers (like me) may recognize that what I want is "return" instead of "end."
For example, in Filter A, any message that has "abc*@yahoo.com" in the To: field (it's a Yahoo disposable address) is sent off to Filter B. Now some of these addresses are for online banking accounts, and Filter B sends them to a variety of folders. Everything else should be sent back to Filter A to run through the rest of its filter rules.
I found a thread about this, but I'm getting somewhat different behavior. The suggestion there was that the rule in Filter A that sends messages off to Filter B should have "continue filtering" checked so that when Filter B is done with it, Filter A picks up where it left off.
The behavior I'm seeing is that if I check "continue filtering" in either filter, then it either gets a hit on that rule and goes into a continuous loop (and Courier has to be forced to shut down, ever so graceful), or it doesn't get a hit on that rule, and it stops. So I can't get Filter A to pick up again after Filter B is done, either way.
The only thing we can both *test* and *set* in a filter rule is the priority flag, and I thought about some scheme where I set a flag and have Filter B "switch to" Filter A, and Filter A doesn't send the message back to Filter B if that flag is set, but that sure seems like a lot of hassle, with so many opportunities to have it do something unintentional. Besides, I just started using the priority flags for something else.
On the other hand, maybe I just need to picture it like the branches of a tree: when it heads off of a main branch onto a smaller one, it's not going to come back.
I can think of a number of solutions to fix *this* problem, but I thought that maybe someone else who fiddles around with filters and might have a more elegant solution.
I considered dropping the wildcard in Filter A, creating a separate filter rule for each online banking address, but that sort of defeats the purpose of having a Filter B at all, and it takes us back a step towards what I used to have -- one big filter that has a huge list of rules, where you can watch the "Details" box as messages come in and trickle through it to the end.
Ultimately, I set up a new set of addresses, all of which have "OLB" in them, so I have one rule that sends them to the second filter and I expect that 100% of them belong there, where they are processed and finished. I just hate to be forced to name accounts to fit some kludgy response to this issue.
Besides, there could well be other situations where I'd like to have one filter "switch to" another filter and then continue when that second filter is done.
So if someone knows a way to get Courier filters to do that, I'd love to hear about it!
Stan
Didn't find the information you thought to find? Check out these Similar Threads
I was going to fade into the background now that Courier, as we know it. is no longer in production, but ... this one is just too tempting.
From what I understand, you set one rule in one filter to switch to another, and then a rule in the other fiilter to switch back to the first .. the problem being that when it reverts to the original filter it starts back at the first rule instead of picking up from the matched rule that caused it to switch to begin with (is anyone following me here .. I am almost confusing myself) ... anyway ... yeah, I can see how that gets you in a perpetual loop.
What I would do is create a third filter duplicating the rules of the original filter from where you left off (even though they are not numbered, let's say rule 3 on filter A sent it to filter B, I would make one of the rules in filter B send it to filter C, with filter C being made up of every rule that came after rule 3 of filter A .. so rules 4 through whatever.
I see how this can be a puzzle with different possible solutions and I almost feel ashamed I am not being at least a little more creative here. But .. the way my brain works is sort of like not wanting to reinvent the wheel .. so if you have rules that are set up that work except for the fact that the switch filtering is creating a loop, I would just add another wheel that ends the loop instead of trying to compete with the program to do what I want it to do (and since I am not a programmer, the program often wins ).
On another note, snce scanning the message body does require the maximum processing of any filter rule, I tend to avoid that at all costs, preferring to use a portion of the message ID from specific addresses/companies or the IP adress (wildcarding the last set of digits if needed) that I can rely on to be consistent. This way I avoid message body scanning and can just limit the scan to the headers. I understand that the method I use works for my mailing habits but may be useless for how you do things. It's just an idea I thought I'd mention since we share a common goal, which is getting filters to make a quick hit on any given message and then move on.
If I misunderstood anything please point it out. This is the end of my day right before going to sleep so my comprehension level is probably only slightly better than that of a goldfish at this moment.
I was going to fade into the background now that Courier, as we know it. is no longer in production, but ... this one is just too tempting.
Well, I've seen some indications that we shouldn't give up all hope just yet. Personally, I plan to keep using Courier until it no longer does the job. That happened when it was Calypso, and my ISP changed to a new authentication method, and after several months using Thunderbird, along came Courier and I couldn't switch back fast enough!
Quote:
Originally Posted by xeno
From what I understand, you set one rule in one filter to switch to another, and then a rule in the other fiilter to switch back to the first ..
Not quite. Thanks for giving me an opportunity to clarify.
All I have is Filter A with a "Switch to filter" to Filter B. I *want* for Filter B to do its thing and, if none of the rules get a hit, to then go back to Filter A and pick up where it left off.
What actually happens in that case is the filtering stops. Checking the "continue filtering" box doesn't make any difference. The "Details" box says that the next rule in Filter A is either a Junkyard base rule or is disabled (it isn't). The same thing happens if one of the rules gets a hit and "continue filtering" isn't checked anywhere.
If one of the rules gets a hit, though, and "continue filtering" is checked, it appears to go into a continuous loop. I say "appears," because I can see messages flickering in the "Details" box, but Courier stops responding so I can't read what's happening. None of the filter rules registers any increase in the hit count, though.
Quote:
Originally Posted by xeno
What I would do is create a third filter duplicating the rules of the original filter from where you left off
Yeah, I don't want to do that, and then every time I change one, I have to remember to change the other one. Part of how I ended up here was that I was streamlining the filter processing and trying to clean up a lot of stuff like that. I had so many places where, say, a color marker was set and then there was a "switch to filter" rule and the color marker was set again, and messages moved into folders everywhere... I cleaned up a lot of that sort of stuff, and I've standardized how I call this action. How long it will stay this tidy is anyone's guess, but as soon as the word "duplicate" hits my brain, I balk.
Quote:
Originally Posted by xeno
On another note, snce scanning the message body does require the maximum processing of any filter rule, I tend to avoid that at all costs, preferring to use a portion of the message ID from specific addresses/companies or the IP adress (wildcarding the last set of digits if needed) that I can rely on to be consistent.
I do not pick it as a first choice by any means. But when it's message content that I'm filtering out, scanning the message body is really the only option. This is a separate issue from the online banking situation I described. This has to do with being in certain discussion groups (I don't know how I'd manage without Courier's filters -- I run through about 300 messages a day from these groups).
In some discussion groups, when I get tired of a thread and it's going on and on and on, for days on end, with dozens of messages each day, I can filter those messages out by subject. But as an example, there was a thread in one group recently where someone started telling about her kidney stones, and several others chimed in about *their* kidney stones, and what it felt like and all kinds of gory details, and people kept changing the Subject field, and I finally set up a filter rule to discard anything that had "kidney" in the message body.
Now, if there were several rules like this, and if every message that came in had to run through those rules, it would slow things down a lot. In fact, that's not too far off from what I used to have, and that's why I started breaking it up.
Now, that discussion group's messages are diverted into their own filter, which gives me two advantages. One, only that group's messages hit that filter rule that scans the message body for "kidney," which is an enormous help to performance. And two, messages in other groups aren't accidentally sent to the trash (say, a message in a recipes group that calls for kidney beans).
But yeah, I'm well aware that scanning message bodies is not a trivial matter.
Thanks so much for your reply and for the thought you put into it!
Well, I've seen some indications that we shouldn't give up all hope just yet. Personally, I plan to keep using Courier until it no longer does the job. That happened when it was Calypso, and my ISP changed to a new authentication method, and after several months using Thunderbird, along came Courier and I couldn't switch back fast enough!
Good point. My sense of hope is just under a little wear and tear from recent developments. Please let me explain. The focus of the Beta was to gain Vista compliance as Courier no longer worked in Vista, to do so they had to switch developers and the whole code sort of had to be rewritten. Other updates, like native SSL, unicode support, etc. were planned to be the focus of updates after the next release was slated. One bug would get fixed and then new bugs would pop up. The Beta was improving with stability on Vista (it seems relatively stable on XP) but just couldn't shake the pattern of fixing one issue without breaking another. That coupled with the fact that it was still to be behind on other technologies that have become somewhat standard now-a-days, the focus went to just working with another company to build on a stable program they already have and try to include some of Courier's similarities and key features. Towards the end of the Beta (really more like Alpha) cycle I was talking to another tester who informed me he was running 3.5 on Vista .. I was like "***?" and asked him to alert the project manager about it. It was a little too late to really do much about it. Then another Vista user posted a more detailed account about getting it to work after indicators that it wouldn't .. basically just ignoring the error messages and persisting. Both of these were testers and I would attribute to have a few tricks up their sleeves, but I now knew it was possible and it was verified. Not long ago a friend contacted me to discuss if there was any way to get 3.5 to work on her just-bought Vista laptop as she was new to Vista and the Beta was crashing on her. This is the clincher to me, I basically told her I had read that she should keep trying despite error messages because that seemed to work for others ... and ... Courier 3.5 runs absolutely fine in Vista with no problems aside from initial error messages which mysteriously vanish afterwards. Lady luck just didn't seem to be smiling on Courier lately, not with the false (or at least fixable) incompatibility messages and the bug count that would simply never cease on subsequent Beta builds.
Now that I've effectively used a tangent to hijack the focus of this thread's purpose, let me try to get back on topic.
Quote:
Originally Posted by stanf
All I have is Filter A with a "Switch to filter" to Filter B. I *want* for Filter B to do its thing and, if none of the rules get a hit, to then go back to Filter A and pick up where it left off.
Thanks for the clarification .. I had a sneaky suspicion first response was over-simplifying the issue.
As a sort of disclaimer I am mostly discussing this with a few suppositions firmly in place, thinking Courier does what I expect it to and that there is no concrete failure to behave as it should. Given you are probably more experienced with filters than am I (and that is saying a lot since I have toyed with them a lot over the years) I am sort of waiting to find out that at least one of my suppositions is wrong. I'll try to list the ones I think are relevant.
1) I have always made rules under the impression hat a "switch to" filter would kick it to the new filter starting at the first rule of that switched-to filter, regardless if it had been there before. So, a message being told in filter A to use Filter B and then hitting a rule telling it to use filter A would not continue from where it left off, but start all over again.
2) I thought the "continue filtering" option only applied to filter rules within its set (filter A or B or whatever), and if the rule told it to consult a different filter then it would always start at the beginning of that filter (as mentioned in "1" above). So under my possibly mistaken impression, it wouldn't continue from where it had left off in the original filter
Neither of those assumptions explains why Courier is giving you a false message about the junkyard or disabled filters ... and even if that message were true it doesn't explain why Courier becomes inoperable.
My plan, if you don't mind, is to have you correct any of my mistaken assumptions so I can use what you say to perform tests myself. Having both 3.5 and the last Beta installed I can at least see if either one handles the circumstances any better than the other. The test will be done later when I have more time to devote to tinkering .. but let me try to think up an example of filter criteria to be tested and please inform me if anything should be adjusted to better match your scenario. I'll start off as simple as I can make it and adjust anything accordingly.
How about this .. filter A
1) set first filter rule to switch to filter B for arriving mail from an address not in my address book AND set to "continue filtering"
2) set filter rule to move all mail (wildcard) to a specific folder
filter B
1) set first filter rule to apply a color marker to mail AND switch to filter A ("continue filtering first left unchecked based on my supposition, then checked if I find out it was mistaken).
According to this, I am expecting a mail sent from an address not in my address book will hit A1, B1, and A2. If a color marker is missing I know it didn't ever hit B1 and if it isn't in a specified folder then I know it never hit A2. If it becomes unresponsive I know that there is a serious bug.
Now, am I missing something (my example also just feels too simplified) or is there any additional filter rule I should add to better match your setup? At least if I can run a test of the most basic setup it can then be made more complex and see what exactly triggers the problem (providing, of course, the sample test works and hits all 3 rules without becoming inoperable).
Quote:
Originally Posted by stanf
What actually happens in that case is the filtering stops. Checking the "continue filtering" box doesn't make any difference.
I had only ever relied on "continue filtering" to apply to the subsequent rules in the same filter, so this part is new to me. Like, I accept HTML mail as a default but have a first filter rule to strip HTML from any address not in my address book, then subsequent rules to assign color markers, play wav alerts, and even another one to run an mp3 if the subject contains a certain trigger word (to indicate an urgency). Although I have used "switch to" filter rules in the past, it was only to save myself from the same thing you dislike, which is making duplicates ... despite considering dupes a viable option, they are never really a great first choice if there's a simple and efficient alternative.
Quote:
Originally Posted by stanf
The "Details" box says that the next rule in Filter A is either a Junkyard base rule or is disabled (it isn't). The same thing happens if one of the rules gets a hit and "continue filtering" isn't checked anywhere.
If one of the rules gets a hit, though, and "continue filtering" is checked, it appears to go into a continuous loop. I say "appears," because I can see messages flickering in the "Details" box, but Courier stops responding so I can't read what's happening.
Heh, not much I can say about that aside from the obvious ... bug!
All your points about duplicates requiring changing ,multiple instances for a single desired effect are quite valid, as well as the need to sometimes use the message body scanning. I could always see reasons to rely on message bodies, but just on my habits I was able to skirt around it and felt it worth mentioning just in case. But ... kidney stones? .. yeah, I would definitely set Courier to scan message bodies to avoid that topic.
Overall, everything I do may simply end up just proving your problem is simply a failure of Courier to do as it should, but I was never the type of person to give up on something without at least giving it a kick or two (and possibly yelling a few vulgarities at it so I can maybe insult it into submission .. which mysteriously has never actually worked, hmmm). So if you don't mind helping me try to duplicate or at least isolate Courier's quirk, I'd like to give this little puzzle a whack and see what we can come up with. Any input on filter rule changes/additions with my example would be greatly appreciated.
filter B
1) set first filter rule to apply a color marker to mail AND switch to filter A ("continue filtering first left unchecked based on my supposition, then checked if I find out it was mistaken).
First, thanks for taking such an interest in this. I have it working to my satisfaction, but I'm always interested in learning more about using filters in Courier, and this issue is really nagging at me.
There is nothing in Filter B with "Switch to filter," so I don't really know how it's going back to Filter A. In fact, if it comes right down to it, maybe it isn't...
When I ran into this problem, I tried isolating it with a test scenario that focused on the problem at hand, so I actually have the definitions, which I will post here (I didn't think anyone would be interested before now):
=== #Filter A ===
Filter Rule 1:
Created: 4/28/2008 11:05 PM
Last Hit: 4/29/2008 1:55 AM
Hits: 8
Mode: Incoming
Case: Off
Pattern 1: stanf*
Objects: To
RegExp: Off
Action: Switch to filter '#Filter B'
Filter Rule 2:
Created: 4/28/2008 11:06 PM
Last Hit:
Hits: 0
Mode: Incoming
Case: Off
Pattern 1: *test*
Objects: Subject
RegExp: Off
Operator: NOT
RegExp: Off
Action: Assign color marker: 'Test Color Marker'
Strip HTML
=== #Filter B ===
Filter Rule 1:
Created: 4/28/2008 11:04 PM
Last Hit: 4/29/2008 12:55 AM
Hits: 4
Mode: Incoming
Case: Off
Pattern 1: howdy doody time
Objects: Subject
RegExp: Off
Operator: NOT
RegExp: Off
Action: Mark for follow-up (High)
Continue filtering
======
That's it.
The test email I kept using to manually send through #Filter A was from one account of mine to another. Both addresses begin with the string "stanf," and the body of the message is simply "01" with a Subject of "Filter Test." The string "howdy doody time" doesn't appear in it (it was late, I was getting a little punchy).
Of course, this is only the last configuration of the filters. I checked and unchecked the boxes to continue filtering, and I also tried toggling the "NOT" operator so I could force a hit or lack of one in #Filter B.
It probably would have been more straightforward to use "01" as the criterion in #Filter B, so it's a hit in its native state. Having it not be a hit, then turning on the NOT operator, so it's not not a hit, well it made my eyes cross a little, well beyond midnight.
You'll notice that #Filter A, Rule 2 never got a hit, ever, no matter what (in my testing, I made it so that Rule 1 always got a hit, sending control over to #Filter B).
Having the scenario drawn up here, maybe it would be a good opportunity to explain that what I *wanted* from it was to have #Filter B with several filter rules that do *not* have "continue filtering" checked, and a last rule that has an asterisk so that anything that wasn't trapped along the way would be hit there, and sent back to #Filter A, Rule 2 (to continue to whatever other rules might come after Rule 2).
But as you can see, nothing I could come up with would make it hit #Filter A, Rule 2. It never occurred to me to have #Filter B explicitly "switch to filter" #Filter A, as I was certain this *would* create a continuous loop.
Oh, here's a new twist. I decided to try doing exactly that, and couldn't! In #Filter B, when I tried "switch to filter," #Filter A didn't show up on the list of available filters, apparently to prevent a continuous loop.
First, thanks for taking such an interest in this. I have it working to my satisfaction, but I'm always interested in learning more about using filters in Courier, and this issue is really nagging at me.
Well, thanks for sharing the puzzle with me. I have run some tests and I can verify much of what you said so we're now close to on the same page .. and, yes, it is quite a nagging issue. I ran multiple tests and could duplicate the looping issue .. organizing my thoughts and keeping track of which factors yielded which results will be my biggest challenge in this post.
Ok, a test in which I set up filters (detailing them at this point would probably complicate the issue because I was constantly changing rules). In this case, filter A and B all had rules to hit the target mail. A1 was set to any address not in my address book and I used a web based disposable account (yahoo really s*cks these days) as that address wasn't in my contacts. Filter A had its 2nd rule set to switch to filter B .. and it did. The mail which matched all rules went to A1, A2, B1 (the first two set to continue filtering and B wasn't). It never made it to A3 as the lack of continuing to filter I suspect prevented that (not sure it matters though). A1 had a non-address-book filter, A2 was set on the subject filter containing "newsletter*", B1 was set to scan the message body for the word kidney (I tend to avoid using blank spaces in filters as, depending on the content, plain text may have a code tag instead of an actual blank space in its source). A3 had the name of the yahoo account set on the "from" field. The color marker and folder settings of the B1 filter were on the mail's final state.
Here is where it got interesting. With the same settings I wrote a mail that did not include the word "kidney" in the message body, so that it was still relayed to filter B but would not hit the rule.... Courier looped! Changing the B filter rule to continue filtering still resulted in a loop. From that I conclude that having it relayed to Filter B and failing to hit any of its rules has severe consequences. I would imagine the rules in Filter B should be set to not continue filtering at any point where it will no longer match a subsequent rule in Filter B, as continuing to be processed without a final (kill) rule sends it back to A and Courier gets toasted. I created a 2nd B filter set to the word "kidney" in the message body, but with the "not" operator, so any message that missed the first rule hit the second. That seems kind of redundant, but, hey, at least it isn't officially a duplicate. It would still require changing two instances for a single desired end result, which bites, but in my tests I was going after one final filtering result so I only had to make sure to match the end-state of the filter in A with the "no match" rule I made in filter B.
The obvious problem here is that if you want the message to end up getting back to A after it has been through B, and that proves detrimental. Making the final settings of the no-hit rule to match the final rule of filter A (leaving most filter settings of the no-hit B rule unused since, in my example, A already did all of the changes I wanted, and only B1 would alter those changes) .. so the no-hit rule was just set to use the same folder since I had to make some filter action to have the filter exist, I just told it to use one of the same actions as the last filter A rule it matched. You could also have it match the same results it would have acquired through filter A had it never went to filter B at all. The way I did this was simply to not have the no-hit filter not change anything, just pointing it to the folder it should have already been in since it did not match B1, which would have sent it to the wastebasket instead.
That seemed to work for me. Since I did a poor job of detailing which settings were changed in their correlating tests there is a chance a detail here or there isn't exact ... tests with the same results I didn't feel warranted details (but in retrospect it would have helped).
Quote:
Originally Posted by stanf
There is nothing in Filter B with "Switch to filter," so I don't really know how it's going back to Filter A. In fact, if it comes right down to it, maybe it isn't...
You're right, it doesn't need to be told to revert to the original filter, it seems to do that if it misses a rule in filter B (and probably if the last rule in B that matches is set to "continue filtering").
Quote:
Originally Posted by stanf
When I ran into this problem, I tried isolating it with a test scenario that focused on the problem at hand, so I actually have the definitions, which I will post here (I didn't think anyone would be interested before now)
Thanks.
Quote:
Originally Posted by stanf
<snipping the actual filters for brevity>
The test email I kept using to manually send through #Filter A was from one account of mine to another. Both addresses begin with the string "stanf," and the body of the message is simply "01" with a Subject of "Filter Test." The string "howdy doody time" doesn't appear in it (it was late, I was getting a little punchy).
Of course, this is only the last configuration of the filters. I checked and unchecked the boxes to continue filtering, and I also tried toggling the "NOT" operator so I could force a hit or lack of one in #Filter B.
I'll have to retest my theory here ... if you forced a hit on filter B via the NOT operator and it was set to not continue filtering ... I thought that would work.
I just retested with and without the kidney word. Without ended on filter B2, with ended on filter B1.
==Filter A==
Filter Rule 1:
Created: 5/1/2008 5:38 AM
Last Hit: 5/1/2008 9:29 AM
Hits: 8628
Mode: Incoming
Case: Off
Pattern 1: [Sender's Address]
Objects: [Address Book]
RegExp: Off
Operator: NOT
RegExp: Off
Action: Strip HTML
Continue filtering
Filter Rule 2:
Created: 5/1/2008 5:45 AM
Last Hit: 5/1/2008 9:29 AM
Hits: 8625
Mode: Incoming
Case: Off
Pattern 1: newsletter*
Objects: Subject
RegExp: Off
Action: Move to folder: 'Received Mail\test'
Assign color marker: '¤'
Switch to filter 'B'
Continue filtering
Filter Rule 3:
Created: 5/1/2008 6:30 AM
Last Hit: 5/1/2008 6:30 AM
Hits: 1
Mode: Incoming
Case: Off
Pattern 1: *random*
Objects: From
RegExp: Off
Operator: OR
Pattern 2: xeno
Objects: From
RegExp: Off
Action: Move to folder: 'Received Mail\test'
Assign color marker: '.'
==Filter B==
Filter Rule 1:
Created: 5/1/2008 5:40 AM
Last Hit: 5/1/2008 9:29 AM
Hits: 3
Mode: Incoming, Outgoing
Case: Off
Pattern 1: kidney
Objects: Message body
RegExp: Off
Action: Move to folder: 'Wastebasket'
Assign color marker: 'þ'
Filter Rule 2:
Created: 5/1/2008 7:37 AM
Last Hit: 5/1/2008 9:28 AM
Hits: 2677
Mode: Incoming
Case: Off
Pattern 1: kidney
Objects: Message body
RegExp: Off
Operator: NOT
RegExp: Off
Action: Move to folder: 'Received Mail\test'
Notes: "random" is part of one of my test account names, nothing special, and my color markers are named with single digit characters to keep the listing clutter free.
Quote:
Originally Posted by stanf
It probably would have been more straightforward to use "01" as the criterion in #Filter B, so it's a hit in its native state. Having it not be a hit, then turning on the NOT operator, so it's not not a hit, well it made my eyes cross a little, well beyond midnight.
LOL .. but the not indicator will cause a hit if the trigger word is not in the configured field. I'm just hoping to verify we have the same intention for the use of the operator. You probably said it properly, but since we are hitting an issue where we get different results so I just want to make absolutely sure we are on the same page.
So far all my experiments have been with the Beta (as I have too many mails since the box was modified and am not sure it can be downgraded easily). I will try to test on 3.5 in a few hours and see if I get different results.
The most notable difference between what we've done seems rather insignificant. I made two opposing rules focusing in on the word kidney, one rule to match its presence in the message body and another to match its absence. You used what should have done the same as a match to an absence, but just regarding an entirely different trigger phrase. That really and truly should not cause any differences. As long as you had B's rules to not continue filtering (as you said you toggled) and an all inclusive hit rule it, well, it's something I will have to test with 3.5.
Quote:
Originally Posted by stanf
You'll notice that #Filter A, Rule 2 never got a hit, ever, no matter what (in my testing, I made it so that Rule 1 always got a hit, sending control over to #Filter B).
Having the scenario drawn up here, maybe it would be a good opportunity to explain that what I *wanted* from it was to have #Filter B with several filter rules that do *not* have "continue filtering" checked, and a last rule that has an asterisk so that anything that wasn't trapped along the way would be hit there, and sent back to #Filter A, Rule 2 (to continue to whatever other rules might come after Rule 2).
That seems to be exactly how it was intended, and possibly how it once worked but has since failed to function properly. In every test, returning to Filter A just killed Courier every time. The only filtering feature I remember was not in Calypso that is present in Courier is the message marking color scheme .... I wonder if that being included in hit filters isthe culprit (or my first guess that the programs code was made during an era when computers processing speed was only a small fraction of today's pcs).
I am not done with this just yet, but will give it a short rest before going at it again. Thanks for all your info and contributions ... I don't really understand why I find such challenges so appealing, but I suppose there are far worse attractions I could have.
Quote:
Originally Posted by stanf
But as you can see, nothing I could come up with would make it hit #Filter A, Rule 2. It never occurred to me to have #Filter B explicitly "switch to filter" #Filter A, as I was certain this *would* create a continuous loop.
Oh, here's a new twist. I decided to try doing exactly that, and couldn't! In #Filter B, when I tried "switch to filter," #Filter A didn't show up on the list of available filters, apparently to prevent a continuous loop.
Indded, it also failed to list an obvious looping trap for me, too.
When I run the test on 3.5 I'll post if it behaves differently than the Beta. If not, I'll toy with it some more and see if I can come up with any bright ideas (like leaving color markers alone until the last rules I expect any messages to hit). 'Til then .. thanks and cheers.
I'll have to retest my theory here ... if you forced a hit on filter B via the NOT operator and it was set to not continue filtering ... I thought that would work.
Sorry, my mistake. Forcing a hit in filter B, with "continue filtering" off, meant that the process stopped at the end of filter B (no continuous looping). I just tried it again and that was the result; I'm sure I just wrote it down in my notes wrong.
===
I've pulled together a couple of thoughts where, along this path, I *almost* got there at least once before, but just didn't keep going...
As I pointed out, this may not actually be a continuous loop, it may be Courier freaking out in some other way. Granted, one goal here would be for Courier not to freak out, but trying to achieve that by avoiding a loop is futile if there isn't one.
One thing that supports this is that, if there were a loop from B to A to B to A, we should see hit counts in each filter skyrocketing. They don't. I explained this to myself by thinking that the hit counts were being held in memory, and when Courier went into a seizure, those updated counts never got written to disk, so forcing Courier to close and reopening it meant returning to a state before the problem occurred. Now I'm not so sure.
In part, I think I kept thinking that it was a loop between filter A and filter B because I had expected the process to return to filter A. But I had noticed a lot earlier in this filter cleanup process that any time I checked "continue filtering" and "switch to filter" in the same rule, I was asking for trouble. It now occurs to me that this is simply a case of that; I was seeing it as different only because I had certain expectations.
I don't know if that has anything to do with this or not, but I notice that "switch to filter" happens before "continue filtering." I wonder if the problem is that there's sort of a conflict between saying "go to this other filter" and "go on to the next step in this filter." I wonder if it would be fixed by changing the order of execution of those two actions, but of course, neither of us has that option.
Heh, okay, I'm back... Saying that made me think about the workaround offered in that thread (make two rules to force the execution to occur in the order you specify). So I had to go try it... Well, DUH, if the first rule doesn't say "continue," it doesn't continue, and the second one, saying "continue," doesn't happen. And if you try to set it up the other way around, well, having the first rule say "for any message, continue filtering," is pretty useless, much like instructing a shoe to fall to the ground when you let go of it. So I'm back to what I said: we can't change the order of execution.
That may or may not be the basis of the problem anyway. I'd say that what we have is a bug wherein you mustn't use "switch to filter" and "continue filtering" in the same filter rule, or you will hang Courier. That's more or less the end of the story.
===
As I was drifting off to sleep last night, a simple workaround to my original problem occurred to me. I still think it's a kludge, but it's probably the most elegant kludge to work around the issue.
Restating the problem:
I want Filter A to call Filter B. Anything that gets to the end of filter B, I want to return to Filter A and continue where it left off. I don't want to duplicate any part of any filter.
Here's my solution, too obvious now that I have noticed it:
Break Filter A into two parts (call them A1 and A2).
Filter A1's second-to-last rule calls Filter B (using desired criteria).
Filter A1's last rule calls Filter A2 (using criterion *: everything hits).
Filter B's last rule also calls Filter A2 using the * criterion.
Filter A2 is, of course, the second half of Filter A.
Anything that is sent to Filter B, and gets to the end of it, goes to Filter A2, which is equivalent to picking up where the current Filter A leaves off.
Anything that isn't sent to Filter B, continues to Filter A2.
I'd rather not have Filter A show up as two filter definitions in my filters list. I'm not sure what all the implications might be if I find a need for more than Filter B (i.e, Filter C, Filter D, etc., where each one executes and then returns control to the Filter A family).
But this certainly solves everything in the problem description.
I set it up and tested a variety of scenarios, and it does what it's supposed to do.
Sorry, my mistake. Forcing a hit in filter B, with "continue filtering" off, meant that the process stopped at the end of filter B (no continuous looping). I just tried it again and that was the result; I'm sure I just wrote it down in my notes wrong.
No worries, keeping all the intricacies detailed and noting the exacts perfect every time was something I knew my own habits had me prone to .. so many different tests and results all involving the same few things forced me to reread every post I made in this thread to try and catch any mistakes (and I did catch at least a couple).
Quote:
Originally Posted by stanf
I've pulled together a couple of thoughts where, along this path, I *almost* got there at least once before, but just didn't keep going...
As I pointed out, this may not actually be a continuous loop, it may be Courier freaking out in some other way. Granted, one goal here would be for Courier not to freak out, but trying to achieve that by avoiding a loop is futile if there isn't one.
Hehe, very true ... there is no logical reason for Courier to just break down like that, so I'd think it has to be a bug, or at least a compatibility issue (faster processors or just some other flaw I am not privy to). There isn't a loop in what it is told to do, just a seeming one when it tries to execute the filters that return it to the original filter set.
Quote:
Originally Posted by stanf
One thing that supports this is that, if there were a loop from B to A to B to A, we should see hit counts in each filter skyrocketing. They don't. I explained this to myself by thinking that the hit counts were being held in memory, and when Courier went into a seizure, those updated counts never got written to disk, so forcing Courier to close and reopening it meant returning to a state before the problem occurred. Now I'm not so sure.
I was inclined to think the same thing, that before it can count the filter hit it has to execute the filter rule, causing the failure as it is executing I would guess keeps it from adding to the hit tally. That's something I'll check in the future, to see if the loop issue ads a hit to the filter rule that was set to switch filters (or allow it to return to the original filter).
Quote:
Originally Posted by stanf
In part, I think I kept thinking that it was a loop between filter A and filter B because I had expected the process to return to filter A. But I had noticed a lot earlier in this filter cleanup process that any time I checked "continue filtering" and "switch to filter" in the same rule, I was asking for trouble.
In my experiments I let the "switch to" rule was set to continue filtering (at least when I tested it on 3.5), but just made sure to get a kill-hit in filter B to prevent it from returning to filter A. My speculation is the problem only exists in B-A ... (and I believe we agree based on your proof of the successful A1-B-A2 test).
Quote:
Originally Posted by stanf
It now occurs to me that this is simply a case of that; I was seeing it as different only because I had certain expectations.
Yep, that can definitely shift one's perspective. It's why I felt the need to state my suppositions because if any of my expectations weren't true to the related function then that would throw my entire game off. And, as it turned out, I did need a little re-adjustment of some of my expectations.
I don't know if that has anything to do with this or not, but I notice that "switch to filter" happens before "continue filtering." I wonder if the problem is that there's sort of a conflict between saying "go to this other filter" and "go on to the next step in this filter." I wonder if it would be fixed by changing the order of execution of those two actions, but of course, neither of us has that option.
I think the witch and continue on the first filter is okay, as I do have the rule in A set to switch to filter B set to continue filtering. It never had the chance to continue back to filter A, though, as it wasn't allowed to pass through Filter B without a hard kill (non-continuing filter rule hit). That was in part of my tests as in to see if it could acquire the color marker in filter A's last rule if it had a kill hit in Filter B .. that never did happen because the returning to match the last A rule would have caused the breakdown. So, in my opinion, the continue to filter option on the filter rule set to switch shouldn't matter if we make sure Courier stops filtering before allowing it to return to A. I think, also, setting the last B rule a given mail were to hit to "continue" would cause the loop. So .. seems to me that it's just a matter of having filtering ended before returning to A, and that the continue to filter on a switch rule is allowed, but (for me) absolutely useless since my goal was stopping it in B.
Quote:
Originally Posted by stanf
Heh, okay, I'm back... Saying that made me think about the workaround offered in that thread (make two rules to force the execution to occur in the order you specify). So I had to go try it... Well, DUH, if the first rule doesn't say "continue," it doesn't continue, and the second one, saying "continue," doesn't happen. And if you try to set it up the other way around, well, having the first rule say "for any message, continue filtering," is pretty useless, much like instructing a shoe to fall to the ground when you let go of it. So I'm back to what I said: we can't change the order of execution.
LOL. Well, we couldn't have an investigation with a proverbial goose chase (and no offense to the composer of the suggesting post, just because it didn't help in this case doesn't mean it isn't helpful advice).
Quote:
Originally Posted by stanf
That may or may not be the basis of the problem anyway. I'd say that what we have is a bug wherein you mustn't use "switch to filter" and "continue filtering" in the same filter rule, or you will hang Courier. That's more or less the end of the story.
To me, as long as it's killed before returning to the original filter set did the trick, but that's just a slightly different angle than what you are saying.
Quote:
Originally Posted by stanf
As I was drifting off to sleep last night, a simple workaround to my original problem occurred to me. I still think it's a kludge, but it's probably the most elegant kludge to work around the issue.
Restating the problem:
I want Filter A to call Filter B. Anything that gets to the end of filter B, I want to return to Filter A and continue where it left off. I don't want to duplicate any part of any filter.
Here's my solution, too obvious now that I have noticed it:
Break Filter A into two parts (call them A1 and A2).
Filter A1's second-to-last rule calls Filter B (using desired criteria).
Filter A1's last rule calls Filter A2 (using criterion *: everything hits).
Filter B's last rule also calls Filter A2 using the * criterion.
Filter A2 is, of course, the second half of Filter A.
Anything that is sent to Filter B, and gets to the end of it, goes to Filter A2, which is equivalent to picking up where the current Filter A leaves off.
Anything that isn't sent to Filter B, continues to Filter A2.
I'd rather not have Filter A show up as two filter definitions in my filters list. I'm not sure what all the implications might be if I find a need for more than Filter B (i.e, Filter C, Filter D, etc., where each one executes and then returns control to the Filter A family).
Great idea! As I was heading towards cramming every end state filter into filter B you came up with a more practical solution.
This would also reword the problem, but .. instead of suggesting a filter Z as I was going to after reading your mentioned potential problems, maybe it's not so much a filter Z as a new organization of the filters to begin with... to follow a different sort of logic to how the filters are filed in the first place. Based on what Courier was supposed to do, your concept of A-B-A was perfect, but apparently Courier won't allow that. Going with A1-B-A2 solves your current problem perfectly, but risks getting complicated if too many more ambiguously named filters are needed, as you said, C, D, E, etc. Even my thinking of an ending filter of Z can work a bit for additional filters, but is prone to running into needs for Z2 and so on if the filtered content varies a great bit.
Okay, one different management scheme I can think of is naming a filter associated with specific mail accounts, fine tuned to the mail those receive. And instead of setting specific hits to switch to filter B, name the filter, for example, after the newsletter, bank, site, whatever. So, let's say a filter is named Gmail, and within Gmail's filter are switch rules that send it to a filter named Newsletter1, or google, or citibank (can have a variety of filters named after the content they are supposed to filter out, such as rule 2 setting it to a filter named newsletter1 and rule three looking for something different sends it to a filter named citibank, each of those secondary rules aimed precisely at the content they will be filtering). The only mandate for this to work is to have each sub-filter (or secondary filter) rule be absolutely sure to make a hit and not continue filltering (or, as you pointed out, send it to yet a different sub-filter).
The above scheme really isn't original or even remotely clever, as I'm sort of just ripping off the concept of folders and sub-folders from Windows, hehe. It's just thinking that associated names may make things a bit simpler to keep track of than assigned letters of the alphabet (which would require a memory or an index of its contents if too many letters get used). Only you can answer how well any naming scheme of your choice would work for you, but I think it's feasible enough in concept to be worth mentioning. It's basically just saying that where pure logic doesn't work because of flawed circumstances beyond our control, that applying a different scheme of logic may keep things simpler in the long run than trying to force something similar to how it should have been in the first place.
Overall, I would say I think our little experiment has given about all the info we're gonna be able to extract from it. The bottom line is something that Courier seems coded to do fails for whatever reason. We know what settings cause the failure so we now know what to avoid. From here on out it's just a matter of creatively coming up with intuitive schemes to get the desired results.
Thanks for bringing this topic up and for all the information and thoughts you've so graciously provided. It's sort of lame to have the solution to a puzzle be that the puzzle is innately flawed, but, hey, it's still a solved mystery.
Okay, one different management scheme I can think of is naming a filter associated with specific mail accounts, fine tuned to the mail those receive. And instead of setting specific hits to switch to filter B, name the filter, for example, after the newsletter, bank, site, whatever. So, let's say a filter is named Gmail, and within Gmail's filter are switch rules that send it to a filter named Newsletter1, or google, or citibank (can have a variety of filters named after the content they are supposed to filter out, such as rule 2 setting it to a filter named newsletter1 and rule three looking for something different sends it to a filter named citibank, each of those secondary rules aimed precisely at the content they will be filtering). The only mandate for this to work is to have each sub-filter (or secondary filter) rule be absolutely sure to make a hit and not continue filltering (or, as you pointed out, send it to yet a different sub-filter).
The above scheme really isn't original or even remotely clever, as I'm sort of just ripping off the concept of folders and sub-folders from Windows, hehe. It's just thinking that associated names may make things a bit simpler to keep track of than assigned letters of the alphabet (which would require a memory or an index of its contents if too many letters get used). Only you can answer how well any naming scheme of your choice would work for you, but I think it's feasible enough in concept to be worth mentioning. It's basically just saying that where pure logic doesn't work because of flawed circumstances beyond our control, that applying a different scheme of logic may keep things simpler in the long run than trying to force something similar to how it should have been in the first place.
In reality, this is pretty much how my filter set *does* work. It's just that there were so many factors in the mix that I decided to set up Filter A and Filter B with just the one or two rules so I could focus on what was hiccuping and why.
But what I normally use has my Yahoo mail coming in to one filter (named "Yahoo Mail") and my SBC Global mail and some other smaller accounts coming in to another (named "!Everything").
I have a third address that is for inquiries to a group I manage, and that address is published in a national magazine. It gets so much spam it's not funny. It's really important that I don't miss any inquiries, but the nature of the "good" mail is so specific that I can have it be super aggressive with certain words in the message bodies (and, I eventually realized, there are six words that pretty well guarantee that it's legitimate, so those are at the top of the filter and kick it out of the filter). There are 196 rules (including comment lines), with most of the actual rules checking the message body. This would never be feasible if this account didn't have its own filter.
My SBC Global account is for most "normal" communication, and doesn't have a lot of rules in its filter. What little mail gets to the end of the Yahoo mail filter goes to the !Everything filter as well.
One of the things I did to streamline all of this was to go through all the Yahoo groups I'm in and switch them over to my Yahoo account, so all of those discussion groups run through that filter (the one for the Yahoo account). There are a lot of disposable addresses, and they come through as if they had been addressed to my Yahoo account (and so through the Yahoo filter), but the "To" field has the disposable address. There's a pattern to the disposable addresses (the first eight characters are always the same).
A huge chunk of what hits the main Yahoo address is discussion group traffic. What hits the disposable addresses is any time I have to give an email address to a merchant (so I can nail them if I start getting spam on an address I gave them). Most of my banking email comes through one of these. Explicitly checking for each banking address in the main filter would mean that there's a lot of traffic that's being tested against about 15 addresses, and that's got to be a big performance hit. So what I was originally shooting for (and now would be able to achieve, by breaking the Yahoo filter in half), was to send all the disposable addresses to a side filter, where each banking address would be tested, mail dumped into folders for each bank, and whatever was left over after all the banking mail had been lifted out would just return to the rest of the main Yahoo mail processing.
Thinking I wasn't really going to solve this, I went through each email address and created a new one, so that the banking addresses all had the usual eight characters, followed by "OLB" and something to identify which bank it was. So I was able to pick out all of the banking mail with 100% accuracy, with one filter rule. But it was tedious, setting up all the addresses at Yahoo and then logging on to all the banks' secure servers and changing my address there.
Quote:
Originally Posted by xeno
It's sort of lame to have the solution to a puzzle be that the puzzle is innately flawed, but, hey, it's still a solved mystery.
On the contrary, Courier may be innately flawed (find a piece of software that isn't), but the solution, at least for me, was breaking up the calling filter, and I think it counts as a genuine solution. I've had plenty of experiences with software where the answer was a disappointing "It doesn't do that." So it's nice this one didn't end up there.
Thanks again for all your interest and all your help!
In reality, this is pretty much how my filter set *does* work. It's just that there were so many factors in the mix that I decided to set up Filter A and Filter B with just the one or two rules so I could focus on what was hiccuping and why.
Ah, true ... the A-B names did help break everything down to the basics in order to figure out exactly where the problem was.
It seems you go through the whole line of all the right things to do with heavy duty filtering and sensible techniques (like being able to determine which entity may have exposed your address to spam lists). As much as I like to think I know about this topic, my bet is if it came down to it that you would probably ending up teaching me a few tricks. On any address that gets any exposure to the outside world I end up sending it through my little gauntlet of a filter, named bullet ... it shoots down anything and everything that does not meet specifics, some to discard and others just sent to the wastebasket so I can give the subject lines quick glance before emptying it. I have the luxury, though, of knowing a little ahead of time what to expect on which addresses .. being open for queries as you are definitely adds a greater need to be able to fine-tune the filtering processes, which it seems you're near making into a perfect science.
Quote:
Originally Posted by stanf
Thinking I wasn't really going to solve this, I went through each email address and created a new one, so that the banking addresses all had the usual eight characters, followed by "OLB" and something to identify which bank it was. So I was able to pick out all of the banking mail with 100% accuracy, with one filter rule. But it was tedious, setting up all the addresses at Yahoo and then logging on to all the banks' secure servers and changing my address there.
Yeah, that sounds a bit much, noting all the server addresses is a great move that, if a standard, could probably rid the world of the vast majority of phishing emails (or at least force the phishers to spoof specific server addresses), but it should kill all the typical zombie bots. My point being that a list of relevant server addresses is still a very handy item, and it would be nice if there was some form of standard in which use of such information could become more commonplace. After all, the reason phishing and general spamming is so popular is for one simple reason .. it works on far too many people.
Quote:
Originally Posted by stanf
On the contrary, Courier may be innately flawed (find a piece of software that isn't), but the solution, at least for me, was breaking up the calling filter, and I think it counts as a genuine solution. I've had plenty of experiences with software where the answer was a disappointing "It doesn't do that." So it's nice this one didn't end up there.
Okay, I cede to your point. While I was shooting to find a way to have it sorted like I believe Courier/Calypso originally intended, isolating the problem and a little creative thinking did manage a nice work-around to get the desired effect, just not strictly between two complete (undivided) filter sets. You're completely right, though, in the end what matters is the ability to get the desired results while keeping everything streamlined and efficient. That much was accomplished.
Quote:
Originally Posted by stanf
Thanks again for all your interest and all your help!
Heh, I just helped you focus on it long enough for you to mull it over and derive the solution yourself. Thanks right back at ya ... I think we both benefited with a little knowledge from this discussion, and if anyone else encounters the same issue there is now a pretty well defined answer to the mystery and work-around solution available.
).
