<HTML>
<HEAD>
<META NAME="GENERATOR" CONTENT="SGML-Tools 1.0.9">
- <TITLE>The DXSpider Installation and Administration Manual : Filtering (New Style v1.45 and later)</TITLE>
+ <TITLE>The DXSpider Installation and Administration Manual : Filtering (Old Style upto v1.44)</TITLE>
<LINK HREF="adminmanual-7.html" REL=next>
<LINK HREF="adminmanual-5.html" REL=previous>
<LINK HREF="adminmanual.html#toc6" REL=contents>
<A HREF="adminmanual-5.html">Previous</A>
<A HREF="adminmanual.html#toc6">Contents</A>
<HR>
-<H2><A NAME="s6">6. Filtering (New Style v1.45 and later)</A></H2>
+<H2><A NAME="s6">6. Filtering (Old Style upto v1.44)</A></H2>
-<H2><A NAME="ss6.1">6.1 General filter rules</A>
-</H2>
-
-<P>Upto v1.44 it was not possible for the user to set their own filters. From
-v1.45 though that has all changed. It is now possible to set filters for just
-about anything you wish. If you have just updated from an older version of
-DXSpider you will need to update your new filters. You do not need to do
-anything with your old filters, they will be renamed as you update.
-<P>
-<P>There are 3 basic commands involved in setting and manipulating filters. These
-are <EM>accept</EM>, <EM>reject</EM> and <EM>clear</EM>. First we will look
-generally at filtering. There are a number of things you can filter in the
-DXSpider system. They all use the same general mechanism.
-<P>
-<P>In general terms you can create a 'reject' or an 'accept' filter which can have
-up to 10 lines in it. You do this using, for example ...
-<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-
-accept/spots .....
-reject/spots .....
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>where ..... are the specific commands for that type of filter. There are filters
-for spots, wwv, announce, wcy and (for sysops) connects. See each different
-accept or reject command reference for more details.
-<P>There is also a command to clear out one or more lines in a filter. They are ...
-<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-clear/spots 1
-clear/spots all
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>There is clear/xxxx command for each type of filter.
-<P>
-<P>and you can check that your filters have worked by the command ...
+<P>Filters can be set for spots, announcements and WWV. You will find the
+directories for these under /spider/filter. You will find some examples in
+the directories with the suffix <EM>.issue</EM>. There are two types of
+filter, one for incoming information and one for outgoing information.
+Outgoing filters are in the form <EM>CALLSIGN.pl</EM> and incoming filters
+are in the form <EM>in_CALLSIGN.pl</EM>. Filters can be set for both nodes
+and users.
+<P>
+<P>All filters work in basically the same way. There are several elements
+delimited by commas. There can be many lines in the filter and they are
+read from the top by the program. When writing a filter you need to think
+carefully about just what you want to achieve. You are either going to write
+a filter to <EM>accept</EM> or to <EM>reject</EM>. Think of a filter as
+having 2 main elements. For a reject filter, you would have a line or multiple
+lines rejecting the things you do not wish to receive and then a default line
+accepting everything else that is not included in the filter. Likewise, for an
+accept filter, you would have a line or multiple lines accepting the things you
+wish to receive and a default line rejecting everthing else.
+<P>
+<P>In the example below, a user requires a filter that would only return SSB spots
+posted in Europe on the HF bands. This is achieved by first rejecting the CW
+section of each HF band and rejecting all of VHF, UHF etc based on frequency.
+Secondly, a filter rule is set based on CQ zones to only accept spots posted in
+Europe. Lastly, a default filter rule is set to reject anything outside the filter.
<P>
<BLOCKQUOTE><CODE>
<PRE>
-
-show/filter
+$in = [
+ [ 0, 0, 'r', # reject all CW spots
+ [
+ 1800.0, 1850.0,
+ 3500.0, 3600.0,
+ 7000.0, 7040.0,
+ 14000.0, 14100.0,
+ 18068.0, 18110.0,
+ 21000.0, 21150.0,
+ 24890.0, 24930.0,
+ 28000.0, 28180.0,
+ 30000.0, 49000000000.0,
+ ] ,1 ],
+ [ 1, 11, 'n', [ 14, 15, 16, 20, 33, ], 15 ], #accept EU
+ [ 0, 0, 'd', 0, 1 ], # 1 = want, 'd' = everything else
+];
</PRE>
</CODE></BLOCKQUOTE>
<P>
-<P>For now we are going to use spots for the examples, but you can apply the same
-principles to all types of filter.
+<P>The actual elements of each filter are described more fully in the following
+sections.
<P>
-<H2><A NAME="ss6.2">6.2 Types of filter</A>
+<H2><A NAME="ss6.1">6.1 Spots</A>
</H2>
-<P>There are two main types of filter, <EM>accept</EM> or <EM>reject</EM>. You
-can use either to achieve the result you want dependent on your own preference
-and which is more simple to do. It is pointless writing 8 lines of reject
-filters when 1 accept filter would do the same thing! Each filter has 10
-lines (of any length) which are tried in order. If a line matches then the
-action you have specified is taken (ie reject means ignore it and accept
-means take it)
-<P>
-<P>If you specify reject filters, then any lines that arrive that match the filter
-will be dumped but all else will be accepted. If you use an accept filter,
-then ONLY the lines in the filter will be accepted and all else will be dumped.
-For example if you have a single line <EM>accept</EM> filter ...
+<P>The elements of the Spot filter are ....
<P>
<BLOCKQUOTE><CODE>
<PRE>
-accept/spots on vhf and (by_zone 14,15,16 or call_zone 14,15,16)
+[action, field_no, sort, possible_values, hops]
</PRE>
</CODE></BLOCKQUOTE>
-<P>then you will <EM>ONLY</EM> get VHF spots <EM>from</EM> or <EM>to</EM> CQ zones
-14, 15 and 16.
<P>
-<P>If you set a reject filter like this ...
+<P>There are 3 elements here to look at. Firstly, the action element. This is
+very simple and only 2 possible states exist, accept (1) or drop (0).
<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-reject/spots on hf/cw
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>Then you will get everything <EM>EXCEPT</EM> HF CW spots. You could make this
-single filter even more flexible. For example, if you are interested in IOTA
-and will work it even on CW even though normally you are not interested in
-CW, then you could say ...
+<P>The second element is the field_no. There are 13 possiblities to choose from
+here ....
<P>
<BLOCKQUOTE><CODE>
<PRE>
-reject/spots on hf/cw and not info iota
+ 0 = frequency
+ 1 = call
+ 2 = date in unix format
+ 3 = comment
+ 4 = spotter
+ 5 = spotted dxcc country
+ 6 = spotter's dxcc country
+ 7 = origin
+ 8 = spotted itu
+ 9 = spotted cq
+ 10 = spotter's itu
+ 11 = spotter's cq
+ 12 = callsign of the channel on which the spot has appeared
</PRE>
</CODE></BLOCKQUOTE>
-<P>But in that case you might only be interested in iota and say:-
+<P>
+<P>The third element tells us what to expect in the fourth element. There are
+4 possibilities ....
<P>
<BLOCKQUOTE><CODE>
<PRE>
-accept/spots not on hf/cw or info iota
+ n - numeric list of numbers e.g. [ 1,2,3 ]
+ r - ranges of pairs of numbers e.g. between 2 and 4 or 10 to 17 - [ 2,4, 10,17 ]
+ a - an alphanumeric regex
+ d - the default rule
</PRE>
</CODE></BLOCKQUOTE>
-<P>which achieves exactly the same thing. You should choose one or the other
-until you are comfortable with the way it works. You can mix them if you
-wish (actually you can have an accept AND a reject on the same line) but
-don't attempt this until you are sure you know what you are doing!
<P>
-<P>You can arrange your filter lines into logical units, either for your own
-understanding or simply convenience. Here is an example ...
+<P>The fifth element is simply the hops to set in this filter. This would only
+be used if the filter was for a node of course and overrides the hop count in
+hop_table.pl.
+<P>
+<P>So, let's look at an example spot filter. It does not matter in the example
+who the filter is to be used for. So, what do we need in the filter? We need
+to filter the spots the user/node requires and also set a default rule for
+anything else outside the filter. Below is a simple filter that stops spots
+arriving from outside Europe.
<P>
<BLOCKQUOTE><CODE>
<PRE>
-reject/spots 1 on hf/cw
-reject/spots 2 on 50000/1400000 not (by_zone 14,15,16 or call_zone 14,15,16)
+$in = [
+ [ 0, 4, 'a', '^(K|N|A|W|VE|VA|J)'], # 0 = drop, 'a' = alphanumeric
+ [ 1, 0, 'd', 0, 1 ], # 1 = want, 'd' = everything else
+ ];
</PRE>
</CODE></BLOCKQUOTE>
-<P>What this does is to ignore all HF CW spots and also rejects any spots on VHF
-which don't either originate or spot someone in Europe.
<P>
-<P>This is an example where you would use a line number (1 and 2 in this case), if
-you leave the digit out, the system assumes '1'. Digits '0'-'9' are available.
-This make it easier to see just what filters you have set. It also makes it
-more simple to remove individual filters, during a contest for example.
+<P>So the filter is wrapped in between a pair of square brackets. This tells
+Spider to look in between these limits. Then each line is contained within
+its own square brackets and ends with a comma. Lets look carefully at the first
+line. The first element is 0 (drop). Therefore anything we put on this line
+will not be accepted. The next element is 4. This means we are filtering by
+the spotter. The third element is the letter "a" which tells the program to
+expect an alphanumeric expression in the fourth element. The fourth element
+is a list of letters separated by the pipe symbol.
<P>
-<P>You will notice in the above example that the second line has brackets. Look
-at the line logically. You can see there are 2 separate sections to it. We
-are saying reject spots that are VHF or above <EM>APART</EM> from those in
-zones 14, 15 and 16 (either spotted there or originated there). If you did
-not have the brackets to separate the 2 sections, then Spider would read it
-logically from the front and see a different expression entirely ...
+<P>What this line does is tell the program to drop any spots posted by anyone in
+the USA, Canada or Japan.
<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-(on 50000/1400000 and by_zone 14,15,16) or call_zone 14,15,16
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>The simple way to remember this is, if you use OR - use brackets. Whilst we are
-here CASE is not important. 'And BY_Zone' is just the same as 'and by_zone'.
-<P>As mentioned earlier, setting several filters can be more flexible than
-simply setting one complex one. Doing it in this way means that if you want
-to alter your filter you can just redefine or remove one or more lines of it or
-one line. For example ...
+<P>The second line is the default rule for anything else. The "d" tells us this
+and the line simply reads... accept anything else.
<P>
-<BLOCKQUOTE><CODE>
-<PRE>
-reject/spots 1 on hf/ssb
-</PRE>
-</CODE></BLOCKQUOTE>
-<P>would redefine our earlier example, or
+<P>You can add as many lines as you need to complete the filter but if there are
+several lines of the same type it is neater to enclose them all as one line.
+An example of this is where specific bands are set. We could write this like
+this ....
<P>
<BLOCKQUOTE><CODE>
<PRE>
-clear/spots 1
+[ 0,0,'r',[1800.0, 2000.0], 1],
+[ 0,0,'r',[10100.0, 10150.0], 1],
+[ 0,0,'r',[14000.0, 14350.0], 1],
+[ 0,0,'r',[18000.0, 18200.0], 1],
</PRE>
</CODE></BLOCKQUOTE>
-<P>To remove all the filter lines in the spot filter ...
+<P>
+<P>But the line below achieves the same thing and is more efficient ....
<P>
<BLOCKQUOTE><CODE>
<PRE>
-clear/spots all
+ [ 0, 0, 'r',
+ [
+ 1800.0, 2000.0, # top band
+ 10100.0, 10150.0, # WARC
+ 14000.0, 14350.0, # 20m
+ 18000.0, 18200.0, # WARC
+ [ ,1 ],
</PRE>
</CODE></BLOCKQUOTE>
<P>
-<H2><A NAME="ss6.3">6.3 Filter options</A>
-</H2>
-
-<P>You can filter in several different ways. The options are listed in the
-various helpfiles for accept, reject and filter.
<P>
-<H2><A NAME="ss6.4">6.4 Default filters</A>
+<H2><A NAME="ss6.2">6.2 Announcements</A>
</H2>
-<P>Sometimes all that is needed is a general rule for node connects. This can
-be done with a node_default filter. This rule will always be followed, even
-if the link is isolated, unless another filter is set specifically. Default
-rules can be set for nodes and users. They can be set for spots, announces,
-WWV and WCY. They can also be used for hops. An example might look like
-this ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-accept/spot node_default by_zone 14,15,16,20,33
-set/hops node_default spot 50
+
+# This is an example announce or filter allowing only West EU announces
+#
+# The element list is:-
+# 0 - callsign of announcer
+# 1 - destination * = all, <callsign> = routed to the node
+# 2 - text
+# 3 - * - sysop, <some text> - special list eg 6MUK, ' ', normal announce
+# 4 - origin
+# 5 - 0 - announce, 1 - wx
+# 6 - channel callsign (the interface from which this spot came)
+
+$in = [
+ [ 1, 0, 'a', '^(P[ABCDE]|DK0WCY|G|M|2|EI|F|ON)' ],
+ [ 0, 0, 'd', 0 ]
+];
</PRE>
</CODE></BLOCKQUOTE>
-<P>This filter is for spots only, you could set others for announce, WWV and WCY.
-This filter would work for ALL nodes unless a specific filter is written to
-override it for a particular node. You can also set a user_default should
-you require. It is important to note that default filters should be
-considered to be "connected". By this I mean that should you override the
-default filter for spots, you need to add a rule for the hops for spots also.
-<P>
-<H2><A NAME="ss6.5">6.5 Advanced filtering</A>
+<P>In this example, only the prefixes listed will be allowed. It is possible to
+be quite specific. The Dutch prefix "P" is followed by several secondary
+identifiers which are allowed. So, in the example, "PA" or "PE" would be ok
+but not "PG". It is even possible to allow information from a single callsign.
+In the example this is DK0WCY, to allow the posting of his Aurora Beacon.
+<P>
+<H2><A NAME="ss6.3">6.3 WWV</A>
</H2>
-<P>Once you are happy with the results you get, you may like to experiment.
-<P>
-<P>The previous example that filters hf/cw spots and accepts vhf/uhf spots from EU
-can be written with a mixed filter, for example ...
<P>
<BLOCKQUOTE><CODE>
<PRE>
-rej/spot on hf/cw
-acc/spot on 0/30000
-acc/spot 2 on 50000/1400000 and (by_zone 14,15,16 or call_zone 14,15,16)
+
+# This is an example WWV filter
+#
+# The element list is:-
+# 0 - nominal unix date of spot (ie the day + hour:13)
+# 1 - the hour
+# 2 - SFI
+# 3 - K
+# 4 - I
+# 5 - text
+# 6 - spotter
+# 7 - origin
+# 8 - incoming interface callsign
+
+# this one doesn't filter, it just sets the hop count to 6 and is
+# used mainly just to override any isolation from WWV coming from
+# the internet.
+
+$in = [
+ [ 1, 0, 'd', 0, 6 ]
+];
</PRE>
</CODE></BLOCKQUOTE>
-<P>Note that the first filter has not been specified with a number. This will
-automatically be assumed to be number 1. In this case, we have said <EM>reject all
-HF spots in the CW section of the bands but accept all others at HF. Also
-accept anything in VHF and above spotted in or by operators in the zones
-14, 15 and 16</EM>. Each filter slot actually has a 'reject' slot and
-an 'accept' slot. The reject slot is executed BEFORE the accept slot.
-<P>
-<P>It was mentioned earlier that after a reject test that doesn't match, the default
-for following tests is 'accept', the reverse is true for 'accept'. In the example
-what happens is that the reject is executed first, any non hf/cw spot is passed
-to the accept line, which lets through everything else on HF. The next filter line
-lets through just VHF/UHF spots from EU.
<P>
+<P>It should be noted that the filter will start to be used only once a user/node
+has logged out and back in again.
+<P>I am not going to spend any more time on these filters now as they will become
+more "comprehensive" in the near future.
<P>
<HR>
<A HREF="adminmanual-7.html">Next</A>
projects / spider.git / blobdiff