X-Git-Url: http://gb7djk.dxcluster.net/gitweb/gitweb.cgi?a=blobdiff_plain;f=Changes;h=dc40b96a9a1ad1b845fb0075a3258c8ee9e00a30;hb=710e02b70cb2530802812577229cd62a50da8090;hp=0acea0c7877c1ae78f0188e042e46cd60135eb52;hpb=f15a1b5b704df2ea24cbda637ad21ad70b197cf9;p=spider.git diff --git a/Changes b/Changes index 0acea0c7..dc40b96a 100644 --- a/Changes +++ b/Changes @@ -1,3 +1,321 @@ +05Jan22======================================================================= +1. Mark nodes that send PC92 K records as spider. These will include VE7CC + nodes. NOTE: there appear to be user records marked as user or other sorts + of node, which (now) are actually spider (compatible) nodes and will be + marked accordingly. +2. Adjust nodes currently marked as spider nodes, but are sending versions + not in the spider range of versions on PC92 A records as AK1A. +3. Try to undo some damage where users have been autocreated with similar + attributes as nodes (locked out with privilege set to 1). This will + slowly fix this problem over time, but see item 4 for a 'big bang' + approach. +4. It has come to my attention that there are a large number of users (of + all sorts) that have incompatible SSIDs. See 03Jan22/4 for details. + + These are now being scrubbed out of the users file and also will present + as their normalised selves. If a -0* SSID is encountered then, if the + normalised version of that call is not present, it will be renamed to + that normalised call. If the normalised version of that user record is + already present, the un-normalised user record (-0*) will be removed. +5. Make export_users do a batch clean (as in 3. above) and also get rid of + (default) 12+ year old unaccessed user records and (default) 2+ year old + "empty" records (with no qra/latlog/qth or handle). + + NOTE: if you do an manual export_users (as opposed to the automatic one + done once a week), do not be alarmed by the number of old (i.e. more than + 12 years old) callsigns that it will get rid of. In my case it was about + ~2/5th of the users file. Still left me with over 100,000 "active" users. + + In you are a bit twitchy about this, the code will copy the current + user_json and user_json.ooooo to user_json.keep and user_json.backstop + respectively. These files will never be overwritten unless you remove one + or both, when they will be regenerated on the next export_user. +04Jan22======================================================================= +1. Fix issue in the RBN (and probably other places) with callsigns that + contain trailing / in callsigns like: OH0K/6, K2PO/7 etc. +2. Regard strange callsigns like DR4W-HB (seen in skimmer spots) as invalid. + This *should be* something like HB9/DR4W or (spit) DR4W/HB9. +3. Fix the (probably) spurious locking out of users that are unknown to this + node, that come in from other nodes. These create new user records which + where then automatically locked. +03Jan22======================================================================= +1. Allow overrides (on modern versions of perl) with things in DXVars.pm, such + $clusterport. This is really only of use for people trying to run more than + one instance of DXSpider on the same machine. +2. Fix who command to make RBN connections as RBN and not USER. +3. Prevent other nodes claiming that $myalias or $mycall is a different type + (user or node) from changing our route table and thence the user type. +4. Normalise callsigns of incoming connections to G1TST if G1TST-0 or G1TST-00 + amd G1TST-2 if G1TST-02. There are 800+ instances of callsigns with extra + 0 characters in the SSID in my users file. Allow SSIDs up to 99. +02Jan22======================================================================= +1. Fix dx bug introduced to handle dx by ip for webclusters. +2. Remove _add_thingy dbg message from general view. +3. Increase sh/ann cache to 130 to cope with some client programs' profligate + need to fill their users' screens with needless erm... data. +01Jan22======================================================================= +1. Fixed missing ';' in spoof +30Dec21======================================================================= +1. Try to fix (nuke) tabs in comment field. +28Dec21======================================================================= +1. Fix issues with wwv & wcy updates into the cache. +2. Add a simple sh/announce cache to stop spawning when used (especially) in + a script. +27Dec21======================================================================= +1. Fix "kwalitee control" issue in /spider/cmd/dx.pl. +2. Make sh/wcy and sh/wwv cacheble for simple queries. +3. Fix set/nodxxxxxx etc. +22Dec21======================================================================= +1. Add the possibility to do "sender verify" that spots (and later on, other + things like announce etc) are coming from spotters that exist on the node + that the PC11/61 says it does. At the moment, this is advisory as many + users seem to be on web clusters and many of them do not send PC61 and also + don't update the node's routing tables that normal node usage would do. +2. Prioritise PC61 spots over PC11 spots wherever possible. Also promote PC11 + spots to PC61 where the spotter's IP is known. +3. Add a shell script "perlbrew-dxspider" that can be used for starting the + node when using perlbrew (i.e. more modern perls) on older distros such + as Centos 7/8. +4. Allow to "dx by ip . + This is added as a convenience measure for webcluster authors while waiting + for the internal websocket interface (or because they cannot use that). + This can only be used by suitably privileged users. + This likely to be a temporary measure and will be immediately withdrawn if + there is any sign of abuse. +11Dec21======================================================================= +1. Removed dependency on perl Git package. +2. Force code to change directory to $DXSPIDER_ROOT || '/spider'. Also force + git -C $root describe. +3. Change git commands AGAIN to work on old perls (which mojo won't work on, + but it makes my life easier). +4. Change version strings to include the perl version in use. +10Dec21======================================================================= +1. Fix the output of set/dxgrid, set/usstate and set/dxcq or set/dxitu to how + it was always supposed to be since 2003. +09Dec21======================================================================= +1. Moved isregistered to DXChannel for safety... +08Dec21======================================================================= +1. Get to the bottom of why ssid'd calls were using the base call's + credentials. This should now work as expected. +06Dec21======================================================================= +1. Improve console.pl scrolling. Split long lines (eg on announcements. +04Dec21======================================================================= +1. Fix illogicalities in USDB creations and make sure that O_CREAT on tie does + NOT encounter an existing file to barf about. Even though it shouldn't. + Thanks Howard WB3FFV. +2. Fixed a typo in show/registered that prevents a list of callsigns being + searched for. Got rid of some over complex code. Thanks Fabrizio iZ0UIN. +3. Fix long line wrapping in console.pl +03Dec21======================================================================= +1. Move motd and issue files to local_data if not already there. +30Nov21======================================================================= +1. Fix sh/dx with callsigns that have /p or VE/G1TLH in them. +2. Add unset/ak1a, unset/arcluster aliases and some minimal help for UNSET/ + SPIDER, NODE, ARCLUSTER, AKIA and also SET/USER. +26Nov21======================================================================= +1. *Really* change spot display format and sh/dx format "back the way they + were. But They won't stay that way for long!!!! There are four (yes, count + them) completely unused spaces at the end of the line!!!! +25Nov21======================================================================= +1. Restored spot format back to the way it was. +2. Added CTY-3127 prefixes +24Nov21======================================================================= +1. Make /spider/perl/create_sysop.pl work on a base git cloned tree. +2. Add /spider/local /spider/local_cmd /spider/local_data to git. +3. Make user supplied console width (set/width) sticky. +19Nov21======================================================================= +1. Add the ability of console.pl to resize and have dx spots also resize so + that the comment field's size will increase accordingly. Sh/dx amd sh/mydx + also work similarly. If you make the console width smaller or wider then + spots already on the screen will not resize. +2. For client programs that parse the output (why? use set/ve7cc to get more + easily machine parsed output), a new command "set/width " is + available. +18Nov21======================================================================= +1. Add a dx spot cache which will store 2 days worth (configurable) of spots. + This is then used by the vast majority of simple sh/dx queries without + using a sub process. A simple sh/dx is any sh/dx that does not have an + "expression" such as "sh/dx g1tlh". So "sh/dx", "sh/dx 200", "sh/mydx" will + always run in the main program via the cache. Certain client programs (no + names, no pack drill) will notice the difference immediately. It's also a + bit snappier. +2. set/badspotter or set/badnode will work for RBN input. +3. RBN input with invalid QRGs will be dropped with a nice friendly message + rather than a scary exception message about multiplication thrown by one + of the libraries that I use. +4. Added CTY-3126 prefixes. +5. Silently ignore self spotting below 1240000Khz (change + $Spot::minselfspot to 0 to disable or some other freq in Khz). NOTE this + will appear as a spot to the spotter's feed, but will not be passed on. +15Jul21======================================================================= +1. Fix long standing bug in crontabs that commands with " rather than ' around + strings could cause those commands not to work. +2. Add CTY 3117 prefixes. +15Mar21======================================================================= +1. Fix DXUtil::localdata so that it does actually prefer the latest version + of a file regardless of whether it is in /spider/data or /spider/localdata. +2. Add CTY-3105 prefixes. +05Mar21======================================================================= +1. Fix DXCron::spawn_cmd so that more than one spawn_cmd can be active at a + time. This affects many nodes where lots of regular spawn_cmds happen at + a time and this prevented crucial things like 'start_connect' from working! +2. Sort user lockout issues to cope with all likely scenarios - including + "phantom" SSID working. +3. Fix "close_gracefully" error messages which are useless, but benign. These + happen (mainly) when outgoing connects timeout. +24Feb21======================================================================= +1. modify RBN timing arrangements + + There are two new twiddle pots: + + $limbotime (default 5*60 secs) which allows potential spots to + hang around longer to allow $minqual *different* skimmers to spot + them. At which point they are emitted. This is to catch the "slow + burning" spots where a call is spotted by different skimmers but + separated in time by more than $dwelltime - in fact maybe by + minutes. + + $maxqual (default 9 skimmers) which short circuits the normal + hard maximum $quality (9 skimmer spots) and $dwelltime (10secs) + to allow a spot that has $maxqual *different* skimmers (usually + as the result of a burst from the RBN) to defeat $dwelltime and + be emitted the moment that condition is satisfied. There maybe + a better name for this. +21Feb21======================================================================= +1. add data section on 160m in bands.pl. +07Sep20======================================================================= +1, Change interface to watchdbg & grepdbg slightly so that multiple search + regexes are ANDed rather than ORed together. ORing is easily achieved + already by the usual regex pattern 'PATT..|PATT..|..' whereas ANDing could + not be done as easily without resorting lots of 'PATT.*PATH' things which + would not necessarily get what was wanted. +2. Make sure that the pc92 C record only contains nodes and users and not + other extranoeus things like skimmers... +15Aug20======================================================================= +1. Simplify the skimmer scoring mechanism. +13Aug20======================================================================= +1. Improve the (displayed) RBN frequency weighting the skimmers' frequencies + w.r.t majority view on each spot. Any skimmer that disagrees with a + gets docked a 'point' from their 'good' score (if any) and have a 'point' + added to their 'bad' score. This score is then taked into account during + the 'choosing the real frequency' process. If a skimmer agrees with the + majority the process is reversed. To see this in action: set/deb rbnskim +06Aug20======================================================================= +1, Add CTY-3013 Prefixes +2. Make RBN more efficient. Start the process of skimmer node performance + caching. Add minimum quality allowed (at 2), which will be overrideable. + The format of the rbn_cache has changed and so a full restart will occur. +3. Collect channel input/output stats. New command: show/data_stats to show + them. +4. Add local::lib qw{/spider/perl5lib} to store cpanm loaded modules just for + DXSpider. This is done so that updates needed by future changes can be done + as the sysop user and doesn't have to be done as root. This paves the way + for UPDATE.pl which will pull down new modules that it needs automatically. + When it's written, which will be soon. +29Jul20======================================================================= +1. Add show/rbn command that allows one to see who is online and configured + for RBN. See help sh/rbn for details. +2. Fixed issue with set/ve7cc mode adding unwanted newlines to output. +3. Attempted to make the QRG normalisation more "intelligent". This would be + sooooooooo much easier if skimmers were calibrated more accurately. +4. Added CTY-3012 prefixes. +23Jul20======================================================================= +1. fix Filtering so that it does less of what it was doing badly, thus + the perl interpreter to do more of the work. Which means you *should* be + able to define more or less any allowable operation for that type of + filter. Which is a complicated way of saying: you can now use brackets. +2. Regex expressions now appear as readable text in sh/filter. +3. Make the RBN status file JSON, rather than DD format. In future versions + it will also be VERSIONed to enable more features to be added + transparently. +4. Add show/rbn command to allow sysops to see who is using rbn. +11Jul20======================================================================= +1. Fix (embarrassing) gratuitous '#'s in bands.pl +2. IMPORTANT: certain PC92 strings can crash DXSpider if the nologchan debug + category is set (which reduces debug files to sizes that non-developers can + reasonably store (especially during big contests)), +10Jul20======================================================================= +1. Fix console.pl permissions problem when running as another user not in + debug mode. Console.pl WILL NOT WORK in debug mode unless it is running + as the sysop user. +08Jul20======================================================================= +1. "Finish" the RBN system :-) +2. This includes enabling the coarse selection of spot modes using set/wantrbn + with arguments like 'set/wantrbn cw beacon'. This limits your output to + just CW, BCN and DXF modes. +3. The RBN spot is now cached. With a following wind, this means that even a + node restart, done in a timely fashion (within a few minutes) will not + cause a "cache warmup" delay for users on a restart. +4. Added the "full fat" set/wantrbn command and aliased it to 'set/skimmer'. + I use both terms (whenever I remembered) in the help text. +5. Help text has been written. +6. The UPGRADE.mojo file has been tweeked to point out the users file format + change. +7. Merge in users.v3j to the mojo branch. +8. Add CTY-3011 prefixes. +07Jul20======================================================================= +1. Fix show/node command. +2. Fix show/cluster command to take into account the presence of skimmer nodes + which are a new category of thing which is neither a node nor a user. +06Jul20======================================================================= +1. Add RBN.mojo with information of the RBN capabilities of DXSpider. +05Jul20======================================================================= +1. Fix show/dxcc. +2. Add HAPROXY "real ip" type 1 handling for incoming connections. +04Jul20======================================================================= +1. Give console.pl (or dx) a good going over with a bog brush to *finally* + (cough) make it work correctly with a full 80 column window (and not just + to a width of 79 really). Also fix scrolling. +28Jun20======================================================================= +1. Merge mojo with users.v3j to remove all vestages of Storable from DXSpider + in an effort to make the whole storage thing more reliable (and also a + bit faster). The user file will be auto-upgraded on restart. This may take + up to 20 seconds on slower hardware (and maybe a bit longer on huge user + files). On my 180,000 odd users, on my hardware, it takes 4 seconds. +2. The DXQSL system storage is also upgraded, Please run + /spider/perl/create_dxqsl.pl in a spare shell. This will recreate the + dxqsl.v1j file. Run 'load/dxqsl' in the console to activate it. +17Jun20======================================================================= +1. Change the Spot file reading mechanism back to the default of using 'tac'. +08Jun20======================================================================= +1. Fix show/mydx (lack of) filtering bug. +2. Add qra locator to prefix_data.pl. +3. Add 4 digit qra square for spotted callsign if show/dxgrid is enabled. +4. Fix general filtering bug where erroneous input causes crashing. +03Jun20======================================================================= +1. Make sure that all possible regexes get passed across to the search engine. +2. Fix out of order logging on sh/log queries spanning more than one month. +3. Do not read backwards on sh/dx, reading forwards seems a 1/3 quicker. +4. Add , and shortcuts to sh/dx (e.g sh/dx on 2m). +02Jun20======================================================================= +1. Fix the small whoopsie in sh/dx. +01Jun20======================================================================= +1. Fix sh/dx iota and qra. This completes the conversion of sh/dx's limited + "old style" parsing to using Filter style parsing with something that + allows expressions include brackets ( and ) as well as the 'not' keyword. + NOTE: the precedence rules are the same as perl's '!', '||' and '&&' + operators. +31May20======================================================================= +1. Improve links command layout slightly. +2. Issue an *accurate* UPGRADE.mojo with all the new packages included! +3. Fix issue with sh/dx +30May20======================================================================= +1. Fix sh/dx! It appears that there is a long standing problem with sh/dx not + actually return all the answers it should. I have also had a request to + allow / enable the "not" keyword (just like the filtering system dones). + + Now this was one of the earliest modules that I wrote and it took quite + a bit of work to tease out the important bits and then use the Filter + module (as in acc/spot) to generate the sh/dx filtering expressions. This + seems to have been done. + + NB This will likely not work if you are using a SQL backend to do sh/dx + commands. This was never actually an official feature. It is now + deprecated. +29May20======================================================================= +1. Please install Math::Round and Data::Structure::Util. Instructions are + available in UPGRADE.mojo. +2. ** Withdrawn ** user file format change to json not yet committed. 20May20======================================================================= 1. Backport convert-users-v3-to-v4.pl to allow creation of the new json formatted *text* based user file from the old v3 DB_File and Storable