spider.git
2 years agofix typo in Changes cachespots
Dirk Koopman [Thu, 18 Nov 2021 10:29:44 +0000 (10:29 +0000)]
fix typo in Changes

2 years agofix self spotting by proxy (using by command)
Dirk Koopman [Wed, 17 Nov 2021 22:26:00 +0000 (22:26 +0000)]
fix self spotting by proxy (using by command)

Add ipaddr to error message

2 years agostop self spotting bwlow 23cms (configurable)
Dirk Koopman [Wed, 17 Nov 2021 22:18:10 +0000 (22:18 +0000)]
stop self spotting bwlow 23cms (configurable)

2 years agoadd cty3126, enable spot cache as default
Dirk Koopman [Wed, 17 Nov 2021 20:10:09 +0000 (20:10 +0000)]
add cty3126, enable spot cache as default

2 years agofix spot cache clean, some detail rbn changes
Dirk Koopman [Wed, 17 Nov 2021 11:41:16 +0000 (11:41 +0000)]
fix spot cache clean, some detail rbn changes

2 years agodon't fork sh/dx on simple query
Dirk Koopman [Mon, 15 Nov 2021 10:20:20 +0000 (10:20 +0000)]
don't fork sh/dx on simple query

If the query does not contain an "expression" (or even just a
callsign and is garanteed to be satified by the cache - if it is
enabled - then do not fork / run as a sub command.

So 'sh/dx', 'sh/mydx' with or without a count will be run from
the cache - if it is enabled.

2 years ago"working" version of spot cache
Dirk Koopman [Sun, 14 Nov 2021 22:49:49 +0000 (22:49 +0000)]
"working" version of spot cache

2 years agoexperimental branch to improve spot query
Dirk Koopman [Sat, 13 Nov 2021 23:08:22 +0000 (23:08 +0000)]
experimental branch to improve spot query

This includes some SQLite changes as well as spot searching

2 years agofix Dan's crontab problem, add CTY-3117
Dirk Koopman [Thu, 15 Jul 2021 18:25:13 +0000 (19:25 +0100)]
fix Dan's crontab problem, add CTY-3117

3 years agochang date in Changes!
Dirk Koopman [Mon, 15 Mar 2021 13:53:29 +0000 (13:53 +0000)]
chang date in Changes!

3 years agofix localdata(), add CTY 3105 prefixes
Dirk Koopman [Mon, 15 Mar 2021 13:50:24 +0000 (13:50 +0000)]
fix localdata(), add CTY 3105 prefixes

It appears that DXUtil::localdata had the mod time test the wrong
way around (how was this not noticed before???). That has been
fixed and the variable names make what is happening a bit more
obvious.

3 years agoremove active debugging from mds
Dirk Koopman [Sat, 6 Mar 2021 10:22:39 +0000 (10:22 +0000)]
remove active debugging from mds

3 years agofix DXCron, lockout and close_gracefully
Dirk Koopman [Fri, 5 Mar 2021 09:41:06 +0000 (09:41 +0000)]
fix DXCron, lockout and close_gracefully

1. Fix DXCron::spawn_cmd so that more than one spawn_cmd can be active at a
   time. This affects many nodes where lots of regular spawn_cmds happen at
   a time and this prevented crucial things like 'start_connect' from working!
2. Sort user lockout issues to cope with all likely scenarios - including
   "phantom" SSID working.
3. Fix "close_gracefully" error messages which are useless, but benign. These
   happen (mainly) when outgoing connects timeout.

3 years agomodify RBN timing arrangements
Dirk Koopman [Wed, 24 Feb 2021 09:18:01 +0000 (09:18 +0000)]
modify RBN timing arrangements

There are two new twiddle pots:

$limbotime (default 5*60 secs) which allows potential spots to
hang around longer to allow $minqual *different* skimmers to spot
them. At which point they are emitted. This is to catch the "slow
burning" spots where a call is spotted by different skimmers but
separated in time by more than $dwelltime - in fact maybe by
minutes.

$maxqual (default 9 skimmers) which short circuits the normal
hard maximum $quality (9 skimmer spots) and $dwelltime (10secs)
to allow a spot that has $maxqual *different* skimmers (usually
as the result of a burst from the RBN) to defeat $dwelltime and
be emitted the moment that condition is satisfied. There maybe
a better name for this.

3 years agoadd data section on 160m in bands.pl
Dirk Koopman [Mon, 22 Feb 2021 19:57:21 +0000 (19:57 +0000)]
add data section on 160m in bands.pl

3 years agouse IO::Socket::IP and fix null reply
Dirk Koopman [Mon, 14 Sep 2020 23:28:48 +0000 (01:28 +0200)]
use IO::Socket::IP and fix null reply

3 years agonominally working wsjtl with tcp listener
Dirk Koopman [Mon, 14 Sep 2020 23:19:13 +0000 (01:19 +0200)]
nominally working wsjtl with tcp listener

3 years agowsjtl WIP
Dirk Koopman [Mon, 14 Sep 2020 15:17:34 +0000 (17:17 +0200)]
wsjtl WIP

3 years agoget wsjtx to work
Dirk Koopman [Sun, 13 Sep 2020 13:10:27 +0000 (14:10 +0100)]
get wsjtx to work

3 years agoadd missing WSJTX.pm
Dirk Koopman [Sun, 13 Sep 2020 12:16:34 +0000 (13:16 +0100)]
add missing WSJTX.pm

3 years agoall dxdebug to have other directories, add wsjtl.pl
Dirk Koopman [Sun, 13 Sep 2020 12:11:00 +0000 (13:11 +0100)]
all dxdebug to have other directories, add wsjtl.pl

3 years agoWatchdbg, grepdbg args change, fix pc92c for rbn
Dirk Koopman [Mon, 7 Sep 2020 22:02:06 +0000 (23:02 +0100)]
Watchdbg, grepdbg args change, fix pc92c for rbn

As Changes says:
1, Change interface to watchdbg & grepdbg slightly so that multiple search
   regexes are ANDed rather than ORed together. ORing is easily achieved
   already by the usual regex pattern 'PATT..|PATT..|..' whereas ANDing could
   not be done as easily without resorting lots of 'PATT.*PATH' things which
   would not necessarily get what was wanted.
2. Make sure that the pc92 C record only contains nodes and users and not
   other extranoeus things like skimmers...

3 years agomake the RESPOTTING work again
Dirk Koopman [Mon, 17 Aug 2020 23:01:07 +0000 (00:01 +0100)]
make the RESPOTTING work again

Whoops. That explains the order of magnitude increase in spots!

3 years agowrite rbncache in raw json unless set/deb rbncache
Dirk Koopman [Mon, 17 Aug 2020 13:44:10 +0000 (14:44 +0100)]
write rbncache in raw json unless set/deb rbncache

3 years agosimply the RBN skimmer scoring system
Dirk Koopman [Sat, 15 Aug 2020 21:45:17 +0000 (22:45 +0100)]
simply the RBN skimmer scoring system

3 years agofix last fix!
Dirk Koopman [Fri, 14 Aug 2020 14:01:07 +0000 (15:01 +0100)]
fix last fix!

3 years agoremove $items warning in RBN
Dirk Koopman [Fri, 14 Aug 2020 13:59:36 +0000 (14:59 +0100)]
remove $items warning in RBN

3 years agofix missing fields in rbn_cache, add timings
Dirk Koopman [Fri, 14 Aug 2020 13:33:19 +0000 (14:33 +0100)]
fix missing fields in rbn_cache, add timings

set/debug rbntimer will show how long a queue takes to process (remember
that this includes sending data out to users).
set/debug rbnskim shows the process of determining the "one true frequency" of
a spot, by running a weighed "auction" of all the spotters's performance
in previous "auctions". A spotter's frequency "score" can be between 5 and -5
(good -> bad) and that is added to each version of the frequency that is
sent by the RBN. The frequency with the highest score is the one posted.
It is the 'QRGScore' in the "progress" spot message.

3 years agonew RBN weighted QRG scoring system.
Dirk Koopman [Thu, 13 Aug 2020 16:52:09 +0000 (17:52 +0100)]
new RBN weighted QRG scoring system.

3 years agoadd uptime cmd, add bells to sh/data
Dirk Koopman [Fri, 7 Aug 2020 15:20:28 +0000 (16:20 +0100)]
add uptime cmd, add bells to sh/data

3 years agoAdd latest RBN chnages and data stats
Dirk Koopman [Thu, 6 Aug 2020 19:13:55 +0000 (20:13 +0100)]
Add latest RBN chnages and data stats

3 years agoAdd CTY-3013 Prefixes
Dirk Koopman [Thu, 6 Aug 2020 18:07:42 +0000 (19:07 +0100)]
Add CTY-3013 Prefixes

3 years agofix mojo complaints in check_cache
Dirk Koopman [Wed, 29 Jul 2020 23:38:16 +0000 (00:38 +0100)]
fix mojo complaints in check_cache

3 years agorbn, ve7cc improvements and fixes
Dirk Koopman [Wed, 29 Jul 2020 22:46:38 +0000 (23:46 +0100)]
rbn, ve7cc improvements and fixes

1. Add show/rbn command that allows one to see who is online and configured
   for RBN. See help sh/rbn for details.
2. Fixed issue with set/ve7cc mode adding unwanted newlines to output.
3. Attempted to make the QRG normalisation more "intelligent". This would be
   sooooooooo much easier if skimmers were calibrated more accurately.
4. Added CTY-3012 prefixes.

3 years agoput sh/rbn into git!
Dirk Koopman [Thu, 23 Jul 2020 15:12:21 +0000 (16:12 +0100)]
put sh/rbn into git!

3 years agoadd sh/rbn command
Dirk Koopman [Thu, 23 Jul 2020 15:09:20 +0000 (16:09 +0100)]
add sh/rbn command

3 years agoFix Filtering, RBN changes
Dirk Koopman [Thu, 16 Jul 2020 22:07:28 +0000 (23:07 +0100)]
Fix Filtering, RBN changes

Filter has changed so that it is now a "simple" transliteration
into perl expressions and relies on perl to do what is required
It is quitely that some more work will be required for error
handling, but the I no longer attempt to do any bracket handling
and leave that up to perl. This means that brackets are fully
working (as opposed to NOT working at all).

Regexes are now robustly translated where necessary. In order to
do this I have ditched Data Dumper because it did not seem to cope
with the regex translations correctly. '\s' would seem have been
translated to '\\s' and not what the regex actually said. So I
have changed the file format to JSON. As this is now the 3rd or
4th JSON change I have made, I have created a "standard" encode and
decode in DXJSON.pm and retrofitted it to all the other places
which had their own purpose built one. DXJSON.pm is just a very
light shim over the standard one.

The RBN changes are:

1. don't short-circuit the zone iteration by a  filter firing.
2. make sure that filtering and not filtering work identically
3. Add constants to the caches records to make it more obvious
   what is going on.

3 years agominor debugging change
Dirk Koopman [Sat, 11 Jul 2020 17:11:48 +0000 (18:11 +0100)]
minor debugging change

3 years agostop crash if node adds itself in pc92
Dirk Koopman [Sat, 11 Jul 2020 14:10:16 +0000 (15:10 +0100)]
stop crash if node adds itself in pc92

3 years agoremove '#' in bands.pl !!!!!!
Dirk Koopman [Sat, 11 Jul 2020 10:23:50 +0000 (11:23 +0100)]
remove '#' in bands.pl !!!!!!

3 years agoa more robust console.pl idle loop fix
Dirk Koopman [Sat, 11 Jul 2020 02:07:44 +0000 (03:07 +0100)]
a more robust console.pl idle loop fix

make all the comparisons 'eq' and check that RESIZE still works

3 years agofix spurious == errors on console.pl input
Dirk Koopman [Sat, 11 Jul 2020 01:56:04 +0000 (02:56 +0100)]
fix spurious == errors on console.pl input

This only happens if it catches a character during the idle loop
which is the only way to catch Curses RESIZE signal. But a stray
"normal" character might get through as well.

3 years agostop dbg init when not in debug mode.
Dirk Koopman [Fri, 10 Jul 2020 22:20:49 +0000 (23:20 +0100)]
stop dbg init when not in debug mode.

This fixes problems reported when running console.pl not as the
sysop user.

3 years agokill some 5.28 warnings
Dirk Koopman [Thu, 9 Jul 2020 22:24:28 +0000 (23:24 +0100)]
kill some 5.28 warnings

3 years agoadd --all to the git pill in UPGRADE.mojo
Dirk Koopman [Thu, 9 Jul 2020 21:29:22 +0000 (22:29 +0100)]
add --all to the git pill in UPGRADE.mojo

3 years agoUpdate the docs for the latest version
Dirk Koopman [Thu, 9 Jul 2020 21:19:41 +0000 (22:19 +0100)]
Update the docs for the latest version

* major change: convert the users file to Json.
* NEW: add an RBN client.

3 years agoMerge branch 'users.v3j' into mojo
Dirk Koopman [Thu, 9 Jul 2020 20:10:07 +0000 (21:10 +0100)]
Merge branch 'users.v3j' into mojo

3 years agoadd warning about filters and set/skimmer users.v3j
Dirk Koopman [Wed, 8 Jul 2020 23:55:43 +0000 (00:55 +0100)]
add warning about filters and set/skimmer

3 years agofix help for priv 9 set/skimmer
Dirk Koopman [Wed, 8 Jul 2020 23:42:47 +0000 (00:42 +0100)]
fix help for priv 9 set/skimmer

3 years agorm rbn from Aliases, change help
Dirk Koopman [Wed, 8 Jul 2020 22:58:42 +0000 (23:58 +0100)]
rm rbn from Aliases, change help

Add a callsign to invalid characters in command handling

3 years agoThe last revision before merge back to mojo?
Dirk Koopman [Wed, 8 Jul 2020 22:01:00 +0000 (23:01 +0100)]
The last revision before merge back to mojo?

See Changes file for details

3 years agofix (bandaid for now) sh/cl local counts
Dirk Koopman [Tue, 7 Jul 2020 17:56:52 +0000 (18:56 +0100)]
fix (bandaid for now) sh/cl local counts

3 years agofix (bandaid for now) sh/cl local counts
Dirk Koopman [Tue, 7 Jul 2020 17:56:52 +0000 (18:56 +0100)]
fix (bandaid for now) sh/cl local counts

3 years agofix show/node
Dirk Koopman [Tue, 7 Jul 2020 17:25:31 +0000 (18:25 +0100)]
fix show/node

3 years agobackport console.pl from users.v3j
Dirk Koopman [Mon, 6 Jul 2020 16:20:32 +0000 (17:20 +0100)]
backport console.pl from users.v3j

3 years agomore RBN.mojo changes
Dirk Koopman [Mon, 6 Jul 2020 16:11:13 +0000 (17:11 +0100)]
more RBN.mojo changes

3 years agomore RBN.mojo changes
Dirk Koopman [Mon, 6 Jul 2020 15:23:28 +0000 (16:23 +0100)]
more RBN.mojo changes

3 years agoupdate RBN.mojo
Dirk Koopman [Mon, 6 Jul 2020 15:11:54 +0000 (16:11 +0100)]
update RBN.mojo

3 years agoadd current node to console.pl
Dirk Koopman [Mon, 6 Jul 2020 14:32:29 +0000 (15:32 +0100)]
add current node to console.pl

3 years agoadd RBN.mojo
Dirk Koopman [Mon, 6 Jul 2020 01:02:45 +0000 (02:02 +0100)]
add RBN.mojo

3 years agoattempt to prevent echoing on ExtMsg login
Dirk Koopman [Sun, 5 Jul 2020 17:23:32 +0000 (18:23 +0100)]
attempt to prevent echoing on ExtMsg login

This is to try to prevent any PROXY message being echoed

3 years agoAdd HAProxy type 1 "real ip" address changes
Dirk Koopman [Sun, 5 Jul 2020 15:21:43 +0000 (16:21 +0100)]
Add HAProxy type 1 "real ip" address changes

3 years agofix show/dxcc
Dirk Koopman [Sun, 5 Jul 2020 00:52:19 +0000 (01:52 +0100)]
fix show/dxcc

3 years agofix show/dxcc
Dirk Koopman [Sun, 5 Jul 2020 00:52:19 +0000 (01:52 +0100)]
fix show/dxcc

3 years agofix console.pl
Dirk Koopman [Sun, 5 Jul 2020 00:27:02 +0000 (01:27 +0100)]
fix console.pl

Including removing the 80-1 limit on local connections, fixing the
scrolling and generally tarting console.pl up.
Change dx spot RBN format to "traditional" '-#' format.

3 years agofix longstanding issues with console EOL handling
Dirk Koopman [Sat, 4 Jul 2020 01:03:29 +0000 (02:03 +0100)]
fix longstanding issues with console EOL handling

Historically there has been an issue with line endings in console.pl
which meant that the spot line (for instance) had to be 1 character
less than the window width (e.g. 79 chars in an 80 char window).

This has now FINALLY been fixed.

In the process I have fixed the issues of extra things like
cq/itu/us state/dxgrid strings overwriting or coming out in a
stupid priority. So the priority is now from nearest/smallest
area -> largest wiz: QRA, US State, ITU or CQ zone. So you
can have them all on, but ITU zone will always trump CQ zone.

3 years agoimprove dbg tagging and dbgdump, Add call to conns
Dirk Koopman [Fri, 3 Jul 2020 23:20:41 +0000 (00:20 +0100)]
improve dbg tagging and dbgdump, Add call to conns

It would appear that the $conn did not always have a callsign set
on incoming connections. It does now.

3 years agoadd RBN stats + other fixes
Dirk Koopman [Tue, 30 Jun 2020 12:33:33 +0000 (13:33 +0100)]
add RBN stats + other fixes

Add some more comprehensive RBN stats (per minute, 10 minutes, hour).
Make the RBN spot cache global and make the inrush preventor code
work only on the first connection to skimmer provider.
Fix the $_isdbg clearing so that non-conditional logging messages
come out with a (*) in watchdbg and not the previous isdbg() sort.

3 years agoMerge branch 'mojo' into users.v3j
Dirk Koopman [Sun, 28 Jun 2020 14:14:44 +0000 (15:14 +0100)]
Merge branch 'mojo' into users.v3j

Also convert QSL.pm and create_qsl.pl to JSON format.

3 years agofix ipv4 addresses masquerading as ipv6
Dirk Koopman [Sun, 28 Jun 2020 12:01:54 +0000 (13:01 +0100)]
fix ipv4 addresses masquerading as ipv6

3 years agovarious RBN fixes, fix links spacing
Dirk Koopman [Thu, 25 Jun 2020 14:00:34 +0000 (15:00 +0100)]
various RBN fixes, fix links spacing

Add incoming RBN filtering from the RBN servers.
Use spot filtering if there is no RBN specific filtering.
Fix formatting issues in the 'links' command for isolated and skimmer nodes.

3 years agouse spot filters for rbn in no rbn filters exist
Dirk Koopman [Tue, 23 Jun 2020 22:32:45 +0000 (23:32 +0100)]
use spot filters for rbn in no rbn filters exist

3 years agomake sure wantrbn is picked up again on restart
Dirk Koopman [Mon, 22 Jun 2020 18:21:33 +0000 (19:21 +0100)]
make sure wantrbn is picked up again on restart

remove words about convert-v3-to-v4.pl from UPDATE.mojo

3 years agofix new RBN spot typos
Dirk Koopman [Sun, 21 Jun 2020 14:05:55 +0000 (15:05 +0100)]
fix new RBN spot typos

3 years agoRBN do filtering one per stored spot not per user
Dirk Koopman [Sat, 20 Jun 2020 22:27:55 +0000 (23:27 +0100)]
RBN do filtering one per stored spot not per user

3 years agoadd RBN filtering
Dirk Koopman [Sat, 20 Jun 2020 22:04:10 +0000 (23:04 +0100)]
add RBN filtering

3 years agoSet $Spot::readback back to 1
Dirk Koopman [Wed, 17 Jun 2020 11:59:03 +0000 (12:59 +0100)]
Set $Spot::readback back to 1

This is to solve the problem of losing found spots when traversing
multiple files.

TO BE FIXED!! So we can read forward again.

3 years agofix respotted spots
Dirk Koopman [Thu, 11 Jun 2020 14:26:18 +0000 (15:26 +0100)]
fix respotted spots

3 years agorbn collect qra info ftom FTx spots
Dirk Koopman [Thu, 11 Jun 2020 14:12:25 +0000 (15:12 +0100)]
rbn collect qra info ftom FTx spots

3 years agoincrease rbn connection startup delay
Dirk Koopman [Thu, 11 Jun 2020 13:08:50 +0000 (14:08 +0100)]
increase rbn connection startup delay

3 years agomake rbn queues local, improve qrg jitter detect
Dirk Koopman [Thu, 11 Jun 2020 12:45:19 +0000 (13:45 +0100)]
make rbn queues local, improve qrg jitter detect

3 years agoRBN "fix" qrg variation, add extra flags
Dirk Koopman [Thu, 11 Jun 2020 10:34:01 +0000 (11:34 +0100)]
RBN "fix" qrg variation, add extra flags

* = multiple QRGs
+ = respot

3 years agoadd missing set/rbn.pl
Dirk Koopman [Thu, 11 Jun 2020 00:20:55 +0000 (01:20 +0100)]
add missing set/rbn.pl

3 years agoLatest version of RBN code
Dirk Koopman [Thu, 11 Jun 2020 00:18:03 +0000 (01:18 +0100)]
Latest version of RBN code

3 years agolengthen trailing numeric part of prefix
Dirk Koopman [Mon, 8 Jun 2020 16:38:43 +0000 (17:38 +0100)]
lengthen trailing numeric part of prefix

to allow things like GB5000GCM etc

3 years agoadd better errors in DXUser::Exportusers
Dirk Koopman [Mon, 8 Jun 2020 16:40:17 +0000 (17:40 +0100)]
add better errors in DXUser::Exportusers

3 years agolengthen trailing numeric part of prefix
Dirk Koopman [Mon, 8 Jun 2020 16:38:43 +0000 (17:38 +0100)]
lengthen trailing numeric part of prefix

to allow things like GB5000GCM etc

3 years agoMerge branch 'mojo' into users.v3j
Dirk Koopman [Mon, 8 Jun 2020 16:19:11 +0000 (17:19 +0100)]
Merge branch 'mojo' into users.v3j

just to keep stuff ticking along

3 years agofix erronoeus input filter cmd causing crashes
Dirk Koopman [Mon, 8 Jun 2020 15:19:09 +0000 (16:19 +0100)]
fix erronoeus input filter cmd causing crashes

3 years agofix sh/mydx, add back qra sq for sh/dxgrid
Dirk Koopman [Mon, 8 Jun 2020 13:49:27 +0000 (14:49 +0100)]
fix sh/mydx, add back qra sq for sh/dxgrid

3 years agofix beacons on RBN
Dirk Koopman [Sun, 7 Jun 2020 01:04:07 +0000 (02:04 +0100)]
fix beacons on RBN

3 years agoput back startup_delay to RBN
Dirk Koopman [Sun, 7 Jun 2020 00:53:00 +0000 (01:53 +0100)]
put back startup_delay to RBN

3 years agofix create_prefix.pl
Dirk Koopman [Sun, 7 Jun 2020 00:49:59 +0000 (01:49 +0100)]
fix create_prefix.pl

3 years agolatest version of RBN. Add qra to prefixes.
Dirk Koopman [Sun, 7 Jun 2020 00:18:27 +0000 (01:18 +0100)]
latest version of RBN. Add qra to prefixes.

Also fix small issue with reading a line from an sql file and
include the ip address.

3 years agoadd <tr>,<es> and <ms> shortcuts to sh/dx
Dirk Koopman [Wed, 3 Jun 2020 23:14:47 +0000 (00:14 +0100)]
add <tr>,<es> and <ms> shortcuts to sh/dx

3 years agoget rid of sh/dx debugging warnings
Dirk Koopman [Wed, 3 Jun 2020 22:56:24 +0000 (23:56 +0100)]
get rid of sh/dx debugging warnings

3 years agofix log and sh/dx ordering issues.
Dirk Koopman [Wed, 3 Jun 2020 22:38:27 +0000 (23:38 +0100)]
fix log and sh/dx ordering issues.

3 years agoadd chenges to sh/dx
Dirk Koopman [Wed, 3 Jun 2020 21:08:30 +0000 (22:08 +0100)]
add chenges to sh/dx

3 years agofix regex parse in sh/dx, add DXBearing::distance
Dirk Koopman [Wed, 3 Jun 2020 19:42:27 +0000 (20:42 +0100)]
fix regex parse in sh/dx, add DXBearing::distance

3 years agofix the small whoopsie in sh/dx
Dirk Koopman [Tue, 2 Jun 2020 20:42:02 +0000 (21:42 +0100)]
fix the small whoopsie in sh/dx