This is an old revision of the document!
the AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array.
The Hobart12 antenna now has a functional wideband system, with reasonable sensitivities. The initial observations of the AUM series will be made using the “VGOS”-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. If the current issues with the DBBC3 and its “Legacy”-mode software are resolved, we sill change over to use this instead, so the instructions may be updated - please check the page carefully each time you are starting or monitoring an AUM experiment.
The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.
Taking the existing e-remote control checklists as a template, here is a listing of the significant differences.
cd /usr2/sched wget ftp://cddis.gsfc.nasa.gov/vlbi/ivsdata/aux/2018/aum001/aum001.skd drudg aum001.skd hb 12 3 5 0 scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/logs/aum001hb.sum
The next step is to convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands
cp aum001hb.snp aum001hb.orig.snp /usr2/sched/snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb
The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc. NB - do not use the line number entries in the sum file!
/usr2/sched/aum.32mhz.py
. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho. vncviewer newsmerd:1
) and go to desktop 3. Refresh the Folly display with 'r' and confirm that S/X SRCP and S/X XRCP are highlighted for channels 1 and 2 respectively. DBBC# Control DDC_V_v121.exe
) and dbbcho (DBBC2 Control DDC v105_2.exe
and SerialServer.sem.py
), respectively. aum.query.py
script on pcfshb. For the DBBC3, all inputs and filters should be 2
and all power levels should have a target level of 42000
. For the dbbcho, the input should be 1
, the filter 2
and the target power level should be 48000
ssh observer@flexbuffhb
) and confirm that the time is correct (with ntpq-np
) and that jive5ab is running (with ps -ef | grep jive
)maserdelay
reads the maser-GPS difference as usual while dbbc3delay
logs the DBBC3-GPS delay as gps-fmout. The existing clkoff
command is measuring the dbbchb-gps difference and logging this as fmout-gps - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the maserdelay
and dbbc3delay
commands, logged as /maser2gps/
and /gps-fmout/
respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled DBBCHO-TACGPS
and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference nd confirm that it is steady.oper@pcfshb
run sxflexbuff.sh test 19
to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g tail -100f /usr2/log/$(lognm).flexbuff.log
). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. sxtest.sh
on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as MJD = 58330/01:09:59.91
) should agree to within 0.01 seconds. maserdelay
and dbbc3delay
, confirm the dbbcho time delay is stable./aum.check.sh
to check the dbbc and recorder status. There is a lot of debugging info stored here, so use a large terminal window to run the script and be aware that you will need to press space to go through the list. Included below is the full output of a check, with the relevant parameters to check commented on below.############################### Checking dbbc3hb ############################### dbbcifc/ 2,33,agc,2,41815,42000 dbbcifd/ 2,22,agc,2,42416,42000
This first block is the equivalent of iread
- check the inputs and filters are set to 2
and that the power level matches the target
dbbc17/ 2204.990000,a,32,1,agc,55,63,14974,15206,0,0 dbbc18/ 2244.990000,a,32,1,agc,58,69,17035,15254,0,0 dbbc19/ 2344.990000,a,32,1,agc,79,100,17160,14828,0,0 dbbc20/ 2504.990000,a,32,1,agc,124,108,15562,14913,0,0 dbbc21/ 2724.990000,a,32,1,agc,107,121,16908,15844,0,0 dbbc22/ 2844.990000,a,32,1,agc,181,181,16086,16144,0,0 dbbc23/ 2904.990000,a,32,1,agc,188,196,16211,16191,0,0 dbbc24/ 2924.990000,a,32,1,agc,200,202,16298,16046,0,0 dbbc25/ 2204.990000,a,32,1,agc,57,58,13976,17561,0,0 dbbc26/ 2244.990000,a,32,1,agc,68,75,15413,16071,0,0 dbbc27/ 2344.990000,a,32,1,agc,90,92,15310,14988,0,0 dbbc28/ 2504.990000,a,32,1,agc,119,100,16736,16911,0,0 dbbc29/ 2724.990000,a,32,1,agc,104,109,16772,14937,0,0 dbbc30/ 2844.990000,a,32,1,agc,149,172,16204,15428,0,0 dbbc31/ 2904.990000,a,32,1,agc,183,175,15675,16114,0,0 dbbc32/ 2924.990000,a,32,1,agc,171,209,16013,16102,0,0
This is the quivalent of bread
confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.
pps_delay/ [1]: 43 ns, [2] 43 ns, [3] 39 ns, [4] 39 ns, [5] 39 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;
This is the translated equivalent of dbbc=pps_delay
, separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable.
core3hstats/ Core3H[3] Power: Sampler 0: 60781106 Sampler 1: 61694675 Sampler 2: 62199248 Sampler 3: 62160250
The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.
Core3H[3] Bstat. Sampler 0: 11: 44 0.28% 10: 7699 49.27% 01: 7846 50.21% 00: 34 0.22% Sampler 1: 11: 71 0.45% 10: 7915 50.66% 01: 7604 48.67% 00: 33 0.21% Sampler 2: 11: 48 0.31% 10: 7768 49.72% 01: 7773 49.75% 00: 33 0.21% Sampler 3: 11: 45 0.29% 10: 7700 49.28% 01: 7844 50.20% 00: 35 0.22%
The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.
Core3H[3] Corr. Sampler 0-1: 180656919 Sampler 1-2: 182717105 Sampler 2-3: 182062361
All inter-sampler correlation values should be ~180000000.
core3hstats/ Core3H[4] Power: Sampler 0: 88700574 Sampler 1: 90430690 Sampler 2: 92322148 Sampler 3: 93681776
The Core3H power readings from each sampler should all agree to within ~3-5%. This is the second sampler of two, and the above readings are ok.
Core3H[4] Bstat. Sampler 0: 11: 170 1.09% 10: 7635 48.86% 01: 7680 49.15% 00: 138 0.88% Sampler 1: 11: 226 1.45% 10: 7680 49.15% 01: 7582 48.52% 00: 135 0.86% Sampler 2: 11: 184 1.18% 10: 7638 48.88% 01: 7658 49.01% 00: 142 0.91% Sampler 3: 11: 181 1.16% 10: 7569 48.44% 01: 7733 49.49% 00: 140 0.90%
The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.
Core3H[4] Corr. Sampler 0-1: 186212284 Sampler 1-2: 190747558 Sampler 2-3: 187420379
All inter-sampler correlation values should be ~180000000.
############################### Checking dbbcho ############################### dbbcifa/1,4,agc,2,47097,48000,1 dbbcifb/1,14,agc,2,47991,48000,1
This is the equivalent of iread
. All inputs for dbbcho should be 1
, all filters 2
and all target power levels 48000
dbbc01/317.990000,a,32,1,agc,97,89,14803,14952,0,0 dbbc03/337.990000,a,32,1,agc,108,89,14945,15083,0,0 dbbc05/357.990000,a,32,1,agc,103,111,14891,14875,0,0 dbbc07/387.990000,a,32,1,agc,119,100,12784,14783,0,0 dbbc09/317.990000,b,32,1,agc,130,119,16077,16249,0,0 dbbc11/337.990000,b,32,1,agc,139,123,16091,16175,0,0 dbbc13/357.990000,b,32,1,agc,149,138,16248,16228,0,0 dbbc15/387.990000,b,32,1,agc,125,145,14144,16119,0,0
This is the equivalent of bread
. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)
pps_delay/61971
The pps_delay should be stable at 61971, to within 1 count of so.
sysstat ^MSystem status: ^M Selected input : vsi1 ^M Input sample rate : 64000000 Hz ^M VSI input swapped : no ^M VSI input bitmask : 0x0000FFFF ^M VSI input width : 16 bit ^M PPS count : 78607 ^M TVG mode : vsi-h ^M MK5B timesync : yes ^M VDIF timesync : yes ^M GPS receiver : installed ^M Output : started ^M Output 0 format : vdif ^M Output 0 dest. : 192.168.1.11:46236 ^M Output 1 format : vdif ^M Output 1 dest. : none ^M Ethernet ARPs : off ^M Selected VSI output : vsi1-2 ^MFiLa10G % ^MFiLa10G %
This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks.
/usr2/log/aum001hb.flexbuff.log
on pcfshb - the name will change when a new FS log file is started). This is written to by the recording script and running tail -100f /usr2/log/aum001hb.flexbuff.log
will show the same output as in the recording tests, e.g 50 !runtime = 0 : stream46230 ;!evlbi? 0 : total : 782143 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ; !runtime = 0 : stream46236 ;!evlbi? 0 : total : 782107 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ; !runtime = 0 : stream46229 ;!evlbi? 0 : total : 782064 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
Please note that the output is “laggy” and is usually delayed by ~1 minute. You can see the current status of the recorder using ptzhb - when recording you will see the blue lights of disk activity scattered over the flexbuff (below the DBBC in rack 13).