User Tools

Site Tools


operations:aum-hb

Some background

The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array.

The Hobart12 and Katherine antennas now have a functional wideband system, with reasonable sensitivity. As of January 2020, we are still observing using the same mode as in the AUM series. This uses the “VGOS”-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across to using that. As such, these instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment.

Major Differences

The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.

Before the Experiment

  • Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the snp and sum files and put them where they're needed.
cd /usr2/sched
wget ftp://cddis.gsfc.nasa.gov/vlbi/ivsdata/aux/2020/aua060/aua060.skd
drudg aua060.skd
   hb
   12
   3
   5
   0
scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/logs/aua060hb.sum

The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of January 2020), the aua060 experiment can be used as the template. e.g:

cd /usr2/proc/
cp aua060hb.prc aua061hb.prc 

* Terminate the FS and restart it with “fs-dummy” for both Hb and Ke. This ensures that the Field system will not connect to the DBBC2 and block the configuration scripts.

* From a terminal on the pcsf machine, run the command ./bin/aum.query. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is talking to them.

  1. First log in to DBBC3 (vncviewer dbbc3hb) and make sure that the DBBC3 Control DDC_V_v123 .exe program is running, and that the final line of output reads Waiting for connection).
  2. Next check the DBBC2 (vncviewer dbbcho for Hb and vncviewer dbbcke for Ke) and check that the correct control program is running (DBBC2 Control DDC v105E_2.exe for Hb, DBBC2 Control DDV v106E_120118.exe for Ke) and displays Waiting for connection)
  • If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait for the DBBC3 after answering “y” to the reconfiguration question).
  • If it does not read waiting for connection, it should give the IP address of the machine that's currently talking to it (e.g if it reads Command from 131.217.63.201: then the connection is from 131.217.63.201). You can lookup up the name of this machine using nslookup 131.217.63.201 - in this example it is the hobart machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running fs-auscope will avoid this problem while fs-flexbuff will block communications.

* Once you can run .bin/aum.query.py successfully, you should be fine to configure the backends with ./bin/aum.setup.py. NB - this script will force a synchronisation of the DBBCs and is equivalent to running setupsx and fmset in sequence. Do not run the setup script during the experiment!

* Set the log file for the experiment with log=aua060hb

* Load the schedule file with sched=aua060hb,#1

Revised checklist

Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. Most of the information is gathered by the ~/oper/bin/aum.query.py script. This can be called from the FS with sy=bin/aum.query.py with the outputs printed to screen and also logged to a file called e.g: /usr2/log/aua060hb.dbbc.log. It's strongly recommended to have a terminal watching this (tail -100f /usr2/log/aua060hb.dbbc.log) as it can be difficult to read from the FS display. An example of the output is given at the end of this page.

* RF and IF paths configured/DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on the pcfs machine called ~/aum.setup.py. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the DBBC2 but the script should not be run during an experiment to avoid unnecessary clock breaks.

  • Check that the dbbcif[abcd] input, filter and power target level readings in the aua060hb.dbbc.log are correct. For Hobart12, these should be
    dbbcifa=4,**11**,agc,2,**47680**, 48000
    dbbcifb=4,**27**,agc,2,**47843**,48000
    dbbcifc=2,32,agc,2,**40073**,40000
    dbbcifd=2,23,agc,2,**40070**,40000

and for Katherine:

dbbcifa=2,**38**,agc,2,**42008**,42000
dbbcifb=2,**37**,agc,2,**41669**,42000
dbbcifc=2,**29**,agc,2,**48178**,48000
dbbcifd=2,**23**,agc,2,**47928**,48000

NB - values with asterisks may vary depending on the input power level. Just make sure that the IF power readings is within ~1000 counts of the target.

  • DBBC server: If the aum.query.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions as described in the setup procedure. Again, please note that this will introduce a clock break so make sure that there's no other issue like another machine having connected to the DBBC server.
  • Maser ok: Check the Hobart26 maser as per usual
  • Mark5: Not used. Instead log in to the Flexbuff recorder (with ssh observer@flexbuffhb) and confirm that the time is correct (with ntpq -np) and that jive5ab is running (with ps -ef | grep jive)
  • The clock offsets can be (partially) read through the FS. maserdelay reads the maser-GPS difference as usual while dbbc3delay logs the DBBC3-GPS delay as gps-fmout for Katherine. dbbc2delay (at Katherine) will log the DBBC2-GPS difference. For Hobart12, the DBBC3-maser difference is being printed in a terminal on in the pcfshb VNC session - this should be ~0.1 microseconds. If the display is not present, or not updating try running cat /dev/ttyS0 as root on flexbuffhb.
  • Test Recordings: To make a test recording, simply run disk_record=on, disk_record=off and checkmk5 in the operator input as per usual.
  • Check for any integer second offsets in the recorded data by running ./bin/sxtimetest.sh on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries should agree to the integer second.
  • No autocorrelation script has been implemented yet.
  • There is no systemp measurewment implemented yet.
  • There is no pointing/SEFD check implemented as yet
  • No fringe check implemented as yet.
Monitoring
  • Antenna checks as per usual
  • LO check unavailable
  • No autocorrelations
  • Check the delays with maserdelay and dbbc3delay (Ke only).
  • Check the Hb and Ke masers as usual.
  • Check the weather data is being logged & add in sky conditions as usual.
  • No Tsys checks.
Additional log output

Most of the additional monitoring of the DBBC3 is currently logged to a separate log file (/usr2/log/aua060hb.dbbc.log), updated every scan when the sy=bin/aum.query.py | tee … script is run. This is part of the midob procedure. I've annotated the output below:

2020.028.12:07:03
###############################
Checking dbbc3hb
###############################
dbbcifc/ 2,32,agc,2,40019,40000;
dbbcifd/ 2,23,agc,2,39843,40000;

This first block is the equivalent of iread - check the inputs and filters are set to 2 and that the power level matches the target

dbbc017/ 2204.990000,a,32,1,agc,64,73,14921,14957,0,0;
dbbc018/ 2244.990000,a,32,1,agc,75,80,14837,14828,0,0;
dbbc019/ 2344.990000,a,32,1,agc,100,121,14883,14875,0,0;
dbbc020/ 2504.990000,a,32,1,agc,151,133,14822,14915,0,0;
dbbc021/ 2724.990000,a,32,1,agc,135,148,15015,14956,0,0;
dbbc022/ 2844.990000,a,32,1,agc,230,233,14936,14889,0,0;
dbbc023/ 2884.990000,a,32,1,agc,227,243,14899,14933,0,0;
dbbc024/ 2924.990000,a,32,1,agc,255,255,14777,14163,0,0;
dbbc025/ 2204.990000,a,32,1,agc,60,64,15830,14184,0,0;
dbbc026/ 2244.990000,a,32,1,agc,65,74,14735,15623,0,0;
dbbc027/ 2344.990000,a,32,1,agc,92,105,14711,15240,0,0;
dbbc028/ 2504.990000,a,32,1,agc,147,114,14965,15007,0,0;
dbbc029/ 2724.990000,a,32,1,agc,122,130,14927,15052,0,0;
dbbc030/ 2844.990000,a,32,1,agc,182,211,14940,15090,0,0;
dbbc031/ 2884.990000,a,32,1,agc,228,220,14919,14975,0,0;
dbbc032/ 2924.990000,a,32,1,agc,209,255,14828,14455,0,0;

This is the quivalent of bread confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.

pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 39 ns, [5] 43 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;

This is the translated equivalent of dbbc=pps_delay, separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable.

core3hstats/ 
Core3H[3] Power:
Sampler 0: 44362700
Sampler 1: 44956375
Sampler 2: 44344937
Sampler 3: 43982701

The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.

Core3H[3] Bstat.
Sampler 0:
11:       10 0.06%
10:     7822 50.06%
01:     7785 49.82%
00:        6 0.04%

Sampler 1:
11:       11 0.07%
10:     7771 49.73%
01:     7833 50.13%
00:        8 0.05%

Sampler 2:
11:       21 0.13%
10:     7986 51.11%
01:     7610 48.70%
00:        6 0.04%

Sampler 3:
11:        9 0.06%
10:     7757 49.64%
01:     7850 50.24%
00:        6 0.04%

The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.

Core3H[3] Corr.
Sampler 0-1: 183273215
Sampler 1-2: 186151539
Sampler 2-3: 183984793

All inter-sampler correlation values should be above ~170000000.

;
core3hstats/ 
Core3H[4] Power:
Sampler 0: 64799981
Sampler 1: 64621395
Sampler 2: 65964916
Sampler 3: 64445221

The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.

Core3H[4] Bstat.
Sampler 0:
11:       50 0.32%
10:     7707 49.32%
01:     7807 49.96%
00:       60 0.38%

Sampler 1:
11:       59 0.38%
10:     7778 49.78%
01:     7743 49.56%
00:       42 0.27%

Sampler 2:
11:       57 0.36%
10:     7804 49.95%
01:     7719 49.40%
00:       43 0.28%

Sampler 3:
11:       53 0.34%
10:     7639 48.89%
01:     7887 50.48%
00:       44 0.28%

The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.

Core3H[4] Corr.
Sampler 0-1: 187136057
Sampler 1-2: 190993079
Sampler 2-3: 186760591

All inter-sampler correlation values should be above ~170000000.

;
###############################
Checking dbbcho
###############################
dbbcifa/4,9,agc,2,47129,48000,1
dbbcifb/4,24,agc,2,48036,48000,1

This is the equivalent of iread. All inputs for dbbcho should be 1, all filters 2 and all target power levels 48000

dbbc01/300.990000,a,32,1,agc,72,136,17885,17699,0,0
dbbc03/332.990000,a,32,1,agc,63,63,17873,17835,0,0
dbbc05/364.990000,a,32,1,agc,53,62,17617,17971,0,0
dbbc07/348.990000,a,32,1,agc,57,61,18100,17633,0,0
dbbc09/300.990000,b,32,1,agc,56,166,17989,17571,0,0
dbbc11/332.990000,b,32,1,agc,63,54,18342,18083,0,0
dbbc13/364.990000,b,32,1,agc,91,64,18088,17794,0,0
dbbc15/348.990000,b,32,1,agc,88,55,17788,18248,0,0

This is the equivalent of bread. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)

pps_delay/61971

The pps_delay should be stable at 61971, to within 1 count of so.

fila10g/sysstat

System status:
 Selected input      : vsi1
 Input sample rate   : 64000000 Hz
 VSI input swapped   : no
 VSI input bitmask   : 0x0000FFFF
 VSI input width     : 16 bit
 PPS count           : 23451

 TVG mode            : vsi-h

 MK5B timesync       : yes
 VDIF timesync       : yes
 GPS receiver        : installed

 Output              : started
 Output 0 format     : vdif
 Output 0 dest.      : 192.168.1.13:46227
 Output 1 format     : vdif
 Output 1 dest.      : none
 Ethernet ARPs       : off

 Selected VSI output : vsi1-2

FiLa10G %

This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks.

/home/www/auscope/opswiki/data/pages/operations/aum-hb.txt · Last modified: 2020/01/28 12:20 by Jamie McCallum