User Tools

Site Tools


operations:aum-hb

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
operations:aum-hb [2018/07/30 06:25]
Jamie McCallum [Revised checklist]
operations:aum-hb [2020/01/28 12:20] (current)
Jamie McCallum
Line 1: Line 1:
 ====== Some background ====== ====== Some background ======
  
-the AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​+The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​
  
-The Hobart12 ​antenna ​now has a functional wideband system, with reasonable ​sensitivitiesThe initial observations ​of the AUM series ​will be made using the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of  32 MHz which will be dealt with in the correlation stage. ​If the current issues with the DBBC3 and its "​Legacy"​-mode software are resolved, we sill change over to use this insteadso the instructions may be updated - please check the page carefully each time you are starting or monitoring an AUM experiment. ​+The Hobart12 ​and Katherine antennas ​now have a functional wideband system, with reasonable ​sensitivityAs of January 2020, we are still observing using the same mode as in the AUM series. This uses the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. ​Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across ​to using that. As suchthese instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment. ​
  
 ===== Major Differences ===== ===== Major Differences =====
  
-The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.+The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.
  
-==== Revised checklist ==== 
- 
-Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. 
  
 === Before the Experiment === === Before the Experiment ===
  
-  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the prc, snp and sum files and put them where they'​re needed.+  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the snp and sum files and put them where they'​re needed.
  
 <​code>​cd /usr2/sched <​code>​cd /usr2/sched
-wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2018/aum001/aum001.skd +wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2020/aua060/aua060.skd 
-drudg aum001.skd+drudg aua060.skd
    hb    hb
    12    12
Line 25: Line 22:
    5    5
    0    0
-scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aum001hb.sum+scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aua060hb.sum
 </​code>​ </​code>​
     ​     ​
-The next step is to convert ​the existing snp file into one which will control the flexbuff ​recordings appropriately. This is done via a script at present - please ​use the following commands ​<​code>​cp aum001hb.snp aum001hb.orig.snp +The next step is to create ​the procedure ​file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of January 2020), the aua060 experiment can be used as  the template. e.g: 
-/usr2/sched/snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb<\code>+ 
 +<​code>​ 
 +cd /​usr2/​proc/​ 
 +cp aua060hb.prc aua061hb.prc  
 +</​code>​ 
 + 
 +* Terminate the FS and restart it with "​fs-dummy"​ for both Hb and Ke. This ensures that the Field system ​will not connect to the DBBC2 and block the configuration scripts.  
 + 
 +* From a terminal on the pcsf machine, run the command ''​./​bin/​aum.query''​. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control ​programs are not running, or that something else is  talking to them.  
 +  - First log in to DBBC3 (''​vncviewer dbbc3hb''​) and make sure that the ''​DBBC3 Control DDC_V_v123 .exe''​ program is running, and that the final line of output reads ''​Waiting for connection''​).  
 +  - Next check the DBBC2 (''​vncviewer dbbcho''​ for Hb and ''​vncviewer dbbcke''​ for Ke) and check that the correct control program is running (''​DBBC2 Control DDC v105E_2.exe''​ for Hb, ''​DBBC2 Control DDV v106E_120118.exe''​ for Ke) and displays ''​Waiting for connection''​) 
 +  * If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait  for the DBBC3 after answering "​y"​ to the reconfiguration question).  
 +  * If it does not read ''​waiting for connection'',​ it should give the IP address of the machine that's currently talking to it (e.g if it reads ''​Command from 131.217.63.201:''​ then the connection is from 131.217.63.201). You can lookup up the name of this machine using ''​nslookup 131.217.63.201''​ - in this example it is the ''​hobart''​ machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running ''​fs-auscope''​ will avoid this problem while ''​fs-flexbuff''​ will block communications.  
 + 
 +* Once you can run ''​.bin/​aum.query.py''​ successfully,​ you should be fine to configure the backends with ''​./​bin/​aum.setup.py''​. NB - this script will force a synchronisation of the DBBCs and is equivalent to running ''​setupsx''​ and ''​fmset''​ in sequence. Do not run the setup script during the experiment! 
 + 
 +* Set the log file for the experiment with ''​log=aua060hb''​ 
 + 
 +* Load the schedule file with ''​sched=aua060hb,#​1''​ 
 + 
 +==== Revised checklist ==== 
 + 
 +Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. Most of the information is gathered by the ''​~/​oper/​bin/​aum.query.py''​ script. This can be called from the FS with ''​sy=bin/​aum.query.py''​ with the outputs printed to screen and also logged to a file called e.g: ''/​usr2/​log/​aua060hb.dbbc.log''​. It's strongly recommended to have a terminal watching this (''​tail -100f /​usr2/​log/​aua060hb.dbbc.log''​) as it can be difficult to read from the FS display. An example of the output ​is given at the end of this page.  
 + 
 + * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into single ​script ​on the pcfs machine called ''​~/​aum.setup.py''​. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the DBBC2 but the script should not be run during an experiment to avoid unnecessary clock breaks. ​   
 +    * Check that the dbbcif[abcd] input, filter and power target level readings in the ''​aua060hb.dbbc.log''​ are correct. For Hobart12, these should be <​code>​ 
 +dbbcifa=4,​**11**,​agc,​2,​**47680**,​ 48000 
 +dbbcifb=4,​**27**,​agc,​2,​**47843**,​48000 
 +dbbcifc=2,​32,​agc,​2,​**40073**,​40000 
 +dbbcifd=2,​23,​agc,​2,​**40070**,​40000 
 +</​code>​  
 + 
 +and for Katherine:​ 
 +<​code>​ 
 +dbbcifa=2,​**38**,​agc,​2,​**42008**,​42000 
 +dbbcifb=2,​**37**,​agc,​2,​**41669**,​42000 
 +dbbcifc=2,​**29**,​agc,​2,​**48178**,​48000 
 +dbbcifd=2,​**23**,​agc,​2,​**47928**,​48000 
 +</​code>​  
 + 
 +NB - values with asterisks may vary depending on the input power level. Just make sure that the IF power readings is within ~1000 counts of the target.  
 +    * DBBC server: If the aum.query.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions as described in the setup procedure. Again, please note that this will introduce a clock break so make sure that there'​s no other issue like another machine having connected to the DBBC server.  
 +  * Maser ok: Check the Hobart26 maser as per usual 
 +  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq -np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​) 
 +  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout// for Katherine. dbbc2delay (at Katherine) will log the DBBC2-GPS difference. For Hobart12, the DBBC3-maser difference is being printed in a terminal on in the pcfshb VNC session - this should be ~0.1 microseconds. If the display is not present, or not updating try running ''​cat /​dev/​ttyS0''​ as root on flexbuffhb. ​  
 + 
 + 
 + 
 +  * Test Recordings: To make a test recording, simply run ''​disk_record=on'',​ ''​disk_record=off''​ and ''​checkmk5''​ in the operator input as per usual.  
 +  * Check for any integer second offsets in the recorded data by running ''​./​bin/​sxtimetest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files all three entries should agree to the integer second. 
 +  * No autocorrelation script has been implemented yet.  
 +  * There is no systemp measurewment implemented yet.  
 +  * There is no pointing/​SEFD check implemented as yet 
 +  * No fringe check implemented as yet. 
 + 
 +== Monitoring == 
 +  * Antenna checks as per usual 
 +  * LO check unavailable 
 +  * No autocorrelations 
 +  * Check the delays with ''​maserdelay''​ and ''​dbbc3delay''​ (Ke only). 
 +  * Check the Hb and Ke masers as usual.  
 +  * Check the weather data is being logged & add in sky conditions as usual.  
 +  * No Tsys checks. 
 + 
 +== Additional log output == 
 + 
 +Most of the additional monitoring of the DBBC3 is currently logged to a separate log file (''/​usr2/​log/​aua060hb.dbbc.log''​),​ updated every scan when the ''​sy=bin/​aum.query.py | tee ...''​ script is run. This is part of the ''​midob''​ procedure. I've annotated the output below: 
 + 
 +<​code>​ 
 +2020.028.12:​07:​03 
 +###############################​ 
 +Checking dbbc3hb 
 +###############################​ 
 +dbbcifc/ 2,​32,​agc,​2,​40019,​40000;​ 
 +dbbcifd/ 2,​23,​agc,​2,​39843,​40000;​ 
 +</​code>​ 
 +This first block is the equivalent of ''​iread''​ - check the inputs and filters are set to ''​2''​ and that the power level matches the target 
 +<​code>​ 
 +dbbc017/ 2204.990000,​a,​32,​1,​agc,​64,​73,​14921,​14957,​0,​0;​ 
 +dbbc018/ 2244.990000,​a,​32,​1,​agc,​75,​80,​14837,​14828,​0,​0;​ 
 +dbbc019/ 2344.990000,​a,​32,​1,​agc,​100,​121,​14883,​14875,​0,​0;​ 
 +dbbc020/ 2504.990000,​a,​32,​1,​agc,​151,​133,​14822,​14915,​0,​0;​ 
 +dbbc021/ 2724.990000,​a,​32,​1,​agc,​135,​148,​15015,​14956,​0,​0;​ 
 +dbbc022/ 2844.990000,​a,​32,​1,​agc,​230,​233,​14936,​14889,​0,​0;​ 
 +dbbc023/ 2884.990000,​a,​32,​1,​agc,​227,​243,​14899,​14933,​0,​0;​ 
 +dbbc024/ 2924.990000,​a,​32,​1,​agc,​255,​255,​14777,​14163,​0,​0;​ 
 +dbbc025/ 2204.990000,​a,​32,​1,​agc,​60,​64,​15830,​14184,​0,​0;​ 
 +dbbc026/ 2244.990000,​a,​32,​1,​agc,​65,​74,​14735,​15623,​0,​0;​ 
 +dbbc027/ 2344.990000,​a,​32,​1,​agc,​92,​105,​14711,​15240,​0,​0;​ 
 +dbbc028/ 2504.990000,​a,​32,​1,​agc,​147,​114,​14965,​15007,​0,​0;​ 
 +dbbc029/ 2724.990000,​a,​32,​1,​agc,​122,​130,​14927,​15052,​0,​0;​ 
 +dbbc030/ 2844.990000,​a,​32,​1,​agc,​182,​211,​14940,​15090,​0,​0;​ 
 +dbbc031/ 2884.990000,​a,​32,​1,​agc,​228,​220,​14919,​14975,​0,​0;​ 
 +dbbc032/ 2924.990000,​a,​32,​1,​agc,​209,​255,​14828,​14455,​0,​0;​ 
 +</​code>​ 
 +This is the quivalent of ''​bread''​ confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above. 
 +<​code>​ 
 +pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 39 ns, [5] 43 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns; 
 +</​code>​ 
 +This is the translated equivalent of ''​dbbc=pps_delay'',​ separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable.  
 +<​code>​ 
 +core3hstats/​  
 +Core3H[3] Power: 
 +Sampler 0: 44362700 
 +Sampler 1: 44956375 
 +Sampler 2: 44344937 
 +Sampler 3: 43982701 
 +</​code>​ 
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.  
 +<​code>​ 
 +Core3H[3] Bstat. 
 +Sampler 0: 
 +11:       10 0.06% 
 +10:     7822 50.06% 
 +01:     7785 49.82% 
 +00:        6 0.04% 
 + 
 +Sampler 1: 
 +11:       11 0.07% 
 +10:     7771 49.73% 
 +01:     7833 50.13% 
 +00:        8 0.05% 
 + 
 +Sampler 2: 
 +11:       21 0.13% 
 +10:     7986 51.11% 
 +01:     7610 48.70% 
 +00:        6 0.04% 
 + 
 +Sampler 3: 
 +11:        9 0.06% 
 +10:     7757 49.64% 
 +01:     7850 50.24% 
 +00:        6 0.04% 
 +</​code>​ 
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them.  
 +<​code>​ 
 +Core3H[3] Corr. 
 +Sampler 0-1: 183273215 
 +Sampler 1-2: 186151539 
 +Sampler 2-3: 183984793 
 +</​code>​ 
 +All inter-sampler correlation values should be above ~170000000.  
 +<​code>​ 
 + 
 +
 +core3hstats/​  
 +Core3H[4] Power: 
 +Sampler 0: 64799981 
 +Sampler 1: 64621395 
 +Sampler 2: 65964916 
 +Sampler 3: 64445221 
 +</​code>​ 
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.  
 +<​code>​ 
 +Core3H[4] Bstat. 
 +Sampler 0: 
 +11:       50 0.32% 
 +10:     7707 49.32% 
 +01:     7807 49.96% 
 +00:       60 0.38% 
 + 
 +Sampler 1: 
 +11:       59 0.38% 
 +10:     7778 49.78% 
 +01:     7743 49.56% 
 +00:       42 0.27% 
 + 
 +Sampler 2: 
 +11:       57 0.36% 
 +10:     7804 49.95% 
 +01:     7719 49.40% 
 +00:       43 0.28% 
 + 
 +Sampler 3: 
 +11:       53 0.34% 
 +10:     7639 48.89% 
 +01:     7887 50.48% 
 +00:       44 0.28% 
 +</code> 
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them.  
 +<​code>​ 
 +Core3H[4] Corr. 
 +Sampler 0-1: 187136057 
 +Sampler 1-2: 190993079 
 +Sampler 2-3: 186760591 
 +</code> 
 +All inter-sampler correlation values should be above ~170000000.  
 +<​code>​ 
 + 
 +
 +###############################​ 
 +Checking dbbcho 
 +###############################​ 
 +dbbcifa/4,​9,​agc,​2,​47129,​48000,​1 
 +dbbcifb/​4,​24,​agc,​2,​48036,​48000,​1 
 +</​code>​ 
 +This is the equivalent of ''​iread''​All inputs for dbbcho should be ''​1'',​ all filters ''​2''​ and all target power levels ''​48000''​ 
 +<​code>​ 
 +dbbc01/300.990000,​a,​32,​1,​agc,​72,​136,​17885,​17699,​0,​0 
 +dbbc03/332.990000,​a,​32,​1,​agc,​63,​63,​17873,​17835,​0,​0 
 +dbbc05/364.990000,​a,​32,​1,​agc,​53,​62,​17617,​17971,​0,​0 
 +dbbc07/​348.990000,​a,​32,​1,​agc,​57,​61,​18100,​17633,​0,​0 
 +dbbc09/​300.990000,​b,​32,​1,​agc,​56,​166,​17989,​17571,​0,​0 
 +dbbc11/​332.990000,​b,​32,​1,​agc,​63,​54,​18342,​18083,​0,​0 
 +dbbc13/​364.990000,​b,​32,​1,​agc,​91,​64,​18088,​17794,​0,​0 
 +dbbc15/​348.990000,​b,​32,​1,​agc,​88,​55,​17788,​18248,​0,​0 
 +</code> 
 +This is the equivalent of ''​bread''​. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes) 
 +<​code>​ 
 +pps_delay/​61971 
 +</​code>​ 
 +The pps_delay should be stable at 61971, to within 1 count of so. 
 +<​code>​ 
 +fila10g/​sysstat 
 + 
 +System status: 
 + ​Selected input      : vsi1 
 + Input sample rate   : 64000000 Hz 
 + VSI input swapped ​  : no 
 + VSI input bitmask ​  : 0x0000FFFF 
 + VSI input width     : 16 bit 
 + PPS count           : 23451 
 + 
 + TVG mode            : vsi-h 
 + 
 + MK5B timesync ​      : yes 
 + VDIF timesync ​      : yes 
 + GPS receiver ​       : installed 
 + 
 + ​Output ​             : started 
 + ​Output 0 format ​    : vdif 
 + ​Output 0 dest.      : 192.168.1.13:​46227 
 + ​Output 1 format ​    : vdif 
 + ​Output 1 dest.      : none 
 + ​Ethernet ARPs       : off 
 + 
 + ​Selected VSI output : vsi1-2 
 + 
 +FiLa10G % 
 +</code>
  
 +This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks. ​
  
  
/home/www/auscope/opswiki/data/attic/operations/aum-hb.1532931933.txt.gz · Last modified: 2018/07/30 06:25 by Jamie McCallum