User Tools

Site Tools


operations:aum-hb

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
operations:aum-hb [2018/07/31 01:22]
Warren Hankey [Revised checklist]
operations:aum-hb [2020/01/27 23:22]
Jamie McCallum
Line 1: Line 1:
 ====== Some background ====== ====== Some background ======
  
-the AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​+The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​
  
-The Hobart12 ​antenna ​now has a functional wideband system, with reasonable ​sensitivitiesThe initial observations ​of the AUM series ​will be made using the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of  32 MHz which will be dealt with in the correlation stage. ​If the current issues with the DBBC3 and its "​Legacy"​-mode software are resolved, we sill change over to use this insteadso the instructions may be updated - please check the page carefully each time you are starting or monitoring an AUM experiment. ​+The Hobart12 ​and Katherine antennas ​now have a functional wideband system, with reasonable ​sensitivityAs of January 2020, we are still observing using the same mode as in the AUM series. This uses the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. ​Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across ​to using that. As suchthese instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment. ​
  
 ===== Major Differences ===== ===== Major Differences =====
  
-The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.+The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.
  
-==== Revised checklist ==== 
- 
-Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. 
  
 === Before the Experiment === === Before the Experiment ===
  
-  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the prc, snp and sum files and put them where they'​re needed.+  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the snp and sum files and put them where they'​re needed.
  
 <​code>​cd /usr2/sched <​code>​cd /usr2/sched
-wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2018/aum001/aum001.skd +wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2020/aua060/aua060.skd 
-drudg aum001.skd+drudg aua060.skd
    hb    hb
    12    12
Line 25: Line 22:
    5    5
    0    0
-scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aum001hb.sum+scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aua060hb.sum
 </​code>​ </​code>​
     ​     ​
-The next step is to convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp+The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of January 2020), the aua060 experiment can be used as  the template. e.g: 
 + 
 +<​code>​ 
 +cd /​usr2/​proc/​ 
 +cp aua060hb.prc aua061hb.prc  
 +</​code>​ 
 + 
 +* Terminate the FS and restart it with "​fs-dummy"​ for both Hb and Ke. This ensures that the Field system will not connect to the DBBC2 and block the configuration scripts.  
 + 
 +* From a terminal on the pcsf machine, run the command ''​./​bin/​aum.query''​. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is  talking to them.  
 +  - First log in to DBBC3 (''​vncviewer dbbc3hb''​) and make sure that the ''​DBBC3 Control DDC_V_v123 .exe''​ program is running, and that the final line of output reads ''​Waiting for connection''​).  
 +  - Next check the DBBC2 (''​vncviewer dbbcho''​ for Hb and ''​vncviewer dbbcke''​ for Ke) and check that the correct control program is running (''​DBBC2 Control DDC v105E_2.exe''​ for Hb, ''​DBBC2 Control DDV v106E_120118.exe''​ for Ke) and displays ''​Waiting for connection''​) 
 +  * If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait  for the DBBC3 after answering "​y"​ to the reconfiguration question).  
 +  * If it does not read ''​waiting for connection'',​ it should give the IP address of the machine that's currently talking to it (e.g if it reads ''​Command from 131.217.63.201:''​ then the connection is from 131.217.63.201). You can lookup up the name of this machine using ''​nslookup 131.217.63.201''​ - in this example it is the ''​hobart''​ machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running ''​fs-auscope''​ will avoid this problem while ''​fs-flexbuff''​ will block communications.  
 + 
 +* Once you can run ''​.bin/​aum.query.py''​ successfully,​ you should be fine to configure the backends with ''​./​bin/​aum.setup.py''​. NB - this script will force a synchronisation of the DBBCs and is equivalent to running ''​setupsx''​ and ''​fmset''​ in sequence. Do not run the setup script during the experiment! 
 + 
 + 
 + 
 +==== Revised checklist ==== 
 + 
 +Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +<​del>​set up convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp
 /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​ /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​
 +</​del>​
 +
 +Updated: The DBBC3KE (which will be recording the X-band data is now configured. "​correlation statistics"​ might be slightly lower from the DBBC than  expected (~16000000 rather than 18000000+). It hopefully won't be too  much of an issue (Lim: cited from Jamie). Use the following script instead:
 +<​code>​snp2flexbuff.aum.sh aum001hb.orig.snp aum001hb.snp hb
 +aum007.setup.dbbc3ke.py</​code>​
 +The flexbuff logs will be under 
 +<​code>/​usr2/​log/​$(lognnm).flexbuffhb.log</​code>​ (S-band recordings only, 46236) and <​code>/​usr2/​log/​$(lognnm).flexbuflyg.log</​code>​ (X-band recordings only, 46229 & 46230)
  
 The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​ The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​
  
-  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''/​usr2/​sched/​aum.32mhz.py''​. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho. ​+  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''​~/aum007.setup.py''​. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho. ​
     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​
-    * DBBC server: If the aum.32mhz.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions on dbbc3hb (''​DBBC#​ Control ​DDC_V_v121.exe''​) and dbbcho (''​DBBC2 Control DDC v105_2.exe''​ and ''​SerialServer.sem.py''​),​ respectively.  +    * DBBC server: If the aum007.setup.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions on dbbc3hb (''​DBBC#​ Control ​DDC_V_v122.exe''​) and dbbcho (''​DBBC2 Control DDC v105_2.exe''​ and ''​SerialServer.sem.py''​),​ respectively.  
-    * IF selections ​ can be checked with the ''​aum.query.py''​ script on pcfshb. For the DBBC3, all inputs and filters should be ''​2'' ​ and all power levels should have a target level of ''​42000''​. For the dbbcho, the input should be ''​1'',​ the filter ''​2''​ and the target power level should be ''​48000''​+    * IF selections ​ can be checked with the ''​aum.check.sh''​ script on pcfshb. For the DBBC3, all inputs and filters should be ''​2'' ​ and all power levels should have a target level of ''​42000''​. For the dbbcho, the input should be ''​1'',​ the filter ''​2''​ and the target power level should be ''​48000''​
   * Maser ok: Check the Hobart26 maser as per usual   * Maser ok: Check the Hobart26 maser as per usual
-  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq-np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​) +  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq -np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​) 
-  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout//​. The existing ''​clkoff''​ command is measuring the dbbchb-gps difference and logging this as //​fmout-gps//​ - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the ''​maserdelay''​ and ''​dbbc3delay''​ commands, logged as ''/​maser2gps/''​ and ''/​gps-fmout/''​ respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled ''​DBBCHO-TACGPS''​and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference ​nd confirm that it is steady.+  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout//​. The existing ''​clkoff''​ command is measuring the dbbchb-gps difference and logging this as //​fmout-gps//​ - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the ''​maserdelay''​ and ''​dbbc3delay''​ commands, logged as ''/​maser2gps/''​ and ''/​gps-fmout/''​ respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled ''​DBBCHO-TACGPS''​and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference ​and confirm that it is steady.
   * Test Recordings: To make a test recording, as ''​oper@pcfshb''​ run  ''​sxflexbuff.sh test 19''​ to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g ''​ tail -100f /​usr2/​log/​$(lognm).flexbuff.log''​). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. ​   * Test Recordings: To make a test recording, as ''​oper@pcfshb''​ run  ''​sxflexbuff.sh test 19''​ to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g ''​ tail -100f /​usr2/​log/​$(lognm).flexbuff.log''​). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. ​
   * Check for any integer second offsets in the recorded data by running ''​sxtest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as ''​MJD = 58330/​01:​09:​59.91''​) should agree to within 0.01 seconds. ​   * Check for any integer second offsets in the recorded data by running ''​sxtest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as ''​MJD = 58330/​01:​09:​59.91''​) should agree to within 0.01 seconds. ​
Line 52: Line 89:
   * No autocorrelations   * No autocorrelations
   * Check the delays with ''​maserdelay''​ and ''​dbbc3delay'',​ confirm the dbbcho time delay is stable   * Check the delays with ''​maserdelay''​ and ''​dbbc3delay'',​ confirm the dbbcho time delay is stable
-  * Check thew Hb maser as usual. ​+  * Check the Hb maser as usual. ​
   * Check the weather data is being logged & add in sky conditions as usual. ​   * Check the weather data is being logged & add in sky conditions as usual. ​
   * No Tsys checks.   * No Tsys checks.
-  * On pcfshb, run ''​./​aum.check.sh''​ to check the dbbc and recorder status. There is a lot of debugging info stored here, so use a large terminal window to run the script. Included below is the full output of a check, with the relevant parameters to check commented on below.+  * On pcfshb, run ''​./​aum.check.sh''​ to check the dbbc and recorder status. There is a lot of debugging info stored here, so use a large terminal window to run the script ​and be aware that you will need to press space to go through the list. Included below is the full output of a check, with the relevant parameters to check commented on below.
 <​code>​ <​code>​
 ###############################​ ###############################​
Line 62: Line 99:
 dbbcifc/ 2,​33,​agc,​2,​41815,​42000 dbbcifc/ 2,​33,​agc,​2,​41815,​42000
 dbbcifd/ 2,​22,​agc,​2,​42416,​42000 dbbcifd/ 2,​22,​agc,​2,​42416,​42000
 +</​code>​
 +This first block is the equivalent of ''​iread''​ - check the inputs and filters are set to ''​2''​ and that the power level matches the target
 +<​code>​
 dbbc17/ 2204.990000,​a,​32,​1,​agc,​55,​63,​14974,​15206,​0,​0 dbbc17/ 2204.990000,​a,​32,​1,​agc,​55,​63,​14974,​15206,​0,​0
 dbbc18/ 2244.990000,​a,​32,​1,​agc,​58,​69,​17035,​15254,​0,​0 dbbc18/ 2244.990000,​a,​32,​1,​agc,​58,​69,​17035,​15254,​0,​0
Line 78: Line 118:
 dbbc31/ 2904.990000,​a,​32,​1,​agc,​183,​175,​15675,​16114,​0,​0 dbbc31/ 2904.990000,​a,​32,​1,​agc,​183,​175,​15675,​16114,​0,​0
 dbbc32/ 2924.990000,​a,​32,​1,​agc,​171,​209,​16013,​16102,​0,​0 dbbc32/ 2924.990000,​a,​32,​1,​agc,​171,​209,​16013,​16102,​0,​0
 +</​code>​
 +This is the quivalent of ''​bread''​ confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.
 +<​code>​
 pps_delay/ [1]: 43 ns, [2] 43 ns, [3] 39 ns, [4] 39 ns, [5] 39 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns; pps_delay/ [1]: 43 ns, [2] 43 ns, [3] 39 ns, [4] 39 ns, [5] 39 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;
 +</​code>​
 +This is the translated equivalent of ''​dbbc=pps_delay'',​ separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable. ​
 +<​code>​
 core3hstats/ ​ core3hstats/ ​
 Core3H[3] Power: Core3H[3] Power:
Line 85: Line 131:
 Sampler 2: 62199248 Sampler 2: 62199248
 Sampler 3: 62160250 Sampler 3: 62160250
 +</​code>​ 
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.  
 +<​code>​
 Core3H[3] Bstat. Core3H[3] Bstat.
 Sampler 0: Sampler 0:
Line 110: Line 158:
 01:     7844 50.20% 01:     7844 50.20%
 00:       35 0.22% 00:       35 0.22%
 +</​code>​ 
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them.  
 +<​code>​
 Core3H[3] Corr. Core3H[3] Corr.
 Sampler 0-1: 180656919 Sampler 0-1: 180656919
 Sampler 1-2: 182717105 Sampler 1-2: 182717105
 Sampler 2-3: 182062361 Sampler 2-3: 182062361
 +</​code>​
 +All inter-sampler correlation values should be ~180000000. ​
 +<​code>​
 core3hstats/ ​ core3hstats/ ​
 Core3H[4] Power: Core3H[4] Power:
Line 121: Line 174:
 Sampler 2: 92322148 Sampler 2: 92322148
 Sampler 3: 93681776 Sampler 3: 93681776
 +</​code>​ 
 +The Core3H power readings from each sampler should all agree to within ~3-5%. This is the second sampler of two, and the above readings are ok. 
 +<​code>​
 Core3H[4] Bstat. Core3H[4] Bstat.
 Sampler 0: Sampler 0:
Line 146: Line 201:
 01:     7733 49.49% 01:     7733 49.49%
 00:      140 0.90% 00:      140 0.90%
 +</​code>​
 +The Bstats can be ignored at this time - they'​re displayed because it's complicated to hide them. 
 +<​code>​
  
 Core3H[4] Corr. Core3H[4] Corr.
Line 151: Line 209:
 Sampler 1-2: 190747558 Sampler 1-2: 190747558
 Sampler 2-3: 187420379 Sampler 2-3: 187420379
 +</​code>​
 +All inter-sampler correlation values should be ~180000000. ​
 +<​code>​
 ###############################​ ###############################​
 Checking dbbcho Checking dbbcho
Line 156: Line 217:
 dbbcifa/​1,​4,​agc,​2,​47097,​48000,​1 dbbcifa/​1,​4,​agc,​2,​47097,​48000,​1
 dbbcifb/​1,​14,​agc,​2,​47991,​48000,​1 dbbcifb/​1,​14,​agc,​2,​47991,​48000,​1
 +</​code>​
 +This is the equivalent of ''​iread''​. All inputs for dbbcho should be ''​1'',​ all filters ''​2''​ and all target power levels ''​48000''​
 +<​code>​
 dbbc01/​317.990000,​a,​32,​1,​agc,​97,​89,​14803,​14952,​0,​0 dbbc01/​317.990000,​a,​32,​1,​agc,​97,​89,​14803,​14952,​0,​0
 dbbc03/​337.990000,​a,​32,​1,​agc,​108,​89,​14945,​15083,​0,​0 dbbc03/​337.990000,​a,​32,​1,​agc,​108,​89,​14945,​15083,​0,​0
Line 164: Line 228:
 dbbc13/​357.990000,​b,​32,​1,​agc,​149,​138,​16248,​16228,​0,​0 dbbc13/​357.990000,​b,​32,​1,​agc,​149,​138,​16248,​16228,​0,​0
 dbbc15/​387.990000,​b,​32,​1,​agc,​125,​145,​14144,​16119,​0,​0 dbbc15/​387.990000,​b,​32,​1,​agc,​125,​145,​14144,​16119,​0,​0
 +</​code>​
 +This is the equivalent of ''​bread''​. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)
 +<​code>​
 pps_delay/​61971 pps_delay/​61971
 +</​code>​
 +The pps_delay should be stable at 61971, to within 1 count of so.
 +<​code>​
 sysstat sysstat
  
Line 193: Line 263:
 ^MFiLa10G %  ^MFiLa10G % 
 </​code>​ </​code>​
 +This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks. ​
 +
 +  * You can check on the status of the recordings by tailing the flexbuff log (''/​usr2/​log/​aum001hb.flexbuff.log''​ on pcfshb - the name will change when a new FS log file is started). This is written to by the recording script and running ''​tail -100f /​usr2/​log/​aum001hb.flexbuff.log''​ will show the same output as in the recording tests, e.g 
 +<​code>​
 +50
 +!runtime = 0 : stream46230 ;!evlbi? 0 : total : 782143 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46236 ;!evlbi? 0 : total : 782107 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46229 ;!evlbi? 0 : total : 782064 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +</​code>​
 +
 +Please note that the output is "​laggy"​ and is usually delayed by ~1 minute. You can see the current status of the recorder using [[http://​ptzhb.phys.utas.edu.au/​web/​admin.html|ptzhb]] - when recording you will see the blue lights of disk activity scattered over the flexbuff (below the DBBC in rack 13). 
/home/www/auscope/opswiki/data/pages/operations/aum-hb.txt · Last modified: 2020/02/25 05:40 by Jamie McCallum