User Tools

Site Tools


operations:aum-hb

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Next revision Both sides next revision
operations:aum-hb [2018/07/31 01:32]
Warren Hankey
operations:aum-hb [2020/01/27 23:22]
Jamie McCallum
Line 1: Line 1:
 ====== Some background ====== ====== Some background ======
  
-the AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​+The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. ​
  
-The Hobart12 ​antenna ​now has a functional wideband system, with reasonable ​sensitivitiesThe initial observations ​of the AUM series ​will be made using the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of  32 MHz which will be dealt with in the correlation stage. ​If the current issues with the DBBC3 and its "​Legacy"​-mode software are resolved, we sill change over to use this insteadso the instructions may be updated - please check the page carefully each time you are starting or monitoring an AUM experiment. ​+The Hobart12 ​and Katherine antennas ​now have a functional wideband system, with reasonable ​sensitivityAs of January 2020, we are still observing using the same mode as in the AUM series. This uses the "​VGOS"​-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. ​Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across ​to using that. As suchthese instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment. ​
  
 ===== Major Differences ===== ===== Major Differences =====
  
-The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.+The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.
  
-==== Revised checklist ==== 
- 
-Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. 
  
 === Before the Experiment === === Before the Experiment ===
  
-  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the prc, snp and sum files and put them where they'​re needed.+  * Preparing the schedule file: Do not use the existing slogit scripts, but instead download the skd file directly to pcfshb and run drudg to generate the snp and sum files and put them where they'​re needed.
  
 <​code>​cd /usr2/sched <​code>​cd /usr2/sched
-wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2018/aum001/aum001.skd +wget ftp://​cddis.gsfc.nasa.gov/​vlbi/​ivsdata/​aux/​2020/aua060/aua060.skd 
-drudg aum001.skd+drudg aua060.skd
    hb    hb
    12    12
Line 25: Line 22:
    5    5
    0    0
-scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aum001hb.sum+scp /​tmp/​sched.tmp observer@ops-serv2:/​vlbobs/​ivs/​logs/​aua060hb.sum
 </​code>​ </​code>​
     ​     ​
-The next step is to convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp+The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of January 2020), the aua060 experiment can be used as  the template. e.g: 
 + 
 +<​code>​ 
 +cd /​usr2/​proc/​ 
 +cp aua060hb.prc aua061hb.prc  
 +</​code>​ 
 + 
 +* Terminate the FS and restart it with "​fs-dummy"​ for both Hb and Ke. This ensures that the Field system will not connect to the DBBC2 and block the configuration scripts.  
 + 
 +* From a terminal on the pcsf machine, run the command ''​./​bin/​aum.query''​. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is  talking to them.  
 +  - First log in to DBBC3 (''​vncviewer dbbc3hb''​) and make sure that the ''​DBBC3 Control DDC_V_v123 .exe''​ program is running, and that the final line of output reads ''​Waiting for connection''​).  
 +  - Next check the DBBC2 (''​vncviewer dbbcho''​ for Hb and ''​vncviewer dbbcke''​ for Ke) and check that the correct control program is running (''​DBBC2 Control DDC v105E_2.exe''​ for Hb, ''​DBBC2 Control DDV v106E_120118.exe''​ for Ke) and displays ''​Waiting for connection''​) 
 +  * If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait  for the DBBC3 after answering "​y"​ to the reconfiguration question).  
 +  * If it does not read ''​waiting for connection'',​ it should give the IP address of the machine that's currently talking to it (e.g if it reads ''​Command from 131.217.63.201:''​ then the connection is from 131.217.63.201). You can lookup up the name of this machine using ''​nslookup 131.217.63.201''​ - in this example it is the ''​hobart''​ machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running ''​fs-auscope''​ will avoid this problem while ''​fs-flexbuff''​ will block communications.  
 + 
 +* Once you can run ''​.bin/​aum.query.py''​ successfully,​ you should be fine to configure the backends with ''​./​bin/​aum.setup.py''​. NB - this script will force a synchronisation of the DBBCs and is equivalent to running ''​setupsx''​ and ''​fmset''​ in sequence. Do not run the setup script during the experiment! 
 + 
 + 
 + 
 +==== Revised checklist ==== 
 + 
 +Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 + 
 +<​del>​set up convert the existing snp file into one which will control the flexbuff recordings appropriately. This is done via a script at present - please use the following commands <​code>​cp aum001hb.snp aum001hb.orig.snp
 /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​ /​usr2/​sched/​snp2flexbuff.sh aum001hb.orig.snp aum001hb.snp hb</​code>​
 +</​del>​
 +
 +Updated: The DBBC3KE (which will be recording the X-band data is now configured. "​correlation statistics"​ might be slightly lower from the DBBC than  expected (~16000000 rather than 18000000+). It hopefully won't be too  much of an issue (Lim: cited from Jamie). Use the following script instead:
 +<​code>​snp2flexbuff.aum.sh aum001hb.orig.snp aum001hb.snp hb
 +aum007.setup.dbbc3ke.py</​code>​
 +The flexbuff logs will be under 
 +<​code>/​usr2/​log/​$(lognnm).flexbuffhb.log</​code>​ (S-band recordings only, 46236) and <​code>/​usr2/​log/​$(lognnm).flexbuflyg.log</​code>​ (X-band recordings only, 46229 & 46230)
  
 The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​ The newly created aum001hb.snp file retains the timing of the source commands, scan names and times but calls a script to run the recorder and omits many of the typical procedures like preob, etc.  **NB - do not use the line number entries in the sum file!** ​
  
-  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''/​usr2/​sched/​aum.32mhz.py''​. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho. ​+  * RF and IF paths configured/​DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on pcfshb ''​~/aum007.setup.py''​. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the 26m backend and dbbcho. ​
     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​     * No RF configuration is needed for the X-band data recorded through the DBBC3, but it is necessary to set the inputs on the 26m's remote backend (e.g "the folly" or "​PalfreymansBackend"​. Go to the hobart26 VNC session (''​vncviewer newsmerd:​1''​) and go to desktop 3. Refresh the Folly display with '​r'​ and confirm that **S/X SRCP** and **S/X XRCP** are highlighted for channels 1 and 2 respectively. ​
-    * DBBC server: If the aum.32mhz.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions on dbbc3hb (''​DBBC#​ Control ​DDC_V_v121.exe''​) and dbbcho (''​DBBC2 Control DDC v105_2.exe''​ and ''​SerialServer.sem.py''​),​ respectively.  +    * DBBC server: If the aum007.setup.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions on dbbc3hb (''​DBBC#​ Control ​DDC_V_v122.exe''​) and dbbcho (''​DBBC2 Control DDC v105_2.exe''​ and ''​SerialServer.sem.py''​),​ respectively.  
-    * IF selections ​ can be checked with the ''​aum.query.py''​ script on pcfshb. For the DBBC3, all inputs and filters should be ''​2'' ​ and all power levels should have a target level of ''​42000''​. For the dbbcho, the input should be ''​1'',​ the filter ''​2''​ and the target power level should be ''​48000''​+    * IF selections ​ can be checked with the ''​aum.check.sh''​ script on pcfshb. For the DBBC3, all inputs and filters should be ''​2'' ​ and all power levels should have a target level of ''​42000''​. For the dbbcho, the input should be ''​1'',​ the filter ''​2''​ and the target power level should be ''​48000''​
   * Maser ok: Check the Hobart26 maser as per usual   * Maser ok: Check the Hobart26 maser as per usual
-  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq-np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​) +  * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''​ssh observer@flexbuffhb''​) and confirm that the time is correct (with ''​ntpq -np''​) and that jive5ab is running (with ''​ps -ef | grep jive''​) 
-  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout//​. The existing ''​clkoff''​ command is measuring the dbbchb-gps difference and logging this as //​fmout-gps//​ - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the ''​maserdelay''​ and ''​dbbc3delay''​ commands, logged as ''/​maser2gps/''​ and ''/​gps-fmout/''​ respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled ''​DBBCHO-TACGPS''​and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference ​nd confirm that it is steady.+  * The clock offsets can be (partially) read through the FS. ''​maserdelay''​ reads the maser-GPS difference as usual while ''​dbbc3delay''​ logs the DBBC3-GPS delay //as gps-fmout//​. The existing ''​clkoff''​ command is measuring the dbbchb-gps difference and logging this as //​fmout-gps//​ - this is irrelevant to the Hb12 recordings but is needed for the simultaneous Ho26 experiments!. In summary - compare the outputs of the ''​maserdelay''​ and ''​dbbc3delay''​ commands, logged as ''/​maser2gps/''​ and ''/​gps-fmout/''​ respectively. You should check the dbbc-gps difference for DBBCHO using the ptzhb camera. The relevant counter is to the left of the mark4 rack at the top of rack 9. It's labeled ''​DBBCHO-TACGPS''​and is typically ~1 microsecond offset from the maserdelay. You do not need to log this value but not the difference ​and confirm that it is steady.
   * Test Recordings: To make a test recording, as ''​oper@pcfshb''​ run  ''​sxflexbuff.sh test 19''​ to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g ''​ tail -100f /​usr2/​log/​$(lognm).flexbuff.log''​). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. ​   * Test Recordings: To make a test recording, as ''​oper@pcfshb''​ run  ''​sxflexbuff.sh test 19''​ to carry out a 20s test recording. You can check the status of the flexbuff by tailing the log file (e.g ''​ tail -100f /​usr2/​log/​$(lognm).flexbuff.log''​). Confirm that the total number of packets is incrementing after each check and that there are no lost or out-of-order packets - these are symptoms of the flexbuff being overloaded. ​
   * Check for any integer second offsets in the recorded data by running ''​sxtest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as ''​MJD = 58330/​01:​09:​59.91''​) should agree to within 0.01 seconds. ​   * Check for any integer second offsets in the recorded data by running ''​sxtest.sh''​ on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries (presented as ''​MJD = 58330/​01:​09:​59.91''​) should agree to within 0.01 seconds. ​
Line 52: Line 89:
   * No autocorrelations   * No autocorrelations
   * Check the delays with ''​maserdelay''​ and ''​dbbc3delay'',​ confirm the dbbcho time delay is stable   * Check the delays with ''​maserdelay''​ and ''​dbbc3delay'',​ confirm the dbbcho time delay is stable
-  * Check thew Hb maser as usual. ​+  * Check the Hb maser as usual. ​
   * Check the weather data is being logged & add in sky conditions as usual. ​   * Check the weather data is being logged & add in sky conditions as usual. ​
   * No Tsys checks.   * No Tsys checks.
Line 180: Line 217:
 dbbcifa/​1,​4,​agc,​2,​47097,​48000,​1 dbbcifa/​1,​4,​agc,​2,​47097,​48000,​1
 dbbcifb/​1,​14,​agc,​2,​47991,​48000,​1 dbbcifb/​1,​14,​agc,​2,​47991,​48000,​1
 +</​code>​
 +This is the equivalent of ''​iread''​. All inputs for dbbcho should be ''​1'',​ all filters ''​2''​ and all target power levels ''​48000''​
 +<​code>​
 dbbc01/​317.990000,​a,​32,​1,​agc,​97,​89,​14803,​14952,​0,​0 dbbc01/​317.990000,​a,​32,​1,​agc,​97,​89,​14803,​14952,​0,​0
 dbbc03/​337.990000,​a,​32,​1,​agc,​108,​89,​14945,​15083,​0,​0 dbbc03/​337.990000,​a,​32,​1,​agc,​108,​89,​14945,​15083,​0,​0
Line 188: Line 228:
 dbbc13/​357.990000,​b,​32,​1,​agc,​149,​138,​16248,​16228,​0,​0 dbbc13/​357.990000,​b,​32,​1,​agc,​149,​138,​16248,​16228,​0,​0
 dbbc15/​387.990000,​b,​32,​1,​agc,​125,​145,​14144,​16119,​0,​0 dbbc15/​387.990000,​b,​32,​1,​agc,​125,​145,​14144,​16119,​0,​0
 +</​code>​
 +This is the equivalent of ''​bread''​. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)
 +<​code>​
 pps_delay/​61971 pps_delay/​61971
 +</​code>​
 +The pps_delay should be stable at 61971, to within 1 count of so.
 +<​code>​
 sysstat sysstat
  
Line 217: Line 263:
 ^MFiLa10G %  ^MFiLa10G % 
 </​code>​ </​code>​
 +This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks. ​
 +
 +  * You can check on the status of the recordings by tailing the flexbuff log (''/​usr2/​log/​aum001hb.flexbuff.log''​ on pcfshb - the name will change when a new FS log file is started). This is written to by the recording script and running ''​tail -100f /​usr2/​log/​aum001hb.flexbuff.log''​ will show the same output as in the recording tests, e.g 
 +<​code>​
 +50
 +!runtime = 0 : stream46230 ;!evlbi? 0 : total : 782143 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46236 ;!evlbi? 0 : total : 782107 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +!runtime = 0 : stream46229 ;!evlbi? 0 : total : 782064 : loss : 0 ( 0.00%) : out-of-order : 0 ( 0.00%) : extent : 0seqnr/pkt ;
 +</​code>​
 +
 +Please note that the output is "​laggy"​ and is usually delayed by ~1 minute. You can see the current status of the recorder using [[http://​ptzhb.phys.utas.edu.au/​web/​admin.html|ptzhb]] - when recording you will see the blue lights of disk activity scattered over the flexbuff (below the DBBC in rack 13). 
/home/www/auscope/opswiki/data/pages/operations/aum-hb.txt · Last modified: 2020/02/25 05:40 by Jamie McCallum