User Tools

Site Tools


operations:correlator_reports2014-11-06

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

operations:correlator_reports2014-11-06 [2014/11/06 00:31] (current)
liza created
Line 1: Line 1:
 +----------------------------------------------------------------------------------------------------------------------
 +AUST37
 +
 +Hb:
 +    wind stows early in experiment, scan missed. Ross didn't record which scans were missed so I will have to manually go through the log at the end
 +
 +Ke:
 +    Tsys bands 09, 11, 12 been a bit high 160-190 all experiment, and this has been happening with band 09 continuously in previous experiments too
 +
 +Yg:
 +    problem with DBBC: scans missed, again Ross didn't note which ones
 +    Tsys band 11 consistently a bit low ~50 (I don't know if this is an issue) I don't know if this has been happening in previous experiments
 +    E-remote crashed for no reason but the field system kept running so no scans missed due to this (06:30 UT)
 +
 ++SUMMARY
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +
 +   ​5-9 ​           95%             -
 +    0              3%             -
 +  4-1,​A-H,​N ​       2%             -
 +  Removed ​         -              -
 +
 +  HOBART12 (Hb/L): Strong RFI in SR5U and SR6U. Late start - Windstowed ​
 +until 218-0815. Manual phase cal applied. Apparent clock breaks at 1147, 
 +1300 and 1451. Cause unknown.
 +
 +  HART15M ​ (Ht/g): Ok.
 +
 +  KATH12M ​ (Ke/i): Small clock breaks apparent at 218-2320 and 218-0530.
 +
 +  WARK12M ​ (Ww/W): Manual phase cal.
 +
 +  YARRA12M (Yg/e): DBBC reconfigured ​ at 1030, with associated clock break.
 +
 +Station Performance ​
 +
 +                           ​Number of Observations
 +
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +
 +  HART15M ​               1491          1320         ​1283 ​       86.0%
 +  HOBART12 ​              ​2091 ​         1932         ​1825 ​       87.3%
 +  KATH12M ​               2088          1991         ​1941 ​       93.0%
 +  WARK12M ​               1912          1801         ​1758 ​       91.9%
 +  YARRA12M ​              ​2126 ​         2000         ​1937 ​       91.1%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       4854          4522         ​4372 ​       90.1%
 +
 +
 +----------------------------------------------------------------------------------------------------------------------
 +AUST38
 +
 +Experiment started OK 00:00, all modules blank: Hb - HOB+0118/​16000,​ Ke: HOB+1008/​16000,​ Yg: HOB+0118/​32000 And yes, that's correct, we have two different modules with identical VSNs: HOB+0118/​16000 in Hb and HOB+0118/​32000 in Yg. Both are used for AUST38 and AUST39.
 +
 +Hb: strong spikes in ch 14,15)
 +
 +Ke: S band, esp. ch11, 12, 15 – strange autocorr bandpass, extremely high Tsys (150-185K). RFI? Bad calibration?​
 +
 +Yg: DIMino program fails sporadically (scans 221-0101 - 0108 are lost); S band Tsys in ch 12, 13, 14 is too high (105-120K)
 +
 +All: autocorr bandpass in ch.14 looks “steep” in all three antennas.
 +
 +
 +
 +The Katherine delay difference has been drifting between .300 and .315 us
 +
 +tsys on SBand at Katherine has also been very high at times throughout my shift
 +
 ++SUMMARY
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +
 +   ​5-9 ​           44%             -
 +    0              2%             -
 +  4-1,​A-H,​N ​      ​54% ​            -
 +  Removed ​         -              -
 +
 +  HART15M ​ (Ht/g): Ok.
 +
 +  HOBART12 (Hb/L): Apparent clock breaks at 1342 and 1752
 +
 +  KATH12M ​ (Ke/i): Ok.
 +
 +  YARRA12M (Yg/e): All data lost due to module failure.
 +
 +Session Statistics ​
 +
 +  Observations: ​    3595 scheduled ​
 +                    1669 correlated (in database)
 +                    1529 recoverable (usable)
 +                    1498 used
 +  Session fit:        50.438 ps
 +
 +-----------------------------------------
 +Station Performance ​
 +                          Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +
 +  HART15M ​               1539           ​854 ​         837        54.4%
 +  HOBART12 ​              ​1863 ​         1110         ​1087 ​       58.3%
 +  KATH12M ​               1862          1094         ​1072 ​       57.6%
 +  YARRA12M ​              ​1926 ​     NOT CORR     NOT CORR         0.0%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       3595          1529         ​1498 ​       41.7%
 +----------------------------------------------------------------------------------------------------------------------
 +AUST39
 +
 +The disk positions are recorded on the printout (fairly early in the experiment) as too many scans were missed and there was no connection to Ke and Yg to figure out the start diks pos of the experiment End person (Warren) will have to look through the logs just for the first couple of scans to see which ones were actually recorded.
 +
 +Hb:
 +    mk5 returned errors stating the that mk5 can't record while data transfers (for some reason this did not set off the alarm like it did at Ke)
 +    these errors were returned during the 1st couple of scans so I don't know if they'​re any good
 +    log reported that recording stopped mid scan 222-0016a (at UT00:16:42) and then reported that the next scan was 0013 (???)
 +    this ridiculous notion of re-recording a previous scan I think is what caused the field system to crash, missing all scans after this until 0036: e-remote doesn'​t work well as a time machine evidently
 +    schedule resumed at scan 0036 (disk pos: 5382.679)
 +    All S band tsys values too high for entire experiment
 +
 +Ke:
 +    Alarms went off right at the beginning of the experiment saying things like (mk5-900): “can'​t record while data transfers”,​ “not while other activity”
 +    This went on until the field system had a fit and Jamie had to restart - again, I have no idea how usable any of these scans are
 +    first recorded scan after FS fit was 222-0041 (disk pos: 5843.831)
 +    Tsys values in bands 09-12 always too high (band 09 approximately 180K)
 +
 +Yg:
 +    no connection throughout the start of the experiment but presumably, similar errors arose
 +    first scan after finally re-connecting was 222-0044 (diskpos: 5996.523)
 +    Tsys values in band 11 consistently 50K
 +
 +Schedule errors day 222 22:32:31 and 22:33:27, requests to Mk5 recorder ( for scan_check perhaps?) while recording. Recording continued despite error messages.
 +
 ++SUMMARY
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +   ​5-9 ​           43%             -
 +    0              3%             -
 +  4-1,​A-H,​N ​      ​55% ​            -
 +  Removed ​         -              -
 +
 +  HART15M ​ (Ht/g): Ok.
 +
 +  HOBART12 (Hb/L): Two clock breaks (1558 and 2122) apparent in data.
 +
 +  KATH12M ​ (Ke/i): Missing first 40 minutes.
 +
 +  YARRA12M (Yg/e): All data lost due to module failure.
 +Session Statistics ​
 +
 +  Observations: ​    3634 scheduled ​
 +                    1643 correlated (in database)
 +                    1481 recoverable (usable)
 +                    1459 used
 +  Session fit:        48.982 ps
 +
 +-----------------------------------------
 +Station Performance ​
 +
 +                           ​Number of Observations
 +
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +
 +  HART15M ​               1539           ​816 ​         807        52.4%
 +  HOBART12 ​              ​1892 ​         1085         ​1069 ​       56.5%
 +  KATH12M ​               1890          1061         ​1042 ​       55.1%
 +  YARRA12M ​              ​1947 ​     NOT CORR     NOT CORR         0.0%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       3634          1481         ​1459 ​       40.1%
 +--------------------------------------------------------------------------------------------------------------------
 +AUST41+42
 +Ke: 
 +Module USN+0224 8000 GB was loaded in Bank B but key not turned on! Martin attempting to get there, but probably expect late start…. ​
 +Recording finally started to USN+0225 at 00:39 after modules sorted. Recorded data is -179 GB short of scheduled, due to late start. ​
 +ATTENTION! Looks like module in bank B USN+0224 has bad soft VSN (name too long warning) will need to sort out when there is a substantial gap in scans. ​
 +Aust42 commenced on bank B module USN+0224, as above. ​
 +Hb: 
 +Recording to HOB+0041 starting at zero GB. 
 +Module changed in A to HOB+0023 6 TB ready for aust42 by Warren on Saturday. ​
 +Hb automatically changed to bank A for start of scan 229-0530. ​
 +Yg: 
 +Antenna stuck. Not in windstow, elevation brakes wont release. Dave local at yg went out to reset power in pedestal. Still no joy. 
 +Move to Current Issues! Elevation brakes will not release. fs, halt schedule. Need to switch power off in pedestal, open cabinet and switch to maintenance mode. Switch power back on, but this time to hand box control. Should be able to drive by hand. Switch power back over to remote mode. HMI interface should now drive telescope might need to “Reboot Central” in HMI. fs, antenna=open,​ antenna=operate,​ resume schedule. Good luck with all of that…. ​
 +Recorded data is -404 GB short of scheduled, due to late start. ​
 +
 +
 ++SUMMARY aust41
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +   ​5-9 ​           96%             -
 +    0              -              -
 +  4-1,​A-H,​N ​       4%             -
 +  Removed ​         -              -
 +
 +  HOBART12 (Hb/L): Strong RFI in SR5U ands SR6U. Weak channel threshold
 +   ​reduced to 0.3 to avoied excessive data loss. Small clock breaks
 +   ​apparent at 0302, 1506 and 2005.
 +
 +  KATH12M ​ (Ke/i): Missed first 40 minutes of schedule - late start.
 +
 +  WARK12M ​ (Ww/W): Manual phase cal applied.
 +
 +  YARRA12M (Yg/e): Missed first 90 minutes of schedule - Windstowed. Small
 +   clock break apparent at 1239. No usable phase cal in this experiment.
 +   ​Manual phase cal used, with two epochs required. Small clock break
 +   ​apparent at 1239 when the pcal solution required updating.
 +
 ++MANUAL PCAL Ww Yg Yg
 +Session Statistics ​
 +
 +  Observations: ​    4006 scheduled ​
 +                    3856 correlated (in database)
 +                    3809 recoverable (usable)
 +                    3744 used
 +  Session fit:        32.515 ps
 +
 +----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                             obs used
 +  HOBART12 ​              ​2015 ​         1931         ​1898 ​       94.2%
 +  KATH12M ​               2011          1933         ​1906 ​       94.8%
 +  WARK12M ​               1972          1884         ​1847 ​       93.7%
 +  YARRA12M ​              ​2014 ​         1870         ​1837 ​       91.2%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       4006          3809         ​3744 ​       93.5%
 +
 ++SUMMARY aust42
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +   ​5-9 ​           99%             -
 +    0              -              -
 +  4-1,​A-H,​N ​       1%             -
 +  Removed ​         -              -
 +
 +  HOBART12 (Hb/L): Strong RFI in SR5U and SR6U. Many clock breaks apparent
 +   in the data, usually of a few ns magnitude. Approximate times are 0929,
 +   1429, 1728 and 2148.
 +
 +  KATH12M ​ (Ke/i): Small clock break apparent at 0645.
 +
 +  WARK12M ​ (Ww/W): Manual pcal.
 +
 +  YARRA12M (Yg/e): Manual pcal.
 +
 ++MANUAL PCAL Ww Yg
 +Session Statistics ​
 +
 +  Observations: ​    4065 scheduled ​
 +                    4065 correlated (in database)
 +                    3991 recoverable (usable)
 +                    3856 used
 +  Session fit:        35.601 ps
 +-----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                             obs used
 +  HOBART12 ​              ​2047 ​         1989         ​1911 ​       93.4%
 +  KATH12M ​               2041          2025         ​1970 ​       96.5%
 +  WARK12M ​               1995          1944         ​1895 ​       95.0%
 +  YARRA12M ​              ​2047 ​         2024         ​1936 ​       94.6%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       4065          3991         ​3856 ​       94.9%
 +----------------------------------------------------------------------------------------------------------------------
 +R1649
 +Hb: at 15:19:05.13 got error ”#​trakl#​Antenna stuck” while at Az: -234.7825 and El: 29.3014. Tried antenna=open (was okay) then antenna=operate and power on timed-out. Also tried Reset Drives button in HMI and the Drives ON and Operate buttons, but it still wouldn'​t switch on. Tried restarting the FS and doing all of the above again - which didn't work either. Abandoned schedule as only ~1.5 hrs left and will need someone out at the telescope to manually get it going. ​
 +Hb and Yg: 11:18UT noticed channel 10 in autocorrelation spectra sometimes have a funny looking step function rather than being flat. 
 ++SUMMARY ​
 + ​Qcode ​     % of Total   % of Correlated ​
 +                scans          scans     
 +   ​5-9 ​           97%             ​- ​
 +   ​0 ​             1%             ​- ​
 + ​4-1,​A-H,​N ​       2%             ​- ​
 + ​Removed ​         -              - 
 +
 ++CORRELATOR_NOTES
 +Katherine data transfer completed in the afternoon of 
 +friday the 29.08.2014. ​
 +
 +HOBART12 (Hb/d): Missed scans after UT 321-1519 due to antenna problems. Those scans contribute to the statistics as                   ​N-codes (uncorrelated scans). ​
 + ​KATH12M ​ (Ke/a): Ok. 
 + ​YARRA12M (Yg/i): Applied manual pcal.\
 +CRDS72
 +Pointing and fringe checks were not performed, otherwise started right in time. 
 +Ho: 
 +erc Status Monitor window shows wrong time: 1969.264.23.0.00 while it is really (according to formatter, FS, computer) is 2014.233.08.20.00. Execution of commands happens according to real time. 
 +At around 16:45, cryogenic show signs of warm-up: T(70K phase)=84K, T(20K)=65K, He pressure goes down recently, but currently 305 psi
 +Frequent error: “Error being ignored: error qk -211 tsys value for device ic overflowed or was less than zero.” and “Error being ignored: error ch -308 vd total power integrator overflow”
 +recording to USN-0109
 +Hb: 
 +Peaks autocorrelation bandpasses in ch 14, 15 – RFI
 +recording to NRCAN-05
 +Ke: 
 +Persistent problem: WARNING: error an -103 pointing computer tracking errors are too large. ; WARNING: error qo -301 warning: onsource status is slewing!
 +recording to NRCAN-07
 +Yg: 
 +Internet connection is flaky, ERC loses connection every now and then, so often that even filling the checklist is problematic
 +Descending slope in the autocorrelation bandpass in channel 15 – RFI
 +recording to USN-0205
 ++SUMMARY
 +Qcode   % of Total   % of Correlated
 +          scans          scans
 + ​5-9 ​      ​85% ​          85%
 +  0        13%           13%
 + ​B-H ​       2%            2%
 +Removed ​    0%
 +Qcodes 7-9 Correlated 83%
 +Qcode N Not Correlated ​ 0%
 +HOBART26 (Ho/​A): ​  ​Manual pcal applied at HOBART26
 +KATH12M ​ (Ke/​K): ​  OK
 +YARRA12M ​ (Yg/​Y): ​ Manual pcal applied at YARRA12m
 +Session Statistics ​
 +  Observations: ​    1911 scheduled ​
 +                    1883 correlated (in database)
 +                    1438 recoverable (usable)
 +                    1383 used
 +  Session fit:        28.810 ps
 +-----------------------------------------
 +Station Performance ​
 +
 +                           ​Number of Observations
 +
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +
 +  HARTRAO ​                ​669 ​          ​497 ​         493        73.7%
 +  HOBART12 ​               669           ​478 ​         435        65.0%
 +  HOBART26 ​               664           ​523 ​         474        71.4%
 +  KATH12M ​                ​579 ​          ​440 ​         437        75.5%
 +  WARK12M ​                ​581 ​          ​419 ​         414        71.3%
 +  YARRA12M ​               660           ​519 ​         513        77.7%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       1911          1438         ​1383 ​       72.4%
 +aust43
 +DISKS: ​
 +hb HOB+0119
 +ke HOB+0122
 +yg HOB+0120
 +Ke
 +1000 UTC: Int. T. reading red at 55 C. The internal temperature of the maser unit has risen to warning levels. Monica shows that the maser room temperature has risen since about 0 UTC on 27/8 reaching a peak at 0800 UTC and is now falling again. Brett informed. Martin to investigate in the morning (~2000 UTC). 
 +1200 UTC: Int. T. now at 48 C and warning light off in maser monitor. ​
 +1300 UTC: Delay difference has been drifted since start of experiment. This seems to be coming from the formatter. ​
 +1315 UTC: Drift rate increased. Rebooting and formatting the DBBC. 
 +2208 UTC: fmout seems to have been stable since UT18:​00 ​
 +1330 UTC: Resumed after reboot. Monitoring offsets. ​
 +1750 UTC: Delay difference drifting again. Trying another reconf. ​
 +2209 UTC: fmout seems to have been stable since UT18:​00 ​
 +0115 UTC: Large difference between formatter and GPS. Halted schedule to run fmset. This did not fix the problem. Jamie restarted the DBBC and reconfigured Schedule starting again at UT02:​14. ​
 +0702 UTC: Formatter to GPS has remained stabled till the end of observations. ​
 ++SUMMARY
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +   ​5-9 ​           89%             -
 +    0              6%             -
 +  4-1,​A-H,​N ​       4%             -
 +  Removed ​         -              -
 +  HOBART12 (Hb/L): Ok.
 +
 +  KATH12M ​ (Ke/i): Very little usable data. The stations maser lost
 +   ​temperature control during this experiment, causing varge clock drifts
 +   and breaks until ~2100. There was a second clock break at 0115 linked
 +   to the DBBC.
 +
 +  YARRA12M (Yg/e): Manual phase cal applied. Small clock break (0.7 ns) 
 +apparent at 1953.
 ++MANUAL PCAL Yg
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +  HOBART12 ​               324           ​279 ​         267        82.4%
 +  KATH12M ​                ​310 ​          ​249 ​         231        74.5%
 +  YARRA12M ​               328           ​276 ​         266        81.1%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​        ​481 ​          ​402 ​         382        79.4%
 +----------------------------------------------------------------------------------------------------------------------
 +R4650
 +hb: 
 +recording to HOB+0119 start at 8692 GB. 
 +ke: 
 +recording to BKG+0126 start 0 GB. Test record OK.
 +After start channels 11, 12 looked suspect in autocorrelation.
 +clock jumped at 19:00. halted schedule and reconfigured dbbc at 19:03. fs crashed also. 
 +one missed scan 240-1902a. next good scan 240-1916. (missed scan record stop means -20 GB approx.)
 +clock delay drifted from -0.29 us to -0.4 us around 240/20:30 and has stabilised see ke_ddiff.txt plot
 +missed scans 240-2312 and 240-2328 due to dbbc reconfig.
 +clock delay stable since 240-2330 (as of 240-0315)
 +At UT13:40 day 241, the delay difference at Katherine started to drift quite badly, had to run fmset at 13.53UT (Imogen) ​
 +For some reason, gnuplot will not plot values of ke_ddif.txt past ~11:00UT - not sure why (Imogen). ​
 +At 14:29UT, restarted DBBC as clock drift was still occurring, ran fmset to resync dbbc and mark5. Delay seems to be stable as of 14:42UT, however cannot plot the values as they do not seem to be reaching the ke_ddif.txt file. 
 +yg: 
 +recording to USN-0019 start 0 GB
 +spurious wind stow 240/21:22, antenna will not release elevation brake. Sending Jack from moblas to try handbox antenna alive
 +also note anemometer cant connect to device
 +schedule halted 240/21:36
 +drives still inoperable at 240/23:59
 +restarted for scan 241-0447
 +Qcode   % of Total   % of Correlated
 +          scans          scans
 + ​5-9 ​      ​68% ​          96%
 +  0         ​1% ​           2%
 + ​B-H ​       1%            2%
 +Removed ​   28% (Ww did not participate)
 +HOBART12 (Hb/H): Ok.
 +  ​
 +KATH12M ​ (Ke/A): Multiple clock breaks/​drifts.
 +
 +                 Clock breaks applied at 240-2313, 241-0501, and 241-1448.
 + ​YARRA12M (Yg/Y): Low phasecal amplitudes throughout, leading to '​H'​ codes.
 +                Manual phase calibration applied at YARRA12M.
 +Clocks: WACO
 +
 +Station ​         fmout-gps ​     Used      rate
 +                  [usec] ​      ​[usec] ​  ​[sec/​sec]
 + ​Ft ​                ​0.57 ​       0.67     ​2.551e-13
 + ​Hb ​               13.41       ​16.06 ​    ​0.000e+00
 + ​Ke ​                ​9.72 ​      ​11.59 ​    ​0.000e+00
 +                               ​12.13 ​    ​0.000e+00 # 241-2313
 +                               ​11.59 ​    ​0.000e+00 # 241-0501
 +                               ​11.72 ​    ​0.000e+00 # 241-1448
 + ​Kk ​                ​9.24 ​      ​10.10 ​   -2.741e-13
 + ​Ny ​               73.59       ​73.71 ​    ​1.793e-12
 + ​Wz ​              ​-29.33 ​     -29.08 ​    ​0.000e+00
 + ​Yg ​               11.36       ​13.56 ​    ​0.000e+00
 +Session Statistics ​
 +
 +  Observations: ​    2271 scheduled ​
 +                    1522 correlated (in database)
 +                    1522 recoverable (usable)
 +                    1443 used
 +  Session fit:        43.179 ps
 +-----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +  FORTLEZA ​               294           ​255 ​         237        80.6%
 +  HOBART12 ​               657           ​434 ​         408        62.1%
 +  KATH12M ​                ​669 ​          ​425 ​         404        60.4%
 +  KOKEE                   ​709 ​          ​578 ​         552        77.9%
 +  NYALES20 ​               588           ​533 ​         508        86.4%
 +  WARK12M ​                ​485 ​       MISSED ​      ​MISSED ​        0.0%
 +  WETTZELL ​               536           ​497 ​         469        87.5%
 +  YARRA12M ​               604           ​322 ​         308        51.0%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       2271          1522         ​1443 ​       63.5%
 +----------------------------------------------------------------------------------------------------------------------
 +AUST44
 +Logs and EndMess sent (Lucia) All experiments started OK. 
 +Hb module: HOB+0119 (~10.66 TB ahead), changed to HOB+0104 part way through as didn't have enough space.
 +Ke module: HOB+0122 (~7.82 TB ahead)
 +Yg module: HOB+0120 (~9.54 TB ahead)
 +Hb 
 +Module only has ~5 TB of space, does this need to be changed?
 +Missed most scans in 242-0754 to 242-0943 due to mk5 issues loading up another module
 +Ke
 +There are some communications issues with eControl - use Dave's “backup control”.
 +Yg
 +There are some communications issues with eControl- use Dave's “backup control”.
 ++SUMMARY
 +  Qcode      % of Total   % of Correlated
 +                 ​scans ​         scans
 +   ​5-9 ​           95%             -
 +    0              1%             -
 +  4-1,​A-H,​N ​       4%             -
 ++CORRELATOR_NOTES
 +Manual phase cal was applied for all stations, after noting loss of phase
 +cal at Yg.
 +
 +  HOBART12 (Hb/L): Strong RFI in SR5U and SR6U. Small clock break apparent
 +   at 2005
 +
 +  KATH12M ​ (Ke/i): Manual pcal. Small clock breaks apparent at 0702 and 1208
 +
 +  YARRA12M (Yg/e): Manual pcal.
 +
 +
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +
 +  HOBART12 ​              ​1468 ​         1356         ​1299 ​       88.5%
 +  KATH12M ​               1469          1417         ​1379 ​       93.9%
 +  YARRA12M ​              ​1469 ​         1409         ​1360 ​       92.6%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       2203          2091         ​2019 ​       91.6%
 +----------------------------------------------------------------------------------------------------------------------
 +aust46
 +Hb: 
 +Recording to HOB+0035. +3 GB
 +Usual RFI from the NBN in last two channels
 +Ke: 
 +Recording to HOB+0122. +20792 GB 
 +RFI ongoing in S band.
 +Yg: 
 +Recording to HOB+0120. +22656 GB 
 +Antenna was stuck for about an hour before the experiment. Onsite staff managed to return control about 15 min into exp. First source was 244-0716b.
 +Ho: 
 +Recording to USN-0027. +3 GB 
 +Receiver is warm, so tsys higher than usual. ​
 +Status monitor clock shows wrong datetime. However, log time agrees with monit2 agrees with UT. So will clarify inconsistency at a later time.
 +
 +
 +----------------------------------------------------------------------------------------------------------------------
 +APSG34
 +Hb: 
 +Recording to HOB+0128 (bank B), +2890 GB.
 +Wind stowed at 02:18 UT, resumed at 02:55 UT. Missed scans 253-0220 through 253-0247a.
 +Wind stowed again at 03:04 UT, resumed at 03:11 UT. Missed scan 253-0306.
 +More wind stow at 08:32:05UT Scan 253-0849a missed
 +and again… scans 1102 to 1114 missed
 +Ke: 
 +Recording to HOB+0124 (bank B), +2786 GB.
 +Bank A has empty 32 GB drive ready for next shift.
 +Tsys high in bands 09, 10 and 11 (>160K).
 +As of 04:00 UT, Tsys >200K in bands 09 and 10, and high in bands 05, 11 and 12 (>150K).
 +Yg: 
 +Recording to BKG-0057 (bank A), +0 GB.
 +Bank B has empty 32 GB drive ready for next shift.
 +Tsys high in band 11 (140K).
 +e-RemoteCtrl connection unreliable and needs to restarted occasionally.
 +----------------------------------------------------------------------------------------------------------------------
 +r1653
 +Hb: 
 +Missed scan 258-1702a due to SpaceX changeover.
 +Recording to USN-0201, +0 GB.
 +Ke: 
 +Missed scans up to and including 258-1901a due to VPN issue.
 +Missed scans up to and including 258-1939 due to bad autocorrelation. First scan is 258-1944b.
 +Recording to BKG-0052, +0 GB.
 +Yg: 
 +Missed scans up to and including 258-1837b due to VPN issue. First scan is 258-1841.
 +Mark 5 at Yarragadee was powered off, however, this problem was not identified and rectified until 258-2037.
 +Recording to USN-0179, +0 GB.
 +Chris' comments
 +Some (all?) scans have not been observed due to problems with the mk5. I appear to have made it run again. First scan is 258-2227a at 22:27:53.
 +Recording to module TR-00014.
 +Was anything recorded to USN-0179? Why are we not using it anymore?
 +Scans 259-0005, 259-0007a, 259-0009a were missed for a module swap. 32 TB module HOB+0119 is now in bank B.
 +Ross' comments
 +USN-0179 has no useful data on it but was swapped out in case it was at fault.
 +0012. Start disk_pos is 0 GB (Vasaant) ​
 ++SUMMARY ​
 + ​Qcode ​     % of Total   % of Correlated ​
 +                scans          scans     
 +   ​5-9 ​           92%            97% 
 +   ​0 ​             1%             ​1% ​
 + ​4-1,​A-H,​N ​       2%             ​2% ​
 + ​Removed ​         5%             ​- ​
 ++CORRELATOR_NOTES
 +
 +Katherine data transferred to Bonn on friday the 26th of September.
 +HOBART12 (Hb/d): Channel XR1L removed from fringe fitting. ​
 +
 + ​KATH12M ​ (Ke/a): Late start at UT 258-1944 due to network problems.
 + ​YARRA12M (Yg/i): Late start at 258-2227 due to VPN network failure.
 +
 +                  Applied manual pcal. 
 ++DROP_CHANNELS:​ Hb -- XR1L  ​
 +Session Statistics ​
 +  Observations: ​    4030 scheduled ​
 +                    3749 correlated (in database)
 +                    3609 recoverable (usable)
 +                    3387 used
 +  Session fit:        33.818 ps
 +-----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +  HART15M ​                ​764 ​          ​667 ​         636        83.2%
 +  HOBART12 ​               741           ​642 ​         534        72.1%
 +  KATH12M ​                ​850 ​          ​716 ​         677        79.6%
 +  KOKEE                  1082           ​982 ​         938        86.7%
 +  NYALES20 ​              ​1205 ​         1132         ​1096 ​       91.0%
 +  TSUKUB32 ​              ​1421 ​         1308         ​1216 ​       85.6%
 +  WETTZELL ​              ​1199 ​         1141         ​1091 ​       91.0%
 +  YARRA12M ​               798           ​630 ​         586        73.4%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       4030          3609         ​3387 ​       84.0%
 +----------------------------------------------------------------------------------------------------------------------
 +r1654
 +Hb: 
 +Recording to FGI-0004. + 0 GB.
 +SpaceX
 +Missed scans 266-0211 through 266-0257
 +Missed scans 266-0357 through 266-0423a
 +Missed scans 266-0526b through 266-0606
 +Data before 266-0704 may have incorrect pointing corrections. Refractive corrections are used for SpaceX tracking, and these may have been erroneously enabled.
 +Missed scans 266-0704 through 266-0736
 +Missed scans 266-0839 through 266-0920b
 +Ke: 
 +Recording to USN-0202. + 0 GB.
 +Missed scans 266-0211 and 266-0216 due to over-eagerness with SpaceX support.
 +Missed scans 266-0400b through 266-0631 due to pcfs crash whilst producing autocorrelations.
 +Yg: 
 +Recording to USN-0086. + 0 GB.
 +Network connection has been off and on since start of experiment, fixed by 266-0000. (<​Chris>​ This seems benign.)
 +Missed scans 266-0211 and 266-0216 due to over-eagerness with SpaceX support.
 +mk5=dot? reveals -1s formatter - Was throughout observations.
 ++SUMMARY ​
 + ​Qcode ​     % of Total   % of Correlated ​
 +                scans          scans     
 +   ​5-9 ​           92%            95% 
 +   ​0 ​             1%             ​1% ​
 + ​4-1,​A-H,​N ​       3%             ​3% ​
 + ​Removed ​         4%             ​- ​
 +HOBART12 (Hb/d): Channel XR1L removed from fringe fitting to avoid 
 +                  losts of G-codes.
 +                  Lost (for SpaceX support) scans:
 +                  from UT 266-0211 to UT 266-0300  ​
 +                  from UT 266-0347 to UT 266-0424  ​
 +                  from UT 266-0704 to UT 266-0736
 +                  from UT 266-0839 to UT 266-0920. ​
 +
 + ​KATH12M ​ (Ke/a): Lost (for SpaceX support) scans 
 +                  from UT 266-0211 to UT 266-0300
 +                  from UT 266-0403 to UT 266-0626.
 + ​YARRA12M (Yg/i): Applied manual pcal.
 +                  Lost (for SpaceX support) scans 
 +                  from UT 266-0214 to UT 266-0226.
 ++DROP_CHANNELS:​ Hb      XR1L
 +Session Statistics ​
 +  Observations: ​    4262 scheduled ​
 +                    5542 correlated (in database)
 +                    4171 recoverable (usable)
 +                    4594 used
 +  Session fit:        30.666 ps
 +-----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +  FORTLEZA ​               533           ​605 ​         588       ​110.3%
 +  HOBART12 ​               972           ​749 ​         741        76.2%
 +  KATH12M ​               1086           ​892 ​         870        80.1%
 +  NYALES20 ​              ​1096 ​         1324         ​1294 ​      ​118.1%
 +  TSUKUB32 ​              ​1448 ​         1357         ​1330 ​       91.9%
 +  WETTZELL ​              ​1195 ​         1459         ​1433 ​      ​119.9%
 +  YARRA12M ​              ​1043 ​          ​885 ​         867        83.1%
 +  YEBES40M ​              ​1151 ​         1071         ​1031 ​       89.6%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       4262          4171         ​4077 ​       95.7%
 +----------------------------------------------------------------------------------------------------------------------
 +R1655
 +Hb recording to HOB+0015. Start disk_pos is 1892 GB 
 +UT 17:00 - Wind stowed at start.
 +UT 17:25 - First source after windstow. 7 missed scans: 272-1700 through to 272-1720 corresponding to 30 GB of data
 +UT 21:02 - Wind stow. Missed scans from 272-2104 through to 272-2204 corresponding to 50 GB of data.
 +UT 22:07 - Observing resumed.
 +UT 22:36 - Wind stow. 
 +UT 23:00 Halted schedule because of wind (Imogen)
 +UT ~01:00 schedule resumed (Imogen)
 +UT ~2:30 drives at Hb got themselves into a state after a windstow - had to reset them several times meaning more scans lost (Imogen)
 +UT ~3:00 no more windstows for Hobart until the end of my shift - Imogen
 +Counters timing out occasionally.
 +UT 11:23 - rebooted pcfshb in large gap to try to fix problem. No missed scans.
 +Ke recording to JIVE-064. Start disk_pos is 0 GB (Vasaant) ​
 +Yg recording to TR-0
 ++SUMMARY
 + ​Qcode ​     % of Total   % of Correlated
 +                scans          scans
 +  5-9            90%            95%
 +   ​0 ​             3%             3%
 + ​4-1,​A-H,​N ​       2%             2%
 + ​Removed ​         5%             -
 +
 ++CORRELATOR_NOTES
 +
 +New submission of correlation report only because a wrong
 +one-letter-station code was given to Sejong (Kv) in the STATION_NOTES
 +section and the Kv clock was missing in the CLOCKS section.
 + ​HOBART12 (Hb/d): Station reports many wind stows and restarts leading to
 +                  numerous non-detections;​
 +                  missed 272-1700 through 272-1720 at experiment start;
 +                  missed 272-2104 through 272-2204; missed 272-2236 through
 +                  273-0318. Wind stowed for scans 273-0338 through 273-0349b
 +                  although recording continued.
 +
 + ​KATH12M ​ (Ke/a): Ok.
 + ​YARRA12M (Yg/i): Applied manual pcal.
 +                  Occasionally low fringe amplitude in S-band channel SR6U on
 +                  baseline Hb - Yg leading to some G codes.
 +Session Statistics ​
 +
 +  Observations: ​    3788 scheduled ​
 +                    3535 correlated (in database)
 +                    3338 recoverable (usable)
 +                    3206 used
 +  Session fit:        35.757 ps
 +-----------------------------------------
 +Station Performance ​
 +                           ​Number of Observations
 +                    Scheduled ​  ​Recoverable* ​       Used   % of scheduled
 +                                                              obs used
 +  FORTLEZA ​               371           ​315 ​         306        82.5%
 +  HOBART12 ​              ​1002 ​          ​680 ​         663        66.2%
 +  KATH12M ​               1121           ​998 ​         963        85.9%
 +  NYALES20 ​              ​1003 ​          ​947 ​         901        89.8%
 +  SEJONG ​                 627           ​555 ​         528        84.2%
 +  TSUKUB32 ​              ​1394 ​         1299         ​1239 ​       88.9%
 +  WETTZELL ​               962           ​909 ​         866        90.0%
 +  YARRA12M ​              ​1096 ​          ​973 ​         946        86.3%
 +  --------------- ​  ​--------- ​  ​----------- ​   --------- ​      ​------
 +  Station Total** ​       3788          3338         ​3206 ​       84.6%
 +---------------------------------------------------------------------------------------------------------------------
 +crds73
 +Hb: 
 +Recording to HOB+0110, + 10443 GB
 +10:15 – Wind-stowed
 +10:24:14 – On source after returning from stow
 +17:05 - another wind stow
 +17:54:30 - manual return from wind stow (onsource @ 17:56:20)
 +Ke: 
 +Recording to UAO-0020, + 0 GB
 +Yg: 
 +Recording to USN-0098, + 0 GB
 +Ho: 
 +Joined observations around 02:15 UT.
 +Wind stow at 02:33.
 +Wind stows are constant throughout observations.
 +LO unlocking problem experienced and fixed early in the observations (~2:30 UT).
 +
 +RD1409
 +
 +Ho 
 +This is a Lunar Lander observation.
 +No schedules on IVS server by the beginning of the observation,​ experiment started late. First scan: 250-1742 to source 0308-611 @ 280.17:​42:​46.00
 +Receiver 20K stage is hotter than 40K 
 +LO unlocked at: 
 +2014.280.21:​14:​28.44?​ERROR ch -348 Receiver LO is unlocked
 +2014.280.21:​14:​49.51?​ERROR ch -348 Receiver LO is unlocked
 +2014.280.21:​15:​10.59?​ERROR ch -348 Receiver LO is unlocked
 +2014.280.21:​29:​19.43?​ERROR ch -348 Receiver LO is unlocked
 +Tried to repair LO unlocked by moving the antenna manually. Missed scans from 22:31:31 to 22:32:55
 +LO unlocked 281-02:24UT - Brett fixed by moving it a bit 
 +281-02:32UT Temperatures were rising rapidly (along with vacuum pressure) so the schedule was halted so Brett can fix it. Scans after 281-0229 missed. 0337 first scan back. 
 +2014.281.09:​00:​28.49#​rxp#​Receiver focus X : -255.5mm ; Y : 250.8mm ; Z : 18.6mm – Y position is a bit off (should be: 248mm); He supply pressure fell from 307psi to 302psi in 4 hours 
  
/home/www/auscope/opswiki/data/pages/operations/correlator_reports2014-11-06.txt · Last modified: 2014/11/06 00:31 by liza