User Tools

Site Tools


operations:handover

This is an old revision of the document!


Achtung! R1 TRANSFER IN PROGRESS! MONITORING REQUIRED!

R1 monitoring instructions: operations:monitoring.ivs.etransfer

Handover notes

Please use this page to write up any notes on anything that needs to be passed on to the next observer or should go into the end of experiment message.

Notes should be placed below the horizontal line as a new blog entry.

Continuing problems should be copied to the Current Issues page.

**Template for Handover notes:**

Be sure to add any specific notes about the experiment if required, for example “Special back end configuration for x reason”. Put as much detail as you can - the more the better.

It is ESSENTIAL to put your name after EVERY entry. Copy and paste the template into the new handover notes

= Experiment Name =

Hobart 12m:

Disk VSN:

Data volume at beginning:

Katherine 12m:

Disk VSN:

Data volume at beginning:

Yarragadee 12m:

Disk VSN:

Data volume at beginning:

Format for entering a comment It is important to stick to this formula - there is a good reason we need this information, so please include it!

Time stamp in UT * Comment * (Your Name)

For Example:

20:00UT: Wind stows, Missed scans 012-3456 to 012-3457 (Jim)

20:50UT: Mark5 lost connection, had to restart Mark5 and reconfigure DBBC, called Jamie on call to fix, missed scans 012-3456 to 012-4567 (Jim)

Again - as much detail as you can. It is important to remember that these notes will be sent to the correlator, so please make sure they make sense to others.


r4935

Yarragadee 12m

Disk VSN: NRAO-172 | Data volume at beginning: 0.0 GB

  • 1830UT Experiment started okay (Eloise).
2020/02/27 18:34 · Wilson-Mayne · 0 Comments
AUA061

The correct monitoring tools were not set up on ops8 when I started (no eremote control, no log monitors or delaylive's)(Lucas)

Yarragadee 12m

Disk VSN: HOB+0056 | Data volume at beginning: 0.0 GB

  1. 1730UT Experiment started OK (Jay).
  2. 6:00UT Very large positive(!?) data difference. +200TB or about 5%. Module is large enough to handle additional data (Lucas).

Hobart 12m

Disk VSN: flexbuffhb | Data volume at beginning: 0.0 GB

  1. 1730UT Experiment started okay (Jay).
  2. 7:00UT When I set up the delaylive, noticed a series of clock-jumps on the dbbcho starting at 5:00UT, then again at 6:00,6:30 and 7:00. Clock jumps are on the order of -0.015us. (Lucas)
    1. Have not reset the dbbcho (yet) as the summary script reports okay delays and samplers. Appears stable at the moment but if it jumps again I will reset (Lucas)
    2. Jumped again at 8:00 UT, halted and reconfigured dbbcho. Missed scan 057-0821b, 0825.
    3. Called Jamie. clkoff isn't actually indicative for dbbcho problems when hooked up to Hobart12m. So don't worry about delaylive hb.

Katherine 12m

Disk VSN: flexbuffke | Data volume at beginning: 0.0 GB

  1. 1730UT Experiment started okay (Jay).
  2. Drives lost connection, got stuck. Required a thorough reboot (reboot drives didn't work), reboot system then a 2min wait+[drives on]+operate+source=disable+antenna=open+antenna=operate worked (Lucas).
    1. Missed scans 057-0839 up to 057-0851.
  3. Flexbuffke got into a nasty state and wasn't happy. The first sign was the data rate on Grafana went from the 'normal' smooth 3Gbps scans to ugly, spiky and somewhat random data rates. Another sign was the number of dropped packets in “mk5=evlbi?” – maybe we should add this to katherine midob?
    1. The current fix is to (halt obviously), umount all the drives, remount them, restart jive5ab and re-setupsx.
      1. On flexbuffke there are two scripts in /tmp/. After logging in as su (ssh observer@flexbuffke then su in random terminal), run script /tmp/UMountAllFlexbuff.sh; wait make sure they all unmount, then run /tmp/MountAllFlexbuff.sh. After this you need to go to the flexbuffke jive5ab window (normally running on the vnc, with lots of “Reply: !”–type stuff) and ctl-C OR just pkill jive5ab from observer@flexbuffke. Then restart by up-arrowing or typing “/opt/jive5ab-2.9.0 -m2 &” or maybe …“-3.0.0” jury is out which one atm. Then setupsx in the field system and make sure the datarate is okay and “mk5=evlbi?” looks happy and is increasing (Lucas).
2020/02/25 17:40 · Patrick Yates · 0 Comments
r1935

Yarragadee 12m

Disk VSN: UAO-0016 | Data volume at beginning: 0.0 GB

  • 1700UT Experiment started okay (Lim).
2020/02/24 12:02 · Patrick Yates
r4934

Yarragadee 12m

Disk VSN: USN-0193/2000| Data volume at beginning: 1934.908GB

  • 1830UT Experiment started okay (Becca).
  • 2020.051.20:11:07 ALARM: Large difference between formatter and maser delays. Check for stability of new offset. 2020.051.20:11:09
    • WARNING: error ib -4 gpib device time-out on response c1 2020.051.20:11:11 DBBC pps_delay1 = 61950 2020.051.20:11:18 Alarm
  • 2020.051.20:40:02 ALARM: Large difference between formatter and maser delays. Check for stability of new offset.
  • Above error happens occasionally an clears itself (Lim)
2020/02/20 20:28 · Patrick Yates
r4933

Yarragadee 12m

Disk VSN: USN-0192| Data volume at beginning: 0 GB

  • 1830UT Experiment started okay (Katie).
2020/02/13 21:04 · Patrick Yates · 0 Comments
AOV044

Hobart 26m

Disk VSN: HOB+0015 | Data volume at beginning: 2561.372 GB

  • 1730UT No communication to antenna (Lim)
  • 1814UT Antenna connection resumed. First scan started on 042-1814a (Lim).
    • There were too many “vdesk” running on newsmerd, all terminated by Jamie.
    • Run vdesk again and “drvon” for about 15 minutes until the connection comes back.
  • 1954UT fs crashed, scans 042-1957b to 042-2024 were probably missed as well (Lim).
  • 2022UT fs-auscope restarted, next scan is 042-2028 (Lim).
  • 2120UT Error: Cannot get monitor info from antenna (8020003); #antcn#Network I/O Timeout occurred on write
  • 2153UT DrivePC (sys26m) and Hobart machine rebooted, that sort of cleared the problem, which was a result of power failure. Next scan 042-2159 . Missed scans: 042-2119 to 042-2157 (Lim).
  • 0812UT - There seemed to be a power failure. Tried restarting the FS and running the schedule however that showed up The allocating memory for antenna monitoring error. Rebooted the drivepc by using the command “rem_reboot -r rakbus sys26m” on observer@newsmerd. The RA/Dec/time values which were static earlier, now started changing (as seen on rack 1 through the camera). Did “source=disable” on the FS to resume the schedule. Scans 043-0811 to 043-0844 affected (Prad).
  • 0910UT - Drives weren't ready on time for the last few scans. “drvon” on vdesk a few times seemed to have done the trick. Left drive (on) delay causing the problem? Scans 043-0848b to 043-0908 likely to have been affected (Prad).

Yarragadee 12m

Disk VSN: HOB+0033 | Data volume at beginning: 1875.572 GB

  • 1730UT Experiment started okay (Lim).
2020/02/11 17:31 · Pradyumna Kiran Kummamuru
/home/www/auscope/opswiki/data/attic/operations/handover.1434552531.txt.gz · Last modified: 2015/06/17 14:48 by liza