User Tools

Site Tools


operations:documentation:yg_manual_weather

Manual extraction and logging of Yarragadee weather data

We usually have a weather station connected to pcfsyg. However, if it is unavailable or broken, there is another weather sensor (used for the SLR) and it is possible to extract the data and merge it with the PCFS logs before they are submitted. Below is the procedure for doing this (a detailed description as well as a summary). Note that the external logs are updated every 30 minutes, so the Yg logs can only be processed 30 mins or more after the experiment finishes.

Using a script

On ops2 in /home/observer/ask/yglogs lives a script called yg_wx_logs.py which can be used to merge the logs with the weather data. To run the script type in

 python yg_wx_logs.py 

The script will prompt the user for

  • Experiment name,
  • Date of the experiment,
  • Start time,
  • End time.

With the given information, the script will

  • Download the required mets files containing the weather data using sftp,
  • Download the log file from pcfyg,
  • Merge the log from pcfs with the downloaded weather data,
  • Backup the original log on pcfsyg with a _original2.log,
  • Copy the modified log back to the pcfsyg.

Once all that is done you can run the flogit script to transfer that log to the IVS servers. As usual please eyeball the log to make sure the script has done the right thing.


It's probably best to modify the PCFS log on pcfsyg, then follow the usual end-of-experiment procedure. This way the log file will be the same on pcfsyg, ops2, and on the IVS servers. The procedure below uses the experiment r4750 as an example:

Detailed description

  1. Identify the start date of the experiment. There's a separate log of weather for every UT day, so you'll need to retrieve the log for the start day as well as the following day. In the case of R4750, the start date was July 28 (day 210) at 18:30 UT, so we need logs from July 28 and 29.
  2. Once the experiment is complete (and it's more than 30 minutes from the end), open a terminal on pcfsyg. Do this in the pcfsyg VNC session, especially if you want to plot the data, which is an option later.
  3. cd /usr2/log/wx2
  4. The weather data are on an sftp server at Yg. Use the usual observer password. The file naming format is mets_<yyyy>_<mm>_<dd>.dat so we want mets_2016_07_28.dat and mets_2016_07_29.dat
    sftp -oPort=26 observer@59.167.111.242
    cd metdata
    dir
    get mets_2016_07_28.dat
    get mets_2016_07_29.dat
    quit
  5. The weather data files now need to be converted to the correct format for a PCFS log. There's a script to do that called metdat_format.sh. By default it writes the output log to the screen, so send it to a temporary file. e.g.
    metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log

    Specify the input data files in chronological order.

  6. Now you need to edit the new log file and delete any weather measurements before or after the experiment. There's a script for that too called metdat_startstop.sh. Usage is
    metdat_startstop.sh <input_file_name> <start_time> <end_time>
    Where the time format is ddd.hh:mm. The output from this should be ready to be merged into the original log file, so give it a sensible output name and put it in the log directory (''/usr2/log''). 

    For example for R4750:

    metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log
  7. You can now merge the original log file and the weather data. Change to the /usr2/log directory
    cd /usr2/log
  8. Check that the weather data start and stop at the right time. Show the first 5 lines
    head -5 wx_r4750yg.log

    Show the last 5 lines

    tail -5 wx_r4750yg.log

    If something is amiss, you may need to go back to the start and work out what happened. You can also plot the weather data quite easily:

    metdat_plot_weather.sh wx_r4750yg.log
  9. It's a good idea to make a backup of the original log first
    cp r4750yg.log r4750yg_original.log
  10. Processing the log is a two step process.
    1. First, you want to remove any existing weather data from the log as it will be blank or contain errors
      grep -v /wx/ r4750yg.log > r4750yg_no_wx.log
    2. Next merge the field system and weather logs to produce a final version:
      sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log
  11. Finally, make a few spot checks of the final log by eye to see if it looks OK. Hunt around for the wx lines and see if they look right and are in the right place.
  12. You can now go back to ops2 and process the log file and submit a end email as usual. Please add a note to the email to say that weather data were merged into the log after the experiment.

Executive summary

This assumes R4750 is being processed.

  1. Identify the start date of the experiment.
  2. After the experiment AND 0 UT open a terminal in the pcfsyg VNC session.
  3. cd /usr2/log/wx2
    sftp -oPort=26 observer@59.167.111.242
    cd metdata
    dir
    get mets_2016_07_28.dat
    get mets_2016_07_29.dat
    quit
    metdat_format.sh mets_2016_07_28.dat mets_2016_07_29.dat > wx_tmp.log
    metdat_startstop.sh wx_tmp.log 210.18:30 211.18:30 > /usr2/log/wx_r4750yg.log
    cd /usr2/log
    head -5 wx_r4750yg.log
    tail -5 wx_r4750yg.log
    metdat_plot_weather.sh wx_r4750yg.log
    cp r4750yg.log r4750yg_original.log
    grep -v /wx/ r4750yg.log > r4750yg_no_wx.log
    sort -s -m -k 1,1.20 -o r4750yg.log r4750yg_no_wx.log wx_r4750yg.log
  4. Make a few spot checks of the final log by eye
  5. You can now go back to ops2 and process the log file and submit a end email as usual. Please add a note to the email to say that weather data were merged into the log after the experiment.
/home/www/auscope/opswiki/data/pages/operations/documentation/yg_manual_weather.txt · Last modified: 2016/11/23 21:31 by Bryn Emptage